Release Notes
Also available as:
PDF

Oozie

This release provides Oozie 4.2.0 with no additional Apache patches.

HDP 2.6.0 provided Oozie 4.2.0 and the following Apache patches:

  • OOZIE-1402: Increase retry interval for non-progressing coordinator action with fix value.

  • OOZIE-1837: LauncherMainHadoopUtils sensitive to clock skew.

  • OOZIE-1922: MemoryLocksService fails if lock is acquired multiple times in same thread and released.

  • OOZIE-1976: Specifying coordinator input datasets in more logical ways.

  • OOZIE-2030: Configuration properties from global section is not getting set in Hadoop job conf when using sub-workflow action in Oozie workflow.xml.

  • OOZIE-2159: 'oozie validate' command should be moved server-side.

  • OOZIE-2165: Job log fetching can fail in Oozie HA mode when using doAs impersonation.

  • OOZIE-2168: Oozie flow and action names have 50 char limit.

  • OOZIE-2173: DISCLAIMER.txt still refers to Apache Incubator.

  • OOZIE-2178: fix javadoc to compile on JDK8.

  • OOZIE-2185: Make oozie cli source conf/oozie-client-env.sh.

  • OOZIE-2187: Add a way to specify a default JT/RM and NN.

  • OOZIE-2215: Support glob in FS EL function.

  • OOZIE-2222: Oozie UI parent job should be clickable.

  • OOZIE-2228: Statustransit service doesn't pick bundle with suspend status.

  • OOZIE-2245: Service to periodically check database schema.

  • OOZIE-2246: CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist.

  • OOZIE-2250: Show log for WAITING and TIMEDOUT coord actions.

  • OOZIE-2251: Expose instrumental matrices in Realtime Graphing tool.

  • OOZIE-2257: BundleStatusTransitXCommand doesn't update last modified time.

  • OOZIE-2260: Only set "java.io.tmpdir" to "tmp" for the AM.

  • OOZIE-2262: Fix log streaming from other server with start/end filter.

  • OOZIE-2264: Fix coord:offset(n,"DAY") to resolve correct data set.

  • OOZIE-2266: Fix 'total' actions returned in coordinator job info.

  • OOZIE-2268: Update ActiveMQ version for security and other fixes.

  • OOZIE-2270: Audit log doesn't get loaded when "audit log" tab is clicked for coord job.

  • OOZIE-2271: Upgrade Tomcat to 6.0.44.

  • OOZIE-2272: Use Hadoop's CredentialProvider for passwords in oozie-site.

  • OOZIE-2277: Honor oozie.action.sharelib.for.spark in Spark jobs.

  • OOZIE-2283: Documentation should not say that System.exit is not allowed in Java Action.

  • OOZIE-2284: HBaseCredentials should only add hbase-default.xml and hbase-site.xml to actionConf.

  • OOZIE-2285: Change in concurrency should trigger coord action ready command.

  • OOZIE-2286: Update Log4j and Log4j-extras to latest 1.2.x release.

  • OOZIE-2287: Add support for deleting hcat partitions in fs action delete.

  • OOZIE-2289: hive-jdbc dependency in core/pom.xml should be compile.

  • OOZIE-2291: Hive2 workflow.xml.security should have "cred" in action tag instead of "hive2" tag.

  • OOZIE-2293: Oozie 4.0.1 build failed while building Catalog.

  • OOZIE-2300: TestAuthFilterAuthOozieClient.testClientAuthTokenCache fails with Hadoop 2.7.0 and later.

  • OOZIE-2303: Typo in documentation.

  • OOZIE-2305: Compile Oozie with Hive-1.2.0.

  • OOZIE-2308: Add support for bundle:conf() function.

  • OOZIE-2309: Enable the coord:dateOffset() function in /coordinator-app/datasets/dataset/@initial-instance.

  • OOZIE-2312: oozie doesn't purge audit and error log.

  • OOZIE-2314: Unable to kill old instance child job by workflow or coord rerun by Launcher.

  • OOZIE-2315: TestOozieCLI.testshareLibUpdate_withSecurity fails with Hadoop 2.

  • OOZIE-2318: Provide better solution for specifying SSL truststore to Oozie Client.

  • OOZIE-2320: TestZKXLogStreamingService.testStreamingWithMultipleOozieServers_coordActionList is failing.

  • OOZIE-2322: Oozie Web UI doesn't work with Kerberos in Internet Explorer 10 or 11 and curl.

  • OOZIE-2324: A syntax error in the kill node causes the workflow to get stuck and other problems.

  • OOZIE-2325: Shell action fails if user overrides oozie.launcher.mapreduce.map.env.

  • OOZIE-2328: Coordinator endtime change should check if the last action is in database.

  • OOZIE-2329: Make handling yarn restarts configurable.

  • OOZIE-2330: Spark action should take the global jobTracker and nameNode configs by default and allow file and archive elements.

  • OOZIE-2332: Add ability to provide Hive and Hive 2 Action queries inline in workflows.

  • OOZIE-2343: Shell Action should take Oozie Action config and setup HADOOP_CONF_DIR.

  • OOZIE-2344: Enabling 'oozie.action.jobinfo.enable' doesn't inject the job information into the map/reduce job's configuration..

  • OOZIE-2345: Parallel job submission for forked actions.

  • OOZIE-2346: Add sub-workflow information like the super-parent id and workflow depth into the 'oozie.job.info' property.

  • OOZIE-2347: Remove unnecessary new Configuration()/new jobConf() calls from oozie.

  • OOZIE-2348: Recovery service keeps on recovering coord action of suspended jobs.

  • OOZIE-2350: Package changes for release.

  • OOZIE-2355: Hive2 Action doesn't pass along oozie configs to jobconf.

  • OOZIE-2356: Add a way to enable/disable credentials in a workflow.

  • OOZIE-2358: Coord rerun cleanup should reuse hcat connections.

  • OOZIE-2360: Spark Action fails due to missing mesos jar.

  • OOZIE-2362: SQL injection in BulkJPAExecutor.

  • OOZIE-2365: oozie fail to start when smtp password not set.

  • OOZIE-2367: fs delete should support skipTrash option.

  • OOZIE-2368: coord:dateOffset and coord:dateTzOffset can't be used for coord initial-instance.

  • OOZIE-2369: coord:offset doesn't resolve correctly.

  • OOZIE-2370: Docs for Coordinator Action Status Notification has wrong property name.

  • OOZIE-2376: Default action configs not honored if no <configuration> section in workflow.

  • OOZIE-2377: Hive2 Action should not propagate oozie.hive2.* properties to Beeline.

  • OOZIE-2378: org.apache.oozie.client.TestOozieCLI.testAdminInstrumentation is flakey.

  • OOZIE-2379: org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand.testRequeueOnException is flakey.

  • OOZIE-2380: Oozie Hive action failed with wrong tmp path.

  • OOZIE-2382: org.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID is flakey.

  • OOZIE-2385: org.apache.oozie.TestCoordinatorEngineStreamLog.testCoordLogStreaming is flakey.

  • OOZIE-2386: org.apache.oozie.util.TestXConfiguration.testSubstituteVar is flakey.

  • OOZIE-2391: spark-opts value in workflow.xml is not parsed properly.

  • OOZIE-2394: Oozie can execute command without holding lock.

  • OOZIE-2397: LAST_ONLY and NONE don't properly handle READY actions.

  • OOZIE-2400: Workflow xml configuration parser cannot deal with namespace prefix.

  • OOZIE-2401: Typo in twiki docs with FilesAchives instead of FilesArchives.

  • OOZIE-2402: oozie-setup.sh sharelib create takes a long time on large clusters.

  • OOZIE-2407: AbandonedService should not send mail if there is no abandoned coord.

  • OOZIE-2411: Add BCC to oozie email action.

  • OOZIE-2413: Kerberos credentials can expire if the KDC is slow to respond.

  • OOZIE-2419: HBase credentials are not correctly proxied.

  • OOZIE-2428: TestSLAService, TestSLAEventGeneration flakey tests.

  • OOZIE-2429: TestEventGeneration test is flakey.

  • OOZIE-2430: Add root logger for hive,sqoop action.

  • OOZIE-2431: Support hive.metastore.uris as alias for hcat.metastore.uri.

  • OOZIE-2432: TestPurgeXCommand fails.

  • OOZIE-2434: inconsistent coord action status and workflow job status.

  • OOZIE-2435: TestCoordChangeXCommand is flakey.

  • OOZIE-2436: Fork/join workflow fails with "oozie.action.yarn.tag must not be null".

  • OOZIE-2437: Exclude additional Hadoop dependencies from Spark Sharelib.

  • OOZIE-2438: Oozie client "jobs -filter" diagnostic message clarification.

  • OOZIE-2439: FS Action no longer uses name-node from global section or default NN.

  • OOZIE-2441: SubWorkflow action with propagate-configuration but no global section throws NPE on submit.

  • OOZIE-2444: Need conditional logic in bundles.

  • OOZIE-2445: Doc for - Specifying coordinator input datasets in more logical ways.

  • OOZIE-2446: Job does not fail during submission if non existent credential is specified.

  • OOZIE-2447: Illegal character 0x0 oozie client.

  • OOZIE-2452: Coordinator Functional Specification - EL Constants Typo.

  • OOZIE-2466: Repeated failure of TestMetricsInstrumentation.testSamplers.

  • OOZIE-2467: Oozie can shutdown itself on long GC pause.

  • OOZIE-2470: Remove infinite socket timeouts in the Oozie email action.

  • OOZIE-2471: Show child job url tab for distcp.

  • OOZIE-2474: <job-xml> is not being applied to the launcher job.

  • OOZIE-2475: Oozie does not cleanup action dir of killed actions.

  • OOZIE-2476: When one of the action from fork fails with transient error, WF never joins.

  • OOZIE-2481: Add YARN_CONF_DIR in the Shell action.

  • OOZIE-2482: Pyspark job fails with Oozie.

  • OOZIE-2485: Oozie client keeps trying to use expired auth token.

  • OOZIE-2486: TestSLAEventsGetForFilterJPAExecutor is flakey.

  • OOZIE-2489: XML parsing is vulnerable.

  • OOZIE-2490: Oozie can't set hadoop.security.token.service.use_ip.

  • OOZIE-2492: JSON security issue in js code.

  • OOZIE-2503: show ChildJobURLs to spark action.

  • OOZIE-2504: Create a log4j.properties under HADOOP_CONF_DIR in Shell Action.

  • OOZIE-2509: SLA job status can stuck in running state.

  • OOZIE-2510: TestLogStreamer.testStreamLog() fails in other timezones.

  • OOZIE-2511: SubWorkflow missing variable set from option if config-default is present in parent workflow.

  • OOZIE-2512: ShareLibservice returns incorrect path for jar.

  • OOZIE-2522: There can be multiple coord submit from bundle in case of ZK glitch.

  • OOZIE-2529: Support adding secret keys to Credentials of Launcher.

  • OOZIE-2531: Prevent Spark trying for token which is already available.

  • OOZIE-2532: patch apply does not handle binary files.

  • OOZIE-2535: User can't disable uber mode.

  • OOZIE-2537: SqoopMain does not set up log4j properly.

  • OOZIE-2541: Possible resource leak in Hive2Credentials.

  • OOZIE-2542: Option to disable OpenJPA BrokerImpl finalization.

  • OOZIE-2546: Improperly closed resources in OozieDBCLI.

  • OOZIE-2547: Add mapreduce.job.cache.files to spark action.

  • OOZIE-2548: Flaky test TestZKLocksService.testLockReaper.

  • OOZIE-2550: Flaky tests in TestZKUUIDService.java.

  • OOZIE-2551: Feature request: epoch timestamp generation.

  • OOZIE-2553: Cred tag is required for all actions in the workflow even if an action does not require it.

  • OOZIE-2556: TestAbandonedCoordChecker.testCatchupJob is flaky.

  • OOZIE-2563: Pass spark-defaults.conf to spark action.

  • OOZIE-2567: HCat connection is not closed while getting hcat cred.

  • OOZIE-2570: remove -PtestPatchCompile from patch testing as there is no such profile.

  • OOZIE-2577: Flaky tests TestCoordActionInputCheckXCommand.testTimeout and testTimeoutWithException.

  • OOZIE-2578: Oozie example distcp job fails to run within an encrypted zone with checksum match error.