Oozie

This release provides Oozie 4.2.0 with no additional Apache patches.

• OOZIE-2571: Add spark.scala.binary.version Maven property so that Scala 2.11 can be used.

• OOZIE-2606: Set spark.yarn.jars to fix Spark 2.0 with Oozie.

• OOZIE-2658: --driver-class-path can overwrite the classpath in SparkMain.

• OOZIE-2787: Oozie distributes application jar twice making the spark job fail.

• OOZIE-2792: Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark.

• OOZIE-2799: Setting log location for spark sql on hive.

• OOZIE-2802: Spark action failure on Spark 2.1.0 due to duplicate sharelibs.

• OOZIE-2923: Improve Spark options parsing.

• OOZIE-3109: SCA : Cross-Site Scripting: Reflected.

• OOZIE-3139: Oozie validates workflow incorrectly.

• OOZIE-3167: Upgrade tomcat version on Oozie 4.3 branch.

HDP 2.6.4 provided Oozie 4.2.0 with no additional Apache patches.

HDP 2.6.3 provided Oozie 4.2.0 with no additional Apache patches.

HDP 2.6.2 provided Oozie 4.2.0 with no additional Apache patches.

HDP 2.6.1 provided Oozie 4.2.0 with no additional Apache patches.

HDP 2.6.0 provided Oozie 4.2.0 and the following Apache patches:

• OOZIE-1402: Increase retry interval for non-progressing coordinator action with fix value.

• OOZIE-1837: LauncherMainHadoopUtils sensitive to clock skew.

• OOZIE-1922: MemoryLocksService fails if lock is acquired multiple times in same thread and released.

• OOZIE-1976: Specifying coordinator input datasets in more logical ways.

• OOZIE-2030: Configuration properties from global section is not getting set in Hadoop job conf when using sub-workflow action in Oozie workflow.xml.

• OOZIE-2159: 'oozie validate' command should be moved server-side.

• OOZIE-2165: Job log fetching can fail in Oozie HA mode when using doAs impersonation.

• OOZIE-2168: Oozie flow and action names have 50 char limit.

• OOZIE-2173: DISCLAIMER.txt still refers to Apache Incubator.

• OOZIE-2178: fix javadoc to compile on JDK8.

• OOZIE-2185: Make oozie cli source conf/oozie-client-env.sh.

• OOZIE-2187: Add a way to specify a default JT/RM and NN.

• OOZIE-2215: Support glob in FS EL function.

• OOZIE-2222: Oozie UI parent job should be clickable.

• OOZIE-2228: Statustransit service doesn't pick bundle with suspend status.

• OOZIE-2245: Service to periodically check database schema.

• OOZIE-2246: CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist.

• OOZIE-2250: Show log for WAITING and TIMEDOUT coord actions.

• OOZIE-2251: Expose instrumental matrices in Realtime Graphing tool.

• OOZIE-2257: BundleStatusTransitXCommand doesn't update last modified time.

• OOZIE-2260: Only set "java.io.tmpdir" to "tmp" for the AM.

• OOZIE-2262: Fix log streaming from other server with start/end filter.

• OOZIE-2264: Fix coord:offset(n,"DAY") to resolve correct data set.

• OOZIE-2266: Fix 'total' actions returned in coordinator job info.

• OOZIE-2268: Update ActiveMQ version for security and other fixes.

• OOZIE-2270: Audit log doesn't get loaded when "audit log" tab is clicked for coord job.

• OOZIE-2271: Upgrade Tomcat to 6.0.44.

• OOZIE-2272: Use Hadoop's CredentialProvider for passwords in oozie-site.

• OOZIE-2277: Honor oozie.action.sharelib.for.spark in Spark jobs.

• OOZIE-2283: Documentation should not say that System.exit is not allowed in Java Action.

• OOZIE-2284: HBaseCredentials should only add hbase-default.xml and hbase-site.xml to actionConf.

• OOZIE-2285: Change in concurrency should trigger coord action ready command.

• OOZIE-2286: Update Log4j and Log4j-extras to latest 1.2.x release.

• OOZIE-2287: Add support for deleting hcat partitions in fs action delete.

• OOZIE-2289: hive-jdbc dependency in core/pom.xml should be compile.

• OOZIE-2291: Hive2 workflow.xml.security should have "cred" in action tag instead of "hive2" tag.

• OOZIE-2293: Oozie 4.0.1 build failed while building Catalog.

• OOZIE-2300: TestAuthFilterAuthOozieClient.testClientAuthTokenCache fails with Hadoop 2.7.0 and later.

• OOZIE-2303: Typo in documentation.

• OOZIE-2305: Compile Oozie with Hive-1.2.0.

• OOZIE-2308: Add support for bundle:conf() function.

• OOZIE-2309: Enable the coord:dateOffset() function in /coordinator-app/datasets/dataset/@initial-instance.

• OOZIE-2312: oozie doesn't purge audit and error log.

• OOZIE-2314: Unable to kill old instance child job by workflow or coord rerun by Launcher.

• OOZIE-2315: TestOozieCLI.testshareLibUpdate_withSecurity fails with Hadoop 2.

• OOZIE-2318: Provide better solution for specifying SSL truststore to Oozie Client.

• OOZIE-2320: TestZKXLogStreamingService.testStreamingWithMultipleOozieServers_coordActionList is failing.

• OOZIE-2322: Oozie Web UI doesn't work with Kerberos in Internet Explorer 10 or 11 and curl.

• OOZIE-2324: A syntax error in the kill node causes the workflow to get stuck and other problems.

• OOZIE-2325: Shell action fails if user overrides oozie.launcher.mapreduce.map.env.

• OOZIE-2328: Coordinator endtime change should check if the last action is in database.

• OOZIE-2329: Make handling yarn restarts configurable.

• OOZIE-2330: Spark action should take the global jobTracker and nameNode configs by default and allow file and archive elements.

• OOZIE-2332: Add ability to provide Hive and Hive 2 Action queries inline in workflows.

• OOZIE-2343: Shell Action should take Oozie Action config and setup HADOOP_CONF_DIR.

• OOZIE-2344: Enabling 'oozie.action.jobinfo.enable' doesn't inject the job information into the map/reduce job's configuration..

• OOZIE-2345: Parallel job submission for forked actions.

• OOZIE-2346: Add sub-workflow information like the super-parent id and workflow depth into the 'oozie.job.info' property.

• OOZIE-2347: Remove unnecessary new Configuration()/new jobConf() calls from oozie.

• OOZIE-2348: Recovery service keeps on recovering coord action of suspended jobs.

• OOZIE-2350: Package changes for release.

• OOZIE-2355: Hive2 Action doesn't pass along oozie configs to jobconf.

• OOZIE-2356: Add a way to enable/disable credentials in a workflow.

• OOZIE-2358: Coord rerun cleanup should reuse hcat connections.

• OOZIE-2360: Spark Action fails due to missing mesos jar.

• OOZIE-2362: SQL injection in BulkJPAExecutor.

• OOZIE-2365: oozie fail to start when smtp password not set.

• OOZIE-2367: fs delete should support skipTrash option.

• OOZIE-2368: coord:dateOffset and coord:dateTzOffset can't be used for coord initial-instance.

• OOZIE-2369: coord:offset doesn't resolve correctly.

• OOZIE-2370: Docs for Coordinator Action Status Notification has wrong property name.

• OOZIE-2376: Default action configs not honored if no <configuration> section in workflow.

• OOZIE-2377: Hive2 Action should not propagate oozie.hive2.* properties to Beeline.

• OOZIE-2378: org.apache.oozie.client.TestOozieCLI.testAdminInstrumentation is flakey.

• OOZIE-2379: org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand.testRequeueOnException is flakey.

• OOZIE-2380: Oozie Hive action failed with wrong tmp path.

• OOZIE-2382: org.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID is flakey.

• OOZIE-2385: org.apache.oozie.TestCoordinatorEngineStreamLog.testCoordLogStreaming is flakey.

• OOZIE-2386: org.apache.oozie.util.TestXConfiguration.testSubstituteVar is flakey.

• OOZIE-2391: spark-opts value in workflow.xml is not parsed properly.

• OOZIE-2394: Oozie can execute command without holding lock.

• OOZIE-2397: LAST_ONLY and NONE don't properly handle READY actions.

• OOZIE-2400: Workflow xml configuration parser cannot deal with namespace prefix.

• OOZIE-2401: Typo in twiki docs with FilesAchives instead of FilesArchives.

• OOZIE-2402: oozie-setup.sh sharelib create takes a long time on large clusters.

• OOZIE-2407: AbandonedService should not send mail if there is no abandoned coord.

• OOZIE-2411: Add BCC to oozie email action.

• OOZIE-2413: Kerberos credentials can expire if the KDC is slow to respond.

• OOZIE-2419: HBase credentials are not correctly proxied.

• OOZIE-2428: TestSLAService, TestSLAEventGeneration flakey tests.

• OOZIE-2429: TestEventGeneration test is flakey.

• OOZIE-2430: Add root logger for hive,sqoop action.

• OOZIE-2431: Support hive.metastore.uris as alias for hcat.metastore.uri.

• OOZIE-2432: TestPurgeXCommand fails.

• OOZIE-2434: inconsistent coord action status and workflow job status.

• OOZIE-2435: TestCoordChangeXCommand is flakey.

• OOZIE-2436: Fork/join workflow fails with "oozie.action.yarn.tag must not be null".

• OOZIE-2437: Exclude additional Hadoop dependencies from Spark Sharelib.

• OOZIE-2438: Oozie client "jobs -filter" diagnostic message clarification.

• OOZIE-2439: FS Action no longer uses name-node from global section or default NN.

• OOZIE-2441: SubWorkflow action with propagate-configuration but no global section throws NPE on submit.

• OOZIE-2444: Need conditional logic in bundles.

• OOZIE-2445: Doc for - Specifying coordinator input datasets in more logical ways.

• OOZIE-2446: Job does not fail during submission if non existent credential is specified.

• OOZIE-2447: Illegal character 0x0 oozie client.

• OOZIE-2452: Coordinator Functional Specification - EL Constants Typo.

• OOZIE-2466: Repeated failure of TestMetricsInstrumentation.testSamplers.

• OOZIE-2467: Oozie can shutdown itself on long GC pause.

• OOZIE-2470: Remove infinite socket timeouts in the Oozie email action.

• OOZIE-2471: Show child job url tab for distcp.

• OOZIE-2474: <job-xml> is not being applied to the launcher job.

• OOZIE-2475: Oozie does not cleanup action dir of killed actions.

• OOZIE-2476: When one of the action from fork fails with transient error, WF never joins.

• OOZIE-2481: Add YARN_CONF_DIR in the Shell action.

• OOZIE-2482: Pyspark job fails with Oozie.

• OOZIE-2485: Oozie client keeps trying to use expired auth token.

• OOZIE-2486: TestSLAEventsGetForFilterJPAExecutor is flakey.

• OOZIE-2489: XML parsing is vulnerable.

• OOZIE-2490: Oozie can't set hadoop.security.token.service.use_ip.

• OOZIE-2492: JSON security issue in js code.

• OOZIE-2503: show ChildJobURLs to spark action.

• OOZIE-2504: Create a log4j.properties under HADOOP_CONF_DIR in Shell Action.

• OOZIE-2509: SLA job status can stuck in running state.

• OOZIE-2510: TestLogStreamer.testStreamLog() fails in other timezones.

• OOZIE-2511: SubWorkflow missing variable set from option if config-default is present in parent workflow.

• OOZIE-2512: ShareLibservice returns incorrect path for jar.

• OOZIE-2522: There can be multiple coord submit from bundle in case of ZK glitch.

• OOZIE-2529: Support adding secret keys to Credentials of Launcher.

• OOZIE-2531: Prevent Spark trying for token which is already available.

• OOZIE-2532: patch apply does not handle binary files.

• OOZIE-2535: User can't disable uber mode.

• OOZIE-2537: SqoopMain does not set up log4j properly.

• OOZIE-2541: Possible resource leak in Hive2Credentials.

• OOZIE-2542: Option to disable OpenJPA BrokerImpl finalization.

• OOZIE-2546: Improperly closed resources in OozieDBCLI.

• OOZIE-2547: Add mapreduce.job.cache.files to spark action.

• OOZIE-2548: Flaky test TestZKLocksService.testLockReaper.

• OOZIE-2550: Flaky tests in TestZKUUIDService.java.

• OOZIE-2551: Feature request: epoch timestamp generation.

• OOZIE-2553: Cred tag is required for all actions in the workflow even if an action does not require it.

• OOZIE-2556: TestAbandonedCoordChecker.testCatchupJob is flaky.

• OOZIE-2563: Pass spark-defaults.conf to spark action.

• OOZIE-2567: HCat connection is not closed while getting hcat cred.

• OOZIE-2570: remove -PtestPatchCompile from patch testing as there is no such profile.

• OOZIE-2577: Flaky tests TestCoordActionInputCheckXCommand.testTimeout and testTimeoutWithException.

• OOZIE-2578: Oozie example distcp job fails to run within an encrypted zone with checksum match error.