Hortonworks Docs » Hortonworks Data Platform 3.1.4 » HDP 3.1.4 Release Notes
HDP 3.1.4 Release Notes
Also available as:
PDF

Oozie

This release provides Oozie 4.3.1 and the additional Apache patches.

  • OOZIE-3194: [tools] Oozie should set proper permissions to sharelib after upload

  • OOZIE-3439: Hive2 action is not parsing application ID for TEZ from log file properly

  • OOZIE-3440 :[action] Oozie Spark action replaces path symlink # to %23

  • OOZIE-3370: Property filtering is not consistent across job submission

HDP 3.0.1, and HDP 3.1.0 provided Oozie 4.3.1 and the following Apache patches:

  • OOZIE-3304: Parsing sharelib timestamps is not threadsafe (dionusos, matijhs via andras.piros)

  • OOZIE-3135: Configure log4j2 in SqoopMain

  • OOZIE-3072: oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files (gezapeti)

  • OOZIE-2916: Set a job name for the MR Action's child job (asasvari)

  • OOZIE-2494: Cron syntax not handling DST properly (kmarton via andras.piros)

  • OOZIE-2726: Flaky test due to daylight saving changes (sasishsaley, andras.piros via gezapeti)

  • OOZIE-3173 : Coordinator job with frequency using cron syntax creates only one action in catchup mode (andras.piros)

  • OOZIE-2691: Show workflow action retry information in UI

  • OOZIE-3109: [log-streaming] Escape HTML-specific characters (dionusos via andras.piros)

  • OOZIE-2815: Oozie not always display job log

  • OOZIE-3075: Follow-up on OOZIE-3054: create the lib directory if it doesn't exist (pbacsko)

  • OOZIE-3054: Disable erasure coding for sharelib if Oozie runs on Hadoop 3 (pbacsko)

  • OOZIE-3048: Check El Functions for the coordinator action (satishsaley)

  • OOZIE-3004 : Forked action retry info is not working

  • OOZIE-2691: Show workflow action retry information in UI

  • OOZIE-3009: Number of Oozie tests executed dropped after OOZIE-2854 (andras.piros via asasvari)

  • OOZIE-2854: Oozie should handle transient database problems (andras.piros via gezapeti)

  • OOZIE-3026: fix openjpa enhancer stage during build for logging (dbdist13, andras.piros via pbacsko)

  • OOZIE-2816: Strip out the first command word from Sqoop action if its "sqoop" (harsh)

  • OOZIE-3113: Retry for ZK lock release (satishsaley)

  • OOZIE-3208: "It should never happen" error messages should be more specific to root cause (kmarton via andras.piros)

  • OOZIE-2940: Possible NullPointerException in WorkflowActionBean (dionusos via gezapeti)

  • OOZIE-3297: Retry logic does not handle the exception from BulkJPAExecutor properly (andras.piros)

  • OOZIE-2733: change org.apache.hadoop.fs.permission.AccessControlException to org.apache.hadoop.security.AccessControlException (gezapeti)

  • OOZIE-3021 : Error on job or SLA event listening: WARN logs instead of DEBUG (andras.piros via asasvari)

  • OOZIE-2387: Oozie is Unable to handle Spaces in file/archive tag. (asasvari via gezapeti)

  • OOZIE-2818: Can't overwrite oozie.action.max.output.data on a per-workflow basis (asasvari via pbacsko)

  • OOZIE-2041: Add an admin command to run the PurgeXCommand (abhishekbafna)

  • OOZIE-2875: Typo in ssh action twiki docs (Dongying Jiao via gezapeti)

  • OOZIE-1401: PurgeCommand should purge the workflow jobs w/o end_time (asasvari)

  • OOZIE-3156: Retry SSH action check when cannot connect to remote host (txsing via andras.piros)

  • OOZIE-3183 :Better logging for SshActionExecutor and extended HA capability when calling to remote host (andras.piros)

  • OOZIE-2769: Extend FS action to allow setrep on a file (Artem Ervits via gezapeti)

  • OOZIE-3085: Improve logging in ActionExecutors: add log entry for start(), end() and kill() (kmarton via andras.piros)

HDP 3.0.0 provided Oozie 4.3.1 and the following Apache patches:

  • OOZIE-2802:: Spark action failure on Spark 2.1.0 due to duplicate sharelibs.

  • OOZIE-2658:: --driver-class-path can overwrite the classpath in SparkMain.

  • OOZIE-2787:: Oozie distributes application jar twice making the spark job fail.

  • OOZIE-2606:: Set spark.yarn.jars to fix Spark 2.0 with Oozie.

  • OOZIE-2436:: Fork/join workflow fails with "oozie.action.yarn.tag must not be null".

  • OOZIE-2578:: Oozie example distcp job fails to run within an encrypted zone with checksum match error.

  • OOZIE-2362:: SQL injection in BulkJPAExecutor.

  • OOZIE-2577:: Flaky tests TestCoordActionInputCheckXCommand.testTimeout and testTimeoutWithException.

  • OOZIE-2570:: remove -PtestPatchCompile from patch testing as there is no such profile.

  • OOZIE-2504:: Create a log4j.properties under HADOOP_CONF_DIR in Shell Action.

  • OOZIE-2567:: HCat connection is not closed while getting hcat cred.

  • OOZIE-2547:: Add mapreduce.job.cache.files to spark action.

  • OOZIE-2550:: Flaky tests in TestZKUUIDService.java.

  • OOZIE-2445:: Doc for - Specifying coordinator input datasets in more logical ways.

  • OOZIE-2541:: Possible resource leak in Hive2Credentials.

  • OOZIE-2563:: Pass spark-defaults.conf to spark action.

  • OOZIE-2556:: TestAbandonedCoordChecker.testCatchupJob is flaky.

  • OOZIE-2522:: There can be multiple coord submit from bundle in case of ZK glitch.

  • OOZIE-2553:: Cred tag is required for all actions in the workflow even if an action does not require it.

  • OOZIE-2503:: show ChildJobURLs to spark action.

  • OOZIE-2551:: Feature request: epoch timestamp generation.

  • OOZIE-2542:: Option to disable OpenJPA BrokerImpl finalization.

  • OOZIE-2447:: Illegal character 0x0 oozie client.

  • OOZIE-2548:: Flaky test TestZKLocksService.testLockReaper.

  • OOZIE-2546:: Improperly closed resources in OozieDBCLI.

  • OOZIE-2476:: When one of the action from fork fails with transient error, WF never joins .

  • OOZIE-2475:: Oozie does not cleanup action dir of killed actions.

  • OOZIE-2535:: User can't disable uber mode.

  • OOZIE-2482:: Pyspark job fails with Oozie.

  • OOZIE-2467:: Oozie can shutdown itself on long GC pause.

  • OOZIE-2537:: SqoopMain does not set up log4j properly.

  • OOZIE-2532:: patch apply does not handle binary files.

  • OOZIE-2330:: Spark action should take the global jobTracker and nameNode configs by default and allow file and archive elements.

  • OOZIE-2510:: TestLogStreamer.testStreamLog. fails in other timezones.

  • OOZIE-2531:: Prevent Spark trying for token which is already available.

  • OOZIE-2509:: SLA job status can stuck in running state.

  • OOZIE-2529:: Support adding secret keys to Credentials of Launcher.

  • OOZIE-1402:: Increase retry interval for non-progressing coordinator action with fix value .

  • OOZIE-2512:: ShareLibservice returns incorrect path for jar.

  • OOZIE-2471:: Show child job url tab for distcp.

  • OOZIE-2511:: SubWorkflow missing variable set from option if config-default is present in parent workflow.

  • OOZIE-2391:: spark-opts value in workflow.xml is not parsed properly.

  • OOZIE-2489:: XML parsing is vulnerable.

  • OOZIE-2485:: Oozie client keeps trying to use expired auth token.

  • OOZIE-2490:: Oozie can't set hadoop.security.token.service.use_ip.

  • OOZIE-2474:: <job-xml> is not being applied to the launcher job.

  • OOZIE-2486:: TestSLAEventsGetForFilterJPAExecutor is flakey.

  • OOZIE-2481:: Add YARN_CONF_DIR in the Shell action.

  • OOZIE-2492:: JSON security issue in js code.

  • OOZIE-2429:: TestEventGeneration test is flakey.

  • OOZIE-2466:: Repeated failure of TestMetricsInstrumentation.testSamplers.

  • OOZIE-2470:: Remove infinite socket timeouts in the Oozie email action.

  • OOZIE-2246:: CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist.

  • OOZIE-2446:: Job does not fail during submission if non existent credential is specified .

  • OOZIE-2283:: Documentation should not say that System.exit is not allowed in Java Action .

  • OOZIE-2400:: Workflow xml configuration parser cannot deal with namespace prefix.

  • OOZIE-2452:: Coordinator Functional Specification - EL Constants Typo.

  • OOZIE-2173:: DISCLAIMER.txt still refers to Apache Incubator.

  • OOZIE-2312:: oozie doesn't purge audit and error log.

  • OOZIE-2431:: Support hive.metastore.uris as alias for hcat.metastore.uri.

  • OOZIE-1976:: Specifying coordinator input datasets in more logical ways.

  • OOZIE-2444:: Need conditional logic in bundles.

  • OOZIE-2394:: Oozie can execute command without holding lock.

  • OOZIE-1922:: MemoryLocksService fails if lock is acquired multiple times in same thread and released.

  • OOZIE-2432:: TestPurgeXCommand fails.

  • OOZIE-2434:: inconsistent coord action status and workflow job status.

  • OOZIE-2438:: Oozie client "jobs -filter" diagnostic message clarification.

  • OOZIE-2165:: Job log fetching can fail in Oozie HA mode when using doAs impersonation.

  • OOZIE-2430:: Add root logger for hive,sqoop action.

  • OOZIE-2441:: SubWorkflow action with propagate-configuration but no global section throws NPE on submit.

  • OOZIE-2370:: Docs for Coordinator Action Status Notification has wrong property name.

  • OOZIE-2419:: HBase credentials are not correctly proxied.

  • OOZIE-2439:: FS Action no longer uses name-node from global section or default NN.

  • OOZIE-2435:: TestCoordChangeXCommand is flakey.

  • OOZIE-2428:: TestSLAService, TestSLAEventGeneration flakey tests.

  • OOZIE-2437:: Exclude additional Hadoop dependencies from Spark Sharelib.

  • OOZIE-2030:: Configuration properties from global section is not getting set in Hadoop job conf when using sub-workflow action in Oozie workflow.xml.

  • OOZIE-2380:: Oozie Hive action failed with wrong tmp path.

  • OOZIE-2222:: Oozie UI parent job should be clickable.

  • OOZIE-2407:: AbandonedService should not send mail if there is no abandoned coord.

  • OOZIE-2402:: oozie-setup.sh sharelib create takes a long time on large clusters.

  • OOZIE-2185:: Make oozie cli source conf/oozie-client-env.sh.

  • OOZIE-2413:: Kerberos credentials can expire if the KDC is slow to respond.

  • OOZIE-2411:: Add BCC to oozie email action.

  • OOZIE-2397:: LAST_ONLY and NONE don't properly handle READY actions.

  • OOZIE-2401:: Typo in twiki docs with FilesAchives instead of FilesArchives.

  • OOZIE-2168:: Oozie flow and action names have 50 char limit.

  • OOZIE-2346:: Add sub-workflow information like the super-parent id and workflow depth into the 'oozie.job.info' property.

  • OOZIE-2303:: Typo in documentation.

  • OOZIE-2328:: Coordinator endtime change should check if the last action is in database .

  • OOZIE-2367:: fs delete should support skipTrash option.

  • OOZIE-2368:: coord:dateOffset and coord:dateTzOffset can't be used for coord initial-instance.

  • OOZIE-2369:: coord:offset doesn't resolve correctly.

  • OOZIE-2251:: Expose instrumental matrices in Realtime Graphing tool.

  • OOZIE-2314:: Unable to kill old instance child job by workflow or coord rerun by Launcher .

  • OOZIE-2386:: org.apache.oozie.util.TestXConfiguration.testSubstituteVar is flakey.

  • OOZIE-2385:: org.apache.oozie.TestCoordinatorEngineStreamLog.testCoordLogStreaming is flakey.

  • OOZIE-2382:: org.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID is flakey.

  • OOZIE-2379:: org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand.testRequeueOnException is flakey.

  • OOZIE-2378:: org.apache.oozie.client.TestOozieCLI.testAdminInstrumentation is flakey.

  • OOZIE-2377:: Hive2 Action should not propagate oozie.hive2.* properties to Beeline.

  • OOZIE-2376:: Default action configs not honored if no <configuration> section in workflow.

  • OOZIE-2365:: oozie fail to start when smtp password not set.

  • OOZIE-2360:: Spark Action fails due to missing mesos jar.

  • OOZIE-2345:: Parallel job submission for forked actions.

  • OOZIE-2358:: Coord rerun cleanup should reuse hcat connections.

  • OOZIE-2356:: Add a way to enable/disable credentials in a workflow.

  • OOZIE-2355:: Hive2 Action doesn't pass along oozie configs to jobconf.

  • OOZIE-2318:: Provide better solution for specifying SSL truststore to Oozie Client.

  • OOZIE-2344:: Enabling 'oozie.action.jobinfo.enable' doesn't inject the job information into the map/reduce job's configuration..

  • OOZIE-2350:: Package changes for release.

  • OOZIE-2347:: Remove unnecessary new Configuration./new jobConf. calls from oozie.

  • OOZIE-2348:: Recovery service keeps on recovering coord action of suspended jobs.

  • OOZIE-2277:: Honor oozie.action.sharelib.for.spark in Spark jobs.

  • OOZIE-2322:: Oozie Web UI doesn't work with Kerberos in Internet Explorer 10 or 11 and curl .

  • OOZIE-2343:: Shell Action should take Oozie Action config and setup HADOOP_CONF_DIR.

  • OOZIE-2245:: Service to periodically check database schema.

  • OOZIE-2332:: Add ability to provide Hive and Hive 2 Action queries inline in workflows .

  • OOZIE-2329:: Make handling yarn restarts configurable.

  • OOZIE-2228:: Statustransit service doesn't pick bundle with suspend status.

  • OOZIE-2325:: Shell action fails if user overrides oozie.launcher.mapreduce.map.env.

  • OOZIE-2324:: A syntax error in the kill node causes the workflow to get stuck and other problems.

  • OOZIE-2309:: Enable the coord:dateOffset. function in /coordinator-app/datasets/dataset/@initial-instance.

  • OOZIE-2305:: Compile Oozie with Hive-1.2.0.

  • OOZIE-2320:: TestZKXLogStreamingService.testStreamingWithMultipleOozieServers_coordActionList is failing.

  • OOZIE-2293:: Oozie 4.0.1 build failed while building Catalog.

  • OOZIE-2308:: Add support for bundle:conf. function.

  • OOZIE-2315:: TestOozieCLI.testshareLibUpdate_withSecurity fails with Hadoop 2.

  • OOZIE-2291:: Hive2 workflow.xml.security should have "cred" in action tag instead of "hive2" tag.

  • OOZIE-2289:: hive-jdbc dependency in core/pom.xml should be compile.

  • OOZIE-2300:: TestAuthFilterAuthOozieClient.testClientAuthTokenCache fails with Hadoop 2.7.0 and later.

  • OOZIE-1837:: LauncherMainHadoopUtils sensitive to clock skew.

  • OOZIE-2187:: Add a way to specify a default JT/RM and NN.

  • OOZIE-2272:: Use Hadoop's CredentialProvider for passwords in oozie-site.

  • OOZIE-2287:: Add support for deleting hcat partitions in fs action delete.

  • OOZIE-2285:: Change in concurrency should trigger coord action ready command.

  • OOZIE-2284:: HBaseCredentials should only add hbase-default.xml and hbase-site.xml to actionConf.

  • OOZIE-2286:: Update Log4j and Log4j-extras to latest 1.2.x release.

  • OOZIE-2250:: Show log for WAITING and TIMEDOUT coord actions.

  • OOZIE-2262:: Fix log streaming from other server with start/end filter.

  • OOZIE-2159:: 'oozie validate' command should be moved server-side.

  • OOZIE-2271:: Upgrade Tomcat to 6.0.44.

  • OOZIE-2266:: Fix 'total' actions returned in coordinator job info.

  • OOZIE-2264:: Fix coord:offset. to resolve correct data set.

  • OOZIE-2178:: fix javadoc to compile on JDK8.

  • OOZIE-2268:: Update ActiveMQ version for security and other fixes.

  • OOZIE-2215:: Support glob in FS EL function.

  • OOZIE-2270:: Audit log doesn't get loaded when "audit log" tab is clicked for coord job.

  • OOZIE-2257:: BundleStatusTransitXCommand doesn't update last modified time.

  • OOZIE-2260:: Only set "java.io.tmpdir" to "./tmp" for the AM.

© 2012–2019, Hortonworks, Inc.
Document licensed under the Creative Commons Attribution ShareAlike 4.0 License.
Hortonworks.com | Documentation | Support | Community