Oozie
This release provides Oozie 4.3.1 and the additional Apache patches.
-
OOZIE-3194: [tools] Oozie should set proper permissions to sharelib after upload
-
OOZIE-3439: Hive2 action is not parsing application ID for TEZ from log file properly
-
OOZIE-3440 :[action] Oozie Spark action replaces path symlink # to %23
-
OOZIE-3370: Property filtering is not consistent across job submission
HDP 3.0.1, and HDP 3.1.0 provided Oozie 4.3.1 and the following Apache patches:
-
OOZIE-3304: Parsing sharelib timestamps is not threadsafe (dionusos, matijhs via andras.piros)
-
OOZIE-3135: Configure log4j2 in SqoopMain
-
OOZIE-3072: oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files (gezapeti)
-
OOZIE-2916: Set a job name for the MR Action's child job (asasvari)
-
OOZIE-2494: Cron syntax not handling DST properly (kmarton via andras.piros)
-
OOZIE-2726: Flaky test due to daylight saving changes (sasishsaley, andras.piros via gezapeti)
-
OOZIE-3173 : Coordinator job with frequency using cron syntax creates only one action in catchup mode (andras.piros)
-
OOZIE-2691: Show workflow action retry information in UI
-
OOZIE-3109: [log-streaming] Escape HTML-specific characters (dionusos via andras.piros)
-
OOZIE-2815: Oozie not always display job log
-
OOZIE-3075: Follow-up on OOZIE-3054: create the lib directory if it doesn't exist (pbacsko)
-
OOZIE-3054: Disable erasure coding for sharelib if Oozie runs on Hadoop 3 (pbacsko)
-
OOZIE-3048: Check El Functions for the coordinator action (satishsaley)
-
OOZIE-3004 : Forked action retry info is not working
-
OOZIE-2691: Show workflow action retry information in UI
-
OOZIE-3009: Number of Oozie tests executed dropped after OOZIE-2854 (andras.piros via asasvari)
-
OOZIE-2854: Oozie should handle transient database problems (andras.piros via gezapeti)
-
OOZIE-3026: fix openjpa enhancer stage during build for logging (dbdist13, andras.piros via pbacsko)
-
OOZIE-2816: Strip out the first command word from Sqoop action if its "sqoop" (harsh)
-
OOZIE-3113: Retry for ZK lock release (satishsaley)
-
OOZIE-3208: "It should never happen" error messages should be more specific to root cause (kmarton via andras.piros)
-
OOZIE-2940: Possible NullPointerException in WorkflowActionBean (dionusos via gezapeti)
-
OOZIE-3297: Retry logic does not handle the exception from BulkJPAExecutor properly (andras.piros)
-
OOZIE-2733: change org.apache.hadoop.fs.permission.AccessControlException to org.apache.hadoop.security.AccessControlException (gezapeti)
-
OOZIE-3021 : Error on job or SLA event listening: WARN logs instead of DEBUG (andras.piros via asasvari)
-
OOZIE-2387: Oozie is Unable to handle Spaces in file/archive tag. (asasvari via gezapeti)
-
OOZIE-2818: Can't overwrite oozie.action.max.output.data on a per-workflow basis (asasvari via pbacsko)
-
OOZIE-2041: Add an admin command to run the PurgeXCommand (abhishekbafna)
-
OOZIE-2875: Typo in ssh action twiki docs (Dongying Jiao via gezapeti)
-
OOZIE-1401: PurgeCommand should purge the workflow jobs w/o end_time (asasvari)
-
OOZIE-3156: Retry SSH action check when cannot connect to remote host (txsing via andras.piros)
-
OOZIE-3183 :Better logging for SshActionExecutor and extended HA capability when calling to remote host (andras.piros)
-
OOZIE-2769: Extend FS action to allow setrep on a file (Artem Ervits via gezapeti)
-
OOZIE-3085: Improve logging in ActionExecutors: add log entry for start(), end() and kill() (kmarton via andras.piros)
HDP 3.0.0 provided Oozie 4.3.1 and the following Apache patches:
-
OOZIE-2802:: Spark action failure on Spark 2.1.0 due to duplicate sharelibs.
-
OOZIE-2658:: --driver-class-path can overwrite the classpath in SparkMain.
-
OOZIE-2787:: Oozie distributes application jar twice making the spark job fail.
-
OOZIE-2606:: Set spark.yarn.jars to fix Spark 2.0 with Oozie.
-
OOZIE-2436:: Fork/join workflow fails with "oozie.action.yarn.tag must not be null".
-
OOZIE-2578:: Oozie example distcp job fails to run within an encrypted zone with checksum match error.
-
OOZIE-2362:: SQL injection in BulkJPAExecutor.
-
OOZIE-2577:: Flaky tests TestCoordActionInputCheckXCommand.testTimeout and testTimeoutWithException.
-
OOZIE-2570:: remove -PtestPatchCompile from patch testing as there is no such profile.
-
OOZIE-2504:: Create a log4j.properties under HADOOP_CONF_DIR in Shell Action.
-
OOZIE-2567:: HCat connection is not closed while getting hcat cred.
-
OOZIE-2547:: Add mapreduce.job.cache.files to spark action.
-
OOZIE-2550:: Flaky tests in TestZKUUIDService.java.
-
OOZIE-2445:: Doc for - Specifying coordinator input datasets in more logical ways.
-
OOZIE-2541:: Possible resource leak in Hive2Credentials.
-
OOZIE-2563:: Pass spark-defaults.conf to spark action.
-
OOZIE-2556:: TestAbandonedCoordChecker.testCatchupJob is flaky.
-
OOZIE-2522:: There can be multiple coord submit from bundle in case of ZK glitch.
-
OOZIE-2553:: Cred tag is required for all actions in the workflow even if an action does not require it.
-
OOZIE-2503:: show ChildJobURLs to spark action.
-
OOZIE-2551:: Feature request: epoch timestamp generation.
-
OOZIE-2542:: Option to disable OpenJPA BrokerImpl finalization.
-
OOZIE-2447:: Illegal character 0x0 oozie client.
-
OOZIE-2548:: Flaky test TestZKLocksService.testLockReaper.
-
OOZIE-2546:: Improperly closed resources in OozieDBCLI.
-
OOZIE-2476:: When one of the action from fork fails with transient error, WF never joins .
-
OOZIE-2475:: Oozie does not cleanup action dir of killed actions.
-
OOZIE-2535:: User can't disable uber mode.
-
OOZIE-2482:: Pyspark job fails with Oozie.
-
OOZIE-2467:: Oozie can shutdown itself on long GC pause.
-
OOZIE-2537:: SqoopMain does not set up log4j properly.
-
OOZIE-2532:: patch apply does not handle binary files.
-
OOZIE-2330:: Spark action should take the global jobTracker and nameNode configs by default and allow file and archive elements.
-
OOZIE-2510:: TestLogStreamer.testStreamLog. fails in other timezones.
-
OOZIE-2531:: Prevent Spark trying for token which is already available.
-
OOZIE-2509:: SLA job status can stuck in running state.
-
OOZIE-2529:: Support adding secret keys to Credentials of Launcher.
-
OOZIE-1402:: Increase retry interval for non-progressing coordinator action with fix value .
-
OOZIE-2512:: ShareLibservice returns incorrect path for jar.
-
OOZIE-2471:: Show child job url tab for distcp.
-
OOZIE-2511:: SubWorkflow missing variable set from option if config-default is present in parent workflow.
-
OOZIE-2391:: spark-opts value in workflow.xml is not parsed properly.
-
OOZIE-2489:: XML parsing is vulnerable.
-
OOZIE-2485:: Oozie client keeps trying to use expired auth token.
-
OOZIE-2490:: Oozie can't set hadoop.security.token.service.use_ip.
-
OOZIE-2474:: <job-xml> is not being applied to the launcher job.
-
OOZIE-2486:: TestSLAEventsGetForFilterJPAExecutor is flakey.
-
OOZIE-2481:: Add YARN_CONF_DIR in the Shell action.
-
OOZIE-2492:: JSON security issue in js code.
-
OOZIE-2429:: TestEventGeneration test is flakey.
-
OOZIE-2466:: Repeated failure of TestMetricsInstrumentation.testSamplers.
-
OOZIE-2470:: Remove infinite socket timeouts in the Oozie email action.
-
OOZIE-2246:: CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist.
-
OOZIE-2446:: Job does not fail during submission if non existent credential is specified .
-
OOZIE-2283:: Documentation should not say that System.exit is not allowed in Java Action .
-
OOZIE-2400:: Workflow xml configuration parser cannot deal with namespace prefix.
-
OOZIE-2452:: Coordinator Functional Specification - EL Constants Typo.
-
OOZIE-2173:: DISCLAIMER.txt still refers to Apache Incubator.
-
OOZIE-2312:: oozie doesn't purge audit and error log.
-
OOZIE-2431:: Support hive.metastore.uris as alias for hcat.metastore.uri.
-
OOZIE-1976:: Specifying coordinator input datasets in more logical ways.
-
OOZIE-2444:: Need conditional logic in bundles.
-
OOZIE-2394:: Oozie can execute command without holding lock.
-
OOZIE-1922:: MemoryLocksService fails if lock is acquired multiple times in same thread and released.
-
OOZIE-2432:: TestPurgeXCommand fails.
-
OOZIE-2434:: inconsistent coord action status and workflow job status.
-
OOZIE-2438:: Oozie client "jobs -filter" diagnostic message clarification.
-
OOZIE-2165:: Job log fetching can fail in Oozie HA mode when using doAs impersonation.
-
OOZIE-2430:: Add root logger for hive,sqoop action.
-
OOZIE-2441:: SubWorkflow action with propagate-configuration but no global section throws NPE on submit.
-
OOZIE-2370:: Docs for Coordinator Action Status Notification has wrong property name.
-
OOZIE-2419:: HBase credentials are not correctly proxied.
-
OOZIE-2439:: FS Action no longer uses name-node from global section or default NN.
-
OOZIE-2435:: TestCoordChangeXCommand is flakey.
-
OOZIE-2428:: TestSLAService, TestSLAEventGeneration flakey tests.
-
OOZIE-2437:: Exclude additional Hadoop dependencies from Spark Sharelib.
-
OOZIE-2030:: Configuration properties from global section is not getting set in Hadoop job conf when using sub-workflow action in Oozie workflow.xml.
-
OOZIE-2380:: Oozie Hive action failed with wrong tmp path.
-
OOZIE-2222:: Oozie UI parent job should be clickable.
-
OOZIE-2407:: AbandonedService should not send mail if there is no abandoned coord.
-
OOZIE-2402:: oozie-setup.sh sharelib create takes a long time on large clusters.
-
OOZIE-2185:: Make oozie cli source conf/oozie-client-env.sh.
-
OOZIE-2413:: Kerberos credentials can expire if the KDC is slow to respond.
-
OOZIE-2411:: Add BCC to oozie email action.
-
OOZIE-2397:: LAST_ONLY and NONE don't properly handle READY actions.
-
OOZIE-2401:: Typo in twiki docs with FilesAchives instead of FilesArchives.
-
OOZIE-2168:: Oozie flow and action names have 50 char limit.
-
OOZIE-2346:: Add sub-workflow information like the super-parent id and workflow depth into the 'oozie.job.info' property.
-
OOZIE-2303:: Typo in documentation.
-
OOZIE-2328:: Coordinator endtime change should check if the last action is in database .
-
OOZIE-2367:: fs delete should support skipTrash option.
-
OOZIE-2368:: coord:dateOffset and coord:dateTzOffset can't be used for coord initial-instance.
-
OOZIE-2369:: coord:offset doesn't resolve correctly.
-
OOZIE-2251:: Expose instrumental matrices in Realtime Graphing tool.
-
OOZIE-2314:: Unable to kill old instance child job by workflow or coord rerun by Launcher .
-
OOZIE-2386:: org.apache.oozie.util.TestXConfiguration.testSubstituteVar is flakey.
-
OOZIE-2385:: org.apache.oozie.TestCoordinatorEngineStreamLog.testCoordLogStreaming is flakey.
-
OOZIE-2382:: org.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID is flakey.
-
OOZIE-2379:: org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand.testRequeueOnException is flakey.
-
OOZIE-2378:: org.apache.oozie.client.TestOozieCLI.testAdminInstrumentation is flakey.
-
OOZIE-2377:: Hive2 Action should not propagate oozie.hive2.* properties to Beeline.
-
OOZIE-2376:: Default action configs not honored if no <configuration> section in workflow.
-
OOZIE-2365:: oozie fail to start when smtp password not set.
-
OOZIE-2360:: Spark Action fails due to missing mesos jar.
-
OOZIE-2345:: Parallel job submission for forked actions.
-
OOZIE-2358:: Coord rerun cleanup should reuse hcat connections.
-
OOZIE-2356:: Add a way to enable/disable credentials in a workflow.
-
OOZIE-2355:: Hive2 Action doesn't pass along oozie configs to jobconf.
-
OOZIE-2318:: Provide better solution for specifying SSL truststore to Oozie Client.
-
OOZIE-2344:: Enabling 'oozie.action.jobinfo.enable' doesn't inject the job information into the map/reduce job's configuration..
-
OOZIE-2350:: Package changes for release.
-
OOZIE-2347:: Remove unnecessary new Configuration./new jobConf. calls from oozie.
-
OOZIE-2348:: Recovery service keeps on recovering coord action of suspended jobs.
-
OOZIE-2277:: Honor oozie.action.sharelib.for.spark in Spark jobs.
-
OOZIE-2322:: Oozie Web UI doesn't work with Kerberos in Internet Explorer 10 or 11 and curl .
-
OOZIE-2343:: Shell Action should take Oozie Action config and setup HADOOP_CONF_DIR.
-
OOZIE-2245:: Service to periodically check database schema.
-
OOZIE-2332:: Add ability to provide Hive and Hive 2 Action queries inline in workflows .
-
OOZIE-2329:: Make handling yarn restarts configurable.
-
OOZIE-2228:: Statustransit service doesn't pick bundle with suspend status.
-
OOZIE-2325:: Shell action fails if user overrides oozie.launcher.mapreduce.map.env.
-
OOZIE-2324:: A syntax error in the kill node causes the workflow to get stuck and other problems.
-
OOZIE-2309:: Enable the coord:dateOffset. function in /coordinator-app/datasets/dataset/@initial-instance.
-
OOZIE-2305:: Compile Oozie with Hive-1.2.0.
-
OOZIE-2320:: TestZKXLogStreamingService.testStreamingWithMultipleOozieServers_coordActionList is failing.
-
OOZIE-2293:: Oozie 4.0.1 build failed while building Catalog.
-
OOZIE-2308:: Add support for bundle:conf. function.
-
OOZIE-2315:: TestOozieCLI.testshareLibUpdate_withSecurity fails with Hadoop 2.
-
OOZIE-2291:: Hive2 workflow.xml.security should have "cred" in action tag instead of "hive2" tag.
-
OOZIE-2289:: hive-jdbc dependency in core/pom.xml should be compile.
-
OOZIE-2300:: TestAuthFilterAuthOozieClient.testClientAuthTokenCache fails with Hadoop 2.7.0 and later.
-
OOZIE-1837:: LauncherMainHadoopUtils sensitive to clock skew.
-
OOZIE-2187:: Add a way to specify a default JT/RM and NN.
-
OOZIE-2272:: Use Hadoop's CredentialProvider for passwords in oozie-site.
-
OOZIE-2287:: Add support for deleting hcat partitions in fs action delete.
-
OOZIE-2285:: Change in concurrency should trigger coord action ready command.
-
OOZIE-2284:: HBaseCredentials should only add hbase-default.xml and hbase-site.xml to actionConf.
-
OOZIE-2286:: Update Log4j and Log4j-extras to latest 1.2.x release.
-
OOZIE-2250:: Show log for WAITING and TIMEDOUT coord actions.
-
OOZIE-2262:: Fix log streaming from other server with start/end filter.
-
OOZIE-2159:: 'oozie validate' command should be moved server-side.
-
OOZIE-2271:: Upgrade Tomcat to 6.0.44.
-
OOZIE-2266:: Fix 'total' actions returned in coordinator job info.
-
OOZIE-2264:: Fix coord:offset. to resolve correct data set.
-
OOZIE-2178:: fix javadoc to compile on JDK8.
-
OOZIE-2268:: Update ActiveMQ version for security and other fixes.
-
OOZIE-2215:: Support glob in FS EL function.
-
OOZIE-2270:: Audit log doesn't get loaded when "audit log" tab is clicked for coord job.
-
OOZIE-2257:: BundleStatusTransitXCommand doesn't update last modified time.
-
OOZIE-2260:: Only set "java.io.tmpdir" to "./tmp" for the AM.