Tez 0.5.2
HDP 2.2.9 provides Apache Tez 0.5.2 and the following Apache patches:
HDP 2.2.8 provides Apache Tez 0.5.2 and the following Apache patches:
TEZ-1521: VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log.
TEZ-1560: Invalid state machine handling for V_SOURCE_VERTEX_RECOVERED in recovery.
TEZ-1642: TestAMRecovery sometimes fails.
TEZ-1687: Use logIdentifier of Vertex for logging.
TEZ-1697: DAG submission fails if a local resource added is already part of tez.lib.uris.
TEZ-1737: Should add taskNum in VertexFinishedEvent.
TEZ-1772: Failing tests post TEZ-1737.
TEZ-1773: Add attempt failure cause enum to the attempt failed/killed history record.
TEZ-1796: Use of DeprecationDelta broke build against 2.2 Hadoop.
TEZ-1808: Job can fail since name of intermediate files can be too long in specific situation.
TEZ-1816: It is possible to receive START event when DAG is failed.
TEZ-1832: TestSecureShuffle fails with NoClassDefFoundError: org/bouncycastle/x509/X509V1CertificateGenerator.
TEZ-1909: Remove need to copy over all events from attempt 1 to attempt 2 dir.
TEZ-1929: pre-empted tasks should be marked as killed instead of failed.
TEZ-1961: Remove misleading exception "No running dag" from AM logs.
TEZ-2000: Source vertex exists error during DAG submission.
TEZ-2011: InputReadyVertexManager not resilient to updates in parallelism.
TEZ-2033: Update TestOrderedWordCount to add processor configs as history text and use MR configs correctly.
TEZ-2036: OneToOneEdgeManager should enforce that source and destination tasks have same number.
TEZ-2064: SessionNotRunning Exception not thrown is all cases.
TEZ-2133: Secured Impersonation: Failed to delete tez scratch data dir.
TEZ-2162: org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat is not recognized.
TEZ-2179: Timeline related entries missing cause exaggerated warning.
TEZ-2186: TEZ-2186 OOM with a simple scatter gather job with re-use.
TEZ-2189: Tez UI live AM tracking URL only works for localhost addresses.
TEZ-2193: Check returned value from EdgeManagerPlugin before using it.
TEZ-2195: TestTezJobs :testInvalidQueueSubmission/testInvalidQueueSubmissionToSession fail with hadoop branch-2.
TEZ-2199: updateLocalResourcesForInputSplits assumes wrongly that split data is on same FS as the default FS.
TEZ-2203: Intern strings in tez counters.
TEZ-2205: Tez still tries to post to ATS when yarn.timeline-service.enabled=false.
TEZ-2221: VertexGroup name should be unique.
TEZ-2224: EventQueue empty doesn't mean events are consumed in RecoveryService.
TEZ-2257: Fix potential NPEs in TaskReporter.
TEZ-2289: ATSHistoryLoggingService can generate ArrayOutOfBoundsException.
TEZ-2290: Scale memory for Default Sorter down to a max of 2047 MB if configured higher.
TEZ-2293: When running in "mr" mode, always use MR config settings.
TEZ-2299: Invalid dag creation in MRRSleepJob post TEZ-2293.
TEZ-2303: ConcurrentModificationException while processing recovery.
TEZ-2304: InvalidStateTransitonException TA_SCHEDULE at START_WAIT during recovery.
TEZ-2305: MR compatibility sleep job fails with IOException: Undefined job output-path.
TEZ-2311: AM can hang if kill received while recovering from previous attempt.
TEZ-2317: Event processing backlog can result in task failures for short tasks.
TEZ-2323: Fix TestOrderedWordcount to use MR memory configs.
TEZ-2379: org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: T_ATTEMPT_KILLED at KILLED.
TEZ-2383: Cleanup input/output/processor contexts in LogicalIOProcessorRuntimeTask.
TEZ-2398: Flaky test: TestFaultTolerance.
TEZ-2436: Tez UI: Add cancel button in column selector.
TEZ-2460: Temporary solution for issue due to YARN-2560.
TEZ-2474: The old taskNum is logged incorrectly when parallelism is changed.
TEZ-2483: Tez should close task if processor fail.
TEZ-2488: Tez AM crashes if a submitted DAG is configured to use invalid resource sizes.
TEZ-2509: YarnTaskSchedulerService should not try to allocate containers if AM is shutting down.
TEZ-2511: Add exitCode to diagnostics when container fails.
TEZ-2533: AM deadlock when shutdown.
TEZ-2534: Error handling summary event when shutting down AM.
TEZ-2538: ADDITIONAL_SPILL_COUNT wrongly populated for DefaultSorter with multiple partitions.
TEZ-2548: TezClient submitDAG can hang if the AM is in the process of shutting down.
TEZ-2552: CRC errors can cause job to run for very long time in large jobs.
TEZ-2561: Port for TaskAttemptListenerImpTezDag should be configurable.
TEZ-2566: Allow TaskAttemptFinishedEvent without TaskAttemptStartedEvent when it is KILLED/FAILED.
TEZ-2629: LimitExceededException in Tez client when DAG has exceeds the default max counters.
TEZ-2635: Limit number of attempts being downloaded in unordered fetch.
TEZ-2636: MRInput and MultiMRInput should work for cases when there are 0 physical inputs.
TEZ-2662: Provide a way to check whether AM or task opts are valid and error if not.
TEZ-2663: SessionNotRunning exceptions are wrapped in a ServiceException from a dying AM.
TEZ-2687: Tez should release/kill all held containers before stopping services during the shutdown phase.
TEZ-2716: DefaultSorter.isRleNeeded not thread safe.
TEZ-2719: Consider reducing logs in unordered fetcher with shared-fetch option.
TEZ-2731: Fix Tez GenericCounter performance bottleneck.
TEZ-2732: DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort buffers.
TEZ-2734: Add a test to verify the filename generated by OnDiskMerge.
TEZ-2742: VertexImpl.finished() terminationCause hides member var of the same name.
TEZ-2744: Improvements and cleanup of logging for the AM and parts of the runtime.
TEZ-2745: ClassNotFoundException of user code should fail dag.
TEZ-2752: logUnsuccessful completion in Attempt should write original finish time to ATS.
TEZ-2758: Remove append API in RecoveryService after TEZ-1909.
TEZ-2761: Tez UI: update the progress on the dag and vertices pages with info from AM.
TEZ-2767: Make TezMxBeanResourceCalculator the default resource calculator.
TEZ-2768: Log a useful error message when the summary stream cannot be closed when shutting down an AM.
TEZ-2775: Improve and consolidate logging in Runtime components.
TEZ-2781: Fallback to send only TaskAttemptFailedEvent if taskFailed heartbeat fails.
TEZ-2787: Tez AM should have java.io.tmpdir=./tmp to be consistent with tasks.
TEZ-2792: Add AM web service API for tasks.
TEZ-2808: Race condition between preemption and container assignment.
TEZ-2813: Tez UI: add counter data for rest api calls to AM Web Services v2.
TEZ-2816: Preemption sometimes does not respect heartbeats between pre-emptions.
TEZ-2825: Report progress in terms of completed tasks to reduce load on AM for Tez UI.
TEZ-2829: Tez UI: minor fixes to in-progress update of UI from AM.
TEZ-2834: Make Tez preemption resilient to incorrect free resource reported by YARN.
TEZ-2850: Tez MergeManager OOM for small Map Outputs.
TEZ-2851: Support a way for upstream applications to pass in a caller context to Tez.
TEZ-2855: Fix a potential NPE while routing VertexManager events.
TEZ-2857: Fix flakey tests in TestDAGImpl.
TEZ-2858: Stop using System.currentTimeMillis in TestInputReadyTracker.
TEZ-2860: NPE in DAGClientImpl.
TEZ-2868: Fix setting Caller Context in Tez Examples.
TEZ-2874: Improved logging for caller context.
TEZ-2882: Consider improving fetch failure handling.
TEZ-2885: Remove counter logs from AMWebController.
TEZ-2887: Tez build failure due to missing dependency in pom files.
TEZ-2896: Fix thread names used during Input/Output initialization.
TEZ-2907: NPE in IFile.Reader.getLength during final merge operation.
TEZ-814: Improve heuristic for determining a task has failed outputs.
TEZ-900: Confusing message for incorrect queue for some tez examples.
HDP 2.2.6 provided Apache Tez 0.5.2 and the following Apache patches:
TEZ-1642: TestAMRecovery sometimes fails.
TEZ-1775: Allow setting log level per logger.
TEZ-1800: Integer overflow in ExternalSorter.getInitialMemoryRequirement()
TEZ-1836: Provide better error messages when tez.runtime.io.sort.mb, spill percentage is incorrectly configured.
TEZ-1851: FileSystem counters do not differentiate between different FileSystems
TEZ-1852: Get examples to work in Local Mode.
TEZ-1861: Fix failing test: TestOnFileSortedOutput.
TEZ-1878: Task-specific log level override not working in certain conditions
TEZ-1924: Tez AM does not register with AM with full FQDN causing jobs to fail in some environments.
TEZ-1931: Publish tez version info to Timeline.
TEZ-1934: TestAMRecovery may fail due to the execution order is not determined.
TEZ-1942: Number of tasks show in Tez UI with auto-reduce parallelism is misleading.
TEZ-1949: Whitelist TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges
TEZ-1962: Fix a thread leak in LocalMode.
TEZ-2024: TaskFinishedEvent may not be logged in recovery.
TEZ-2037: Should log TaskAttemptFinishedEvent if taskattempt is recovered to KILLED
TEZ-2135: ACL checks handled incorrectly in AMWebController.
As part of HDP 2.2.6 Hortonworks is providing a Tez Debugging User Interface. This interface does not impact the behavior or function of jobs that leverage Tez, and its use is optional. Patches added to facilitate the Tez Debugging User Interface include:
TEZ-1990: Tez UI: DAG details page shows Nan for end time when a DAG is running.
TEZ-2031: Tez UI: horizontal scrollbars do not appear in tables, causing them to look truncated.
TEZ-2038: TEZ-UI DAG is always running in Tez-UI when the app is failed but no DAGFinishedEvent is logged.
TEZ-2043: Tez UI: add progress info from am webservice to dag and vertex views.
TEZ-2052: Tez UI: log view fixes, show version from build, better handling of ATS URL config.
TEZ-2056: Tez UI: fix VertexID filter,show only tez configs by default,fix app attempt ID.
TEZ-2063: Tez UI: Flaky log URL in tasks table.
TEZ-2065: Setting up tez.tez-ui.history-url.base with a trailing slash can result in failures to redirect correctly.
TEZ-2068: Tez UI: Dag view should use full window height, disable web UI service in local mode.
TEZ-2069: Tez UI: appId should link to application in dag details view.
TEZ-2077: Tez UI: No diagnostics on Task Attempt Details page if task attempt failed
TEZ-2078: Tez UI: Task logs URL use in-progress URL causing various errors.
TEZ-2079: Tez UI: trailing slash in timelineBaseUrl in UI should be handled.
TEZ-2092: Tez UI history URL handler injects spurious trailing slash.
TEZ-2098: Tez UI: Dag details should be the default page for dag, fix invalid time entries for failed Vertices.
TEZ-2101: Tez UI: Issues on displaying a table.
TEZ-2102: Tez UI: DAG view has hidden edges, dragging DAG by holding vertex causes unintended click.
TEZ-2106: TEZ UI: Display data load time, and add a refresh button for items that can be refreshed.
TEZ-2112: Tez UI: fix offset calculation, add home button to breadcrumbs.
TEZ-2114: Tez UI: task/task attempt status is not available when its running.
TEZ-2116: Tez UI: dags page filter does not work if more than one filter is specified.
TEZ-2134: TEZ UI: On request failure, display request URL and server name in error bar.
TEZ-2136: Some enhancements to the new Tez UI.
TEZ-2142: TEZ UI: Breadcrumb border color looks out of place in wrapped mode.
TEZ-2158: TEZ UI: Display dag/vertex names, and task/attempt index in breadcrumb
TEZ-2160: Tez UI: App tracking URL should support navigation back.
TEZ-2165: Tez UI: DAG shows running status if killed by RM in some cases.