HDP-2.4.2 Release Notes
Also available as:
PDF

Hadoop

HDP 2.4.2 provides the following Apache patches:

  • HADOOP-10365: BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block.

  • HADOOP-11212: NetUtils.wrapException to handle SocketException explicitly.

  • HADOOP-12100: ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermission.

  • HADOOP-12103: Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing.

  • HADOOP-12107: Long running apps may have a huge number of StatisticsData instances under FileSystem.

  • HADOOP-12161: Add getStoragePolicy API to the FileSystem interface.

  • HADOOP-12191: Bzip2Factory is not thread safe.

  • HADOOP-12213: Interrupted exception can occur when Client#stop is called.

  • HADOOP-12348: MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.

  • HADOOP-12374: Description of HDFS expunge command is confusing.

  • HADOOP-12426: Add Entry point for Kerberos health check.

  • HADOOP-12464: Interrupted client may try to fail-over and retry Interrupted client may try to fail-over and retry.

  • HADOOP-12482: Race condition in JMX cache update.

  • HADOOP-12589: Fix intermittent test failure of TestCopyPreserveFlag.

  • HADOOP-12609: Fix intermittent failure of TestDecayRpcScheduler.

  • HADOOP-12699: TestKMS#testKMSProvider intermittently fails during 'test rollover draining'.

  • HADOOP-12706: TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally.

  • HADOOP-12752: Improve diagnostics/use of envvar/sysprop credential propagation.

  • HADOOP-12787: KMS SPNEGO sequence does not work with WEBHDFS.

  • HADOOP-12795: KMS does not log detailed stack trace for unexpected errors.

  • HADOOP-12825: Log slow name resolutions.

  • HADOOP-12829: StatisticsDataReferenceCleaner swallows interrupt exceptions.

  • HADOOP-12851: S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders.

  • HADOOP-12903: IPC Server should allow suppressing exception logging by type, not log 'server too busy' messages.

  • HADOOP-12958: PhantomReference for filesystem statistics can trigger OOM.

  • HADOOP-13026: Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator.

  • HDFS-10199: Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing.

  • HDFS-10270: TestJMXGet:testNameNode() fails.

  • HDFS-10281: TestPendingCorruptDnMessages fails intermittently.

  • HDFS-10283: o.a.h.hdfs.server.namenode.TestFSImageWithSnapshot#testSaveLoadImageWithAppending fails intermittently.

  • HDFS-6101: TestReplaceDatanodeOnFailure fails occasionally.

  • HDFS-8113: Add check for null BlockCollection pointers in BlockInfoContiguous structures.

  • HDFS-8337: Accessing HttpFS via WebHDFS doesn't work from a jar with Kerberos.

  • HDFS-8647: Abstract BlockManager's rack policy into BlockPlacementPolicy.

  • HDFS-8659: Block scanner INFO message is spamming logs.

  • HDFS-8676: Delayed rolling upgrade finalization can cause heartbeat expiration.

  • HDFS-8772: Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails.

  • HDFS-8806: Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared.

  • HDFS-8815: DFS getStoragePolicy implementation using single RPC call.

  • HDFS-8891: HDFS concat should keep srcs order.

  • HDFS-9072: Fix random failures in TestJMXGet.

  • HDFS-9130: Use GenericTestUtils#setLogLevel to the logging level.

  • HDFS-9221: HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array.

  • HDFS-9239: DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness.

  • HDFS-9289: Make DataStreamer#block thread safe and verify genStamp in commitBlock.

  • HDFS-9290: DFSClient#callAppend() is not backward compatible for slightly older NameNodes.

  • HDFS-9313: Possible NullPointerException in BlockManager if no excess replica can be chosen.

  • HDFS-9314: Improve BlockPlacementPolicyDefault's picking of excess replicas.

  • HDFS-9347: Invariant assumption in TestQuorumJournalManager.shutdown() is wrong.

  • HDFS-9383: TestByteArrayManager#testByteArrayManager fails.

  • HDFS-9402: Switch DataNode.LOG to use slf4j.

  • HDFS-9431: DistributedFileSystem#concat fails if the target path is relative.

  • HDFS-9434: Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messages.

  • HDFS-9445: Datanode may deadlock while handling a bad volume.

  • HDFS-9478: Reason for failing ipc.FairCallQueue construction should be thrown.

  • HDFS-9534: Add CLI command to clear storage policy from a path.

  • HDFS-9557: Reduce object allocation in PB conversion.

  • HDFS-9572: Prevent DataNode log spam if a client connects on the data transfer port but sends no data.

  • HDFS-9574: Reduce client failures during datanode restart.

  • HDFS-9600: Do not check replication if the block is under construction.

  • HDFS-9625: Set replication for empty file failed when set storage policy.

  • HDFS-9655: NN should start JVM pause monitor before loading fsimage.

  • HDFS-9661: Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage and createRbw.

  • HDFS-9710: DN can be configured to send block receipt IBRs in batches.

  • HDFS-9724: Degraded performance in WebHDFS listing as it does not reuse ObjectMapper.

  • HDFS-9726: Refactor IBR code to a new class.

  • HDFS-9740: Use a reasonable limit in DFSTestUtil.waitForMetric().

  • HDFS-9743: Fix TestLazyPersistFiles#testFallbackToDiskFull in branch-2.7.

  • HDFS-9752: Permanent write failures may happen to slow writers during datanode rolling upgrades.

  • HDFS-9768: Reuse ObjectMapper instance in HDFS to improve the performance.

  • HDFS-9790: HDFS Balancer should exit with a proper message if upgrade is not finalized.

  • HDFS-9839: Reduce verbosity of processReport logging.

  • HDFS-9851: NameNode throws NPE when setPermission is called on a path that does not exist.

  • HDFS-9854: Log cipher suite negotiation more verbosely.

  • HDFS-9906: Remove spammy log spew when a datanode is restarted.

  • HDFS-9941: Do not log StandbyException on NN, other minor logging fixes.

  • MAPREDUCE-6436: JobHistory cache issue.

  • MAPREDUCE-6460: TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails.

  • MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.

  • MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.

  • MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.

  • MAPREDUCE-6670: TestJobListCache#testEviction sometimes fails on Windows with timeout.

  • MAPREDUCE-6680: JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS.

  • YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.

  • YARN-2871: TestRMRestart#testRMRestartGetApplicationList sometimes fails in trunk.

  • YARN-3102: Decommissioned Nodes not listed in Web UI.

  • YARN-3695: ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.

  • YARN-3769: Consider user limit when calculating total pending resource for preemption policy in Capacity Scheduler.

  • YARN-4155: TestLogAggregationService.testLogAggregationServiceWithInterval failing.

  • YARN-4365: FileSystemNodeLabelStore should check for root dir existence on startup.

  • YARN-4414: Nodemanager connection errors are retried at multiple levels.

  • YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.

  • YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.

  • YARN-4439: Clarify NMContainerStatus#toString method.

  • YARN-4546: ResourceManager crash due to scheduling opportunity overflow.

  • YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.

  • YARN-4610: Reservations continue looking for one app causes other apps to starve.

  • YARN-4623: TestSystemMetricsPublisher#testPublishAppAttemptMetricsForUnmanagedAM fails with NPE on branch-2.7.

  • YARN-4633: Fix random test failure in TestRMRestart#testRMRestartAfterPreemption.

  • YARN-4680: Fix TimerTasks leak in Application Timeline Server (ATS) v1.5 Writer.

  • YARN-4696: TimelineClient to add flush operation for deterministic writes (including testing) and Changes to EntityGroupFSTimelineStore for testability.

  • YARN-4709: NMWebServices produces incorrect JSON for containers.

  • YARN-4723: NodesListManager$UnknownNodeId ClassCastException.

  • YARN-4737: Add CSRF filter support in YARN.

  • YARN-4769: Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI.

  • YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.

  • YARN-4814: ATS 1.5 timelineclient impl call flush after every event write.

  • YARN-4815: ATS 1.5 timelineclinet impl try to create attempt directory for every event call.

  • YARN-4817: TimelineClient ATSv1.5 logging is very noisy.

  • YARN-4916: TestNMProxy.tesNMProxyRPCRetry fails.

  • YARN-4928: Some yarn.server.timeline.* tests fail on Windows attempting to use a test root path containing a colon.

  • YARN-4954: TestYarnClient.testReservationAPIs fails on machines with less than 4 GB available memory.

  • YARN-4955: Add retry for SocketTimeoutException in TimelineClient.

  • YARN-4965: Distributed shell AM failed due to ClientHandlerException thrown by jersey.

  • YARN-4968: Fix two scheduler related UTs in YARN.

HDP 2.4.0 provided the following Apache patches:

  • HADOOP-10406: TestIPC.testIpcWithReaderQueuing may fail.

  • HADOOP-12551: Introduce FileNotFoundException for WASB FileSystem API.

  • HADOOP-12608: Fix exception message in WASB when connecting with anonymous credential.

  • HADOOP-12678: Handle empty rename pending metadata file during atomic rename in redo path.

  • HDFS-8729: Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately which occasionally failed.

  • HDFS-9358: TestNodeCount#testNodeCount timed out.

  • HDFS-9406: FSImage may get corrupted after deleting snapshot.

  • HDFS-9672: o.a.h.hdfs.TestLeaseRecovery2 fails intermittently.

  • MAPREDUCE-6566: Add retry support to mapreduce CLI tool.

  • MAPREDUCE-6618: YarnClientProtocolProvider leaking the YarnClient thread.

  • MAPREDUCE-6621: Memory Link in JobClient#submitJobInternal().

  • YARN-3480: Remove attempts that are beyond max-attempt limit from state store.

  • YARN-4309: Add container launch related debug information to container logs when a container fails.

  • YARN-4497: RM might fail to restart when recovering apps whose attempts are missing.

  • YARN-4565: Sometimes when sizeBasedWeight FairOrderingPolicy is enabled, under stress appears that cluster is virtually in deadlock.

  • YARN-4584: RM startup failure when AM attempts greater than max-attempts.

  • YARN-4625: ApplicationSubmissionContext and ApplicationSubmissionContextInfo more consistent.