Hadoop
HDP 2.4.2 provides the following Apache patches:
HADOOP-10365: BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block.
HADOOP-11212: NetUtils.wrapException to handle SocketException explicitly.
HADOOP-12100: ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermission.
HADOOP-12103: Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing.
HADOOP-12107: Long running apps may have a huge number of StatisticsData instances under FileSystem.
HADOOP-12161: Add getStoragePolicy API to the FileSystem interface.
HADOOP-12191: Bzip2Factory is not thread safe.
HADOOP-12213: Interrupted exception can occur when Client#stop is called.
HADOOP-12348: MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.
HADOOP-12374: Description of HDFS expunge command is confusing.
HADOOP-12426: Add Entry point for Kerberos health check.
HADOOP-12464: Interrupted client may try to fail-over and retry Interrupted client may try to fail-over and retry.
HADOOP-12482: Race condition in JMX cache update.
HADOOP-12589: Fix intermittent test failure of TestCopyPreserveFlag.
HADOOP-12609: Fix intermittent failure of TestDecayRpcScheduler.
HADOOP-12699: TestKMS#testKMSProvider intermittently fails during 'test rollover draining'.
HADOOP-12706: TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally.
HADOOP-12752: Improve diagnostics/use of envvar/sysprop credential propagation.
HADOOP-12787: KMS SPNEGO sequence does not work with WEBHDFS.
HADOOP-12795: KMS does not log detailed stack trace for unexpected errors.
HADOOP-12825: Log slow name resolutions.
HADOOP-12829: StatisticsDataReferenceCleaner swallows interrupt exceptions.
HADOOP-12851: S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders.
HADOOP-12903: IPC Server should allow suppressing exception logging by type, not log 'server too busy' messages.
HADOOP-12958: PhantomReference for filesystem statistics can trigger OOM.
HADOOP-13026: Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator.
HDFS-10199: Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing.
HDFS-10270: TestJMXGet:testNameNode() fails.
HDFS-10281: TestPendingCorruptDnMessages fails intermittently.
HDFS-10283: o.a.h.hdfs.server.namenode.TestFSImageWithSnapshot#testSaveLoadImageWithAppending fails intermittently.
HDFS-6101: TestReplaceDatanodeOnFailure fails occasionally.
HDFS-8113: Add check for null BlockCollection pointers in BlockInfoContiguous structures.
HDFS-8337: Accessing HttpFS via WebHDFS doesn't work from a jar with Kerberos.
HDFS-8647: Abstract BlockManager's rack policy into BlockPlacementPolicy.
HDFS-8659: Block scanner INFO message is spamming logs.
HDFS-8676: Delayed rolling upgrade finalization can cause heartbeat expiration.
HDFS-8772: Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails.
HDFS-8806: Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared.
HDFS-8815: DFS getStoragePolicy implementation using single RPC call.
HDFS-8891: HDFS concat should keep srcs order.
HDFS-9072: Fix random failures in TestJMXGet.
HDFS-9130: Use GenericTestUtils#setLogLevel to the logging level.
HDFS-9221: HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array.
HDFS-9239: DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness.
HDFS-9289: Make DataStreamer#block thread safe and verify genStamp in commitBlock.
HDFS-9290: DFSClient#callAppend() is not backward compatible for slightly older NameNodes.
HDFS-9313: Possible NullPointerException in BlockManager if no excess replica can be chosen.
HDFS-9314: Improve BlockPlacementPolicyDefault's picking of excess replicas.
HDFS-9347: Invariant assumption in TestQuorumJournalManager.shutdown() is wrong.
HDFS-9383: TestByteArrayManager#testByteArrayManager fails.
HDFS-9402: Switch DataNode.LOG to use slf4j.
HDFS-9431: DistributedFileSystem#concat fails if the target path is relative.
HDFS-9434: Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messages.
HDFS-9445: Datanode may deadlock while handling a bad volume.
HDFS-9478: Reason for failing ipc.FairCallQueue construction should be thrown.
HDFS-9534: Add CLI command to clear storage policy from a path.
HDFS-9557: Reduce object allocation in PB conversion.
HDFS-9572: Prevent DataNode log spam if a client connects on the data transfer port but sends no data.
HDFS-9574: Reduce client failures during datanode restart.
HDFS-9600: Do not check replication if the block is under construction.
HDFS-9625: Set replication for empty file failed when set storage policy.
HDFS-9655: NN should start JVM pause monitor before loading fsimage.
HDFS-9661: Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage and createRbw.
HDFS-9710: DN can be configured to send block receipt IBRs in batches.
HDFS-9724: Degraded performance in WebHDFS listing as it does not reuse ObjectMapper.
HDFS-9726: Refactor IBR code to a new class.
HDFS-9740: Use a reasonable limit in DFSTestUtil.waitForMetric().
HDFS-9743: Fix TestLazyPersistFiles#testFallbackToDiskFull in branch-2.7.
HDFS-9752: Permanent write failures may happen to slow writers during datanode rolling upgrades.
HDFS-9768: Reuse ObjectMapper instance in HDFS to improve the performance.
HDFS-9790: HDFS Balancer should exit with a proper message if upgrade is not finalized.
HDFS-9839: Reduce verbosity of processReport logging.
HDFS-9851: NameNode throws NPE when setPermission is called on a path that does not exist.
HDFS-9854: Log cipher suite negotiation more verbosely.
HDFS-9906: Remove spammy log spew when a datanode is restarted.
HDFS-9941: Do not log StandbyException on NN, other minor logging fixes.
MAPREDUCE-6436: JobHistory cache issue.
MAPREDUCE-6460: TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails.
MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.
MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.
MAPREDUCE-6670: TestJobListCache#testEviction sometimes fails on Windows with timeout.
MAPREDUCE-6680: JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS.
YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.
YARN-2871: TestRMRestart#testRMRestartGetApplicationList sometimes fails in trunk.
YARN-3102: Decommissioned Nodes not listed in Web UI.
YARN-3695: ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.
YARN-3769: Consider user limit when calculating total pending resource for preemption policy in Capacity Scheduler.
YARN-4155: TestLogAggregationService.testLogAggregationServiceWithInterval failing.
YARN-4365: FileSystemNodeLabelStore should check for root dir existence on startup.
YARN-4414: Nodemanager connection errors are retried at multiple levels.
YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.
YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.
YARN-4439: Clarify NMContainerStatus#toString method.
YARN-4546: ResourceManager crash due to scheduling opportunity overflow.
YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.
YARN-4610: Reservations continue looking for one app causes other apps to starve.
YARN-4623: TestSystemMetricsPublisher#testPublishAppAttemptMetricsForUnmanagedAM fails with NPE on branch-2.7.
YARN-4633: Fix random test failure in TestRMRestart#testRMRestartAfterPreemption.
YARN-4680: Fix TimerTasks leak in Application Timeline Server (ATS) v1.5 Writer.
YARN-4696: TimelineClient to add flush operation for deterministic writes (including testing) and Changes to EntityGroupFSTimelineStore for testability.
YARN-4709: NMWebServices produces incorrect JSON for containers.
YARN-4723: NodesListManager$UnknownNodeId ClassCastException.
YARN-4737: Add CSRF filter support in YARN.
YARN-4769: Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI.
YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.
YARN-4814: ATS 1.5 timelineclient impl call flush after every event write.
YARN-4815: ATS 1.5 timelineclinet impl try to create attempt directory for every event call.
YARN-4817: TimelineClient ATSv1.5 logging is very noisy.
YARN-4916: TestNMProxy.tesNMProxyRPCRetry fails.
YARN-4928: Some yarn.server.timeline.* tests fail on Windows attempting to use a test root path containing a colon.
YARN-4954: TestYarnClient.testReservationAPIs fails on machines with less than 4 GB available memory.
YARN-4955: Add retry for SocketTimeoutException in TimelineClient.
YARN-4965: Distributed shell AM failed due to ClientHandlerException thrown by jersey.
YARN-4968: Fix two scheduler related UTs in YARN.
HDP 2.4.0 provided the following Apache patches:
HADOOP-10406: TestIPC.testIpcWithReaderQueuing may fail.
HADOOP-12551: Introduce FileNotFoundException for WASB FileSystem API.
HADOOP-12608: Fix exception message in WASB when connecting with anonymous credential.
HADOOP-12678: Handle empty rename pending metadata file during atomic rename in redo path.
HDFS-8729: Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately which occasionally failed.
HDFS-9358: TestNodeCount#testNodeCount timed out.
HDFS-9406: FSImage may get corrupted after deleting snapshot.
HDFS-9672: o.a.h.hdfs.TestLeaseRecovery2 fails intermittently.
MAPREDUCE-6566: Add retry support to mapreduce CLI tool.
MAPREDUCE-6618: YarnClientProtocolProvider leaking the YarnClient thread.
MAPREDUCE-6621: Memory Link in JobClient#submitJobInternal().
YARN-3480: Remove attempts that are beyond max-attempt limit from state store.
YARN-4309: Add container launch related debug information to container logs when a container fails.
YARN-4497: RM might fail to restart when recovering apps whose attempts are missing.
YARN-4565: Sometimes when sizeBasedWeight FairOrderingPolicy is enabled, under stress appears that cluster is virtually in deadlock.
YARN-4584: RM startup failure when AM attempts greater than max-attempts.
YARN-4625: ApplicationSubmissionContext and ApplicationSubmissionContextInfo more consistent.