HDP 3.1.4 Release Notes
Also available as:
PDF

Hadoop

This release provides Hadoop Common 3.1.1 and no additional Apache patches.

HDP 3.1.0 provided Hadoop Common 3.1.1 and no additional Apache patches.

HDP 3.0.1 provided Hadoop Common 3.1.1 and no additional Apache patches.

HDP 3.0.0 provided Hadoop Common 3.1.0 and the following Apache patches:

  • HADOOP-15571: Multiple FileContexts created with the same configuration object should be allowed to have different umask. Contributed by Vinod Kumar Vavilapalli.

  • HDFS-13712: BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas.

  • HDFS-13715:diskbalancer does not work if one of the blockpools are empty on a Federated cluster. Contributed by Bharat Viswanadham

  • HDFS-13715: Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak.

  • HDFS-13635: Incorrect message when block is not found. Contributed by Gabor Bota.

  • HADOOP-15554: Improve JIT performance for Configuration parsing. Contributed by Todd Lipcon.

  • HDFS-13702: Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.

  • HDFS-13703: Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.

  • HADOOP-15574: Suppress build error if there are no docs after excluding private annotations. Contributed by Takanobu Asanuma.

  • HADOOP-15548: Randomize local dirs. Contributed by Jim Brennan.

  • HDFS-13705:The native ISA-L library loading failure should be made warning rather than an error message. Contributed by Shashikant Banerjee.

  • HADOOP-15458: TestLocalFileSystem#testFSOutputStreamBuilder fails on Windows. Contributed by Xiao Liang.

  • HDFS-13692: StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. Contributed by Bharat Viswanadham.

  • HDFS-13682: Cannot create encryption zone after KMS auth token expires.

  • HADOOP-15527: Improve delay check for stopping processes. Fixed script location. Contributed by Vinod Kumar Vavilapalli

  • HADOOP-15527: Improve delay check for stopping processes. Contributed by Vinod Kumar Vavilapalli

  • HADOOP-15533: Make WASB listStatus messages consistent. Contributed by Esfandiar Manii

  • HADOOP-15523: Shell command timeout given is in seconds whereas it is taken as millisec while scheduling. Contributed by Bilwa S T.

  • HDFS-13681: Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang.

  • HDFS-13686: Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak.

  • HDFS-13174: hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth.

  • HDFS-13676: TestEditLogRace fails on Windows. Contributed by Zuoming Zhang.

  • HDFS-13673: TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang.

  • HDFS-13675 Speed up TestDFSAdminWithHA. Contributed by Lukas Majercak.

  • HDFS-13563: TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak.

  • HDFS-13641: Add metrics for edit log tailing. Contributed by Chao Sun.

  • BUG-103209:Dependency upgrades related to blackduck scan.

  • HADOOP-15527: Improve delay check for stopping processes. Contributed by Vinod Kumar Vavilapalli

  • HADOOP-15532: TestBasicDiskValidator fails with NoSuchFileException. Contributed by Giovanni Matteo Fumarola.

  • HDFS-13511: Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota.

  • HADOOP-15529: ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows. Contributed by Giovanni Matteo Fumarola.

  • HADOOP-15506: Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. Contributed by Esfandiar Manii.

  • HDFS-13653: Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.

  • HADOOP-15499: Performance severe drops when running RawErasureCoderBenchmark with NativeRSRawErasureCoder. Contributed by Sammi Chen.

  • HDFS-13667:Typo: Marking all datandoes as stale. Contributed by Nanda Kumar

  • HDFS-12670: can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal.

  • HDFS-13664: Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun.

  • HADOOP-15516: Add test cases to cover FileUtil#readLink. Contributed by Giovanni Matteo Fumarola.

  • HDFS-13659: Add more test coverage for contentSummary for snapshottable path. Contributed by Wei-Chiu Chuang.

  • HDFS-12950: [oiv] ls will fail in secure cluster. Contributed by Wei-Chiu Chuang.

  • HADOOP-15217: FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel.

  • HDFS-13545: "guarded" is misspelled as "gaurded" in FSPermissionChecker.java. Contributed by Jianchao Jia.

  • HDFS-13650: Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy. Contributed by Anbang Hu.

  • HDFS-13649: Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy. Contributed by Anbang Hu.

  • HDFS-13652: Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu.

  • HADOOP-15137: ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster. Contributed by Bharat Viswanadham

  • HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah

  • HDFS-13155: BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL. Contributed by Zsolt Venczel.

  • HDFS-13339: Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by liaoyuxiangqin and Zsolt Venczel.

  • HDFS-13648: Fix TestGetConf#testGetJournalNodes on Windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola.

  • HDFS-13651: TestReencryptionHandler fails on Windows. Contributed by Anbang Hu.

  • MapReduce JHS should honor yarn.webapp.filter-entity-list-by-user(addendum). Contributed by Sunil Govindan.

  • HDFS-13637: RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE. Contributed by CR Hota.

  • HDFS-13636: Cross-Site Scripting vulnerability in HttpServer2 (Contributed by Haibo Yan via Daniel Templeton)

  • HDFS-13582: Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel.

  • HDFS-13602: Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun.

  • HDFS-12978: Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko. (cherry picked from commit 442dd87dcd3e9913a1d97793fde879ce51d701e7)

  • HDFS-13631: TestDFSAdmin#testCheckNumOfBlocksInReportCommand should use a separate MiniDFSCluster path. Contributed by Anbang Hu.

  • BUG-104243:Dependency jackson-databind dependency related to blackduck scan.

  • HADOOP-15406: hadoop-nfs dependencies for mockito and junit are not test scope

  • HDFS-13611: Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient.

  • HDFS-13629: Some tests in TestDiskBalancerCommand fail on Windows due to MiniDFSCluster path conflict and improper path usage. Contributed by Anbang Hu.

  • HDFS-13632: Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu.

  • Additional check when unpacking archives. Contributed by Wilfred Spiegelenburg.

  • HADOOP-15497: TestTrash should use proper test path to avoid failing on Windows. Contributed by Anbang Hu.

  • HADOOP-15498: TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu.

  • HDFS-13591: TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu.

  • HDFS-13627: TestErasureCodingExerciseAPIs fails on Windows. Contributed by Anbang Hu.

  • HADOOP-15449: Increase default timeout of ZK session to avoid frequent NameNode failover

  • HDFS-13628: Update Archival Storage doc for Provided Storage

  • HDFS-13620: Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu.

  • HDFS-13619: TestAuditLoggerWithCommands fails on Windows. Contributed by Anbang Hu.

  • HADOOP-15473: Configure serialFilter in KeyProvider to avoid UnrecoverableKeyException caused by JDK-8189997. Contributed by Gabor Bota.

  • HDFS-13618: Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang.

  • HADOOP-15486: Make NetworkTopology#netLock fair. Contributed by Nanda kumar.

  • Skip the proxy user check if the ugi has not been initialized. Contributed by Daryn Sharp

  • HDFS-13540: DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen.

  • HADOOP-15478: WASB: hflush() and hsync() regression. Contributed by Thomas Marquardt.

  • HDFS-13493: Reduce the HttpServer2 thread count on DataNodes. Contributed by Erik Krogen.

  • Additional check when unpacking archives. Contributed by Jason Lowe and Akira Ajisaka.

  • HDFS-13587: TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu.

  • HDFS-13588: Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang.

  • HADOOP-15450: Avoid fsync storm triggered by DiskChecker and handle disk full situation. Contributed by Arpit Agarwal.

  • HDFS-13601: Optimize ByteString conversions in PBHelper.

  • HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru

  • HADOOP-15299: Bump Jackson 2 version to Jackson 2.9.x.

  • HDFS-13593: TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu.

  • HDFS-13592: TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu.

  • HDFS-13586: Fsync fails on directories on Windows. Contributed by Lukas Majercack.

  • HDFS-13560: Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu.

  • HDFS-13556: TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu.

  • HDFS-13554: TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu.

  • HDFS-13558: TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu.

  • BUG-103397. HDFS-13589. Add dfsAdmin command to query if upgrade is finalized.

  • HADOOP-15469: S3A directory committer commit job fails if _temporary directory created under dest. Contributed by Steve Loughran.

  • HADOOP-15250: Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong Contributed by Ajay Kumar

  • HDFS-13570: TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu.

  • HDFS-13581: DN UI logs link is broken when https is enabled. Contributed by Shashikant Banerjee.

  • HDFS-13559: TestBlockScanner does not close TestContext properly. Contributed by Anbang Hu.

  • HDFS-13512: WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar.

  • HDFS-13550:TestDebugAdmin#testComputeMetaCommand fails on Windows. Contributed by Anbang Hu.

  • HDFS-13557: TestDFSAdmin#testListOpenFiles fails on Windows. Contributed by Anbang Hu.

  • BUG-103055 Remove aiyun, openstack from hadoop-cloud-project

  • HDFS-13567: TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir. Contributed by Anbang Hu.

  • HDFS-13548: TestResolveHdfsSymlink#testFcResolveAfs fails on Windows. Contributed by Anbang Hu.

  • HDFS-11700. TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows. Contributed by Anbang Hu.

  • HDFS-13551: TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster. Contributed by Anbang Hu.

  • HADOOP-15442: ITestS3AMetrics.testMetricsRegister can't know metrics source's name. Contributed by Sean Mackrory.

  • HDFS-13544: Improve logging for JournalNode in federated cluster.

  • HDFS-13539 DFSStripedInputStream NPE when reportCheckSumFailure.

  • Add 2.9.1 release notes and changes documents

  • BUG-102169 Include Google Connector in Hadoop

  • HADOOP-15441: Log kms url and token service at debug level. Contributed by Gabor Bota

  • HDFS-13542: TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows. Contributed by Anbang Hu.

  • HDFS-13346: RBF: Fix synchronization of router quota and nameservice quota.

  • HADOOP-15454: TestRollingFileSystemSinkWithLocal fails on Windows. Contributed by Xiao Liang.

  • HDFS-13537: TestHdfsHelper does not generate jceks path properly for relative path in Windows. Contributed by Xiao Liang.

  • HDFS-12981: renameSnapshot a Non-Existent snapshot to itself should throw error. Contributed by Kitti Nanasi.

  • HADOOP-15446: WASB: PageBlobInputStream.skip breaks HBASE replication. Contributed by Thomas Marquardt

  • HADOOP-14841 Kms client should disconnect if unable to get output stream from connection. Contributed by Rushabh S Shah

  • HDFS-13525: RBF: Add unit test TestStateStoreDisabledNameservice. Contributed by Yiqun Lin.

  • HDFS-13481: TestRollingFileSystemSinkWithHdfs#testFlushThread: test failed intermittently (Contributed by Gabor Bota via Daniel Templeton)

  • HDFS-13512 . WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar.

  • HDFS-13488: RBF: Reject requests when a Router is overloaded. Contributed by Inigo Goiri.

  • HDFS-13503: Fix TestFsck test failures on Windows. Contributed by Xiao Liang.

  • HADOOP-15390: Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens (xiaochen via rkanter)

  • HDFS-13283: Percentage based Reserved Space Calculation for DataNode. Contributed by Lukas Majercak.

  • HDFS-13509: Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows. Contributed by Xiao Liang.

  • HDFS-13508: RBF: Normalize paths (automatically) when adding, updating, removing or listing mount table entries. Contributed by Ekanth S.

  • HDFS-13484: RBF: Disable Nameservices from the federation. Contributed by Inigo Goiri.

  • HDFS-13434: RBF: Fix dead links in RBF document. Contributed by Chetna Chaudhari.

  • HDFS-13490: RBF: Fix setSafeMode in the Router. Contributed by Inigo Goiri.

  • HDFS-13499: RBF: Show disabled name services in the UI. Contributed by Inigo Goiri.

  • HDFS-13326: RBF: Improve the interfaces to modify and view mount tables. Contributed by Gang Li.

  • HADOOP-15411: AuthenticationFilter should use Configuration.getPropsWithPrefix instead of iterator. (Suma Shivaprasad via wangda)

  • HDFS-13336: Test cases of TestWriteToReplica failed in windows. Contributed by Xiao Liang.

  • HDFS-13356: Balancer:Set default value of minBlockSize to 10mb. Contributed by Bharat Viswanadham

  • BUG-101422. Disable HDFS and Hadoop unit tests that are flaky in Mawo.

  • HDFS-13408: MiniDFSCluster to support being built on randomized base directory. Contributed by Xiao Liang

  • HDFS-13433: webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal.

  • HDFS-13478: RBF: Disabled Nameservice store API. Contributed by Inigo Goiri.

  • HDFS-10183: Prevent race condition during class initialization. Contributed by Pavel Avgustinov.

  • HDFS-13453: RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount table. Contributed by Dibyendu Karmakar.

  • HADOOP-15396: Some java source files are executable

  • HDFS-13464: Fix javadoc in FsVolumeList#handleVolumeFailures. Contributed by Shashikant Banerjee

  • HADOOP-15366: Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown. Contributed by Shashikant Banerjee.

  • HDFS-13426: Fix javadoc in FsDatasetAsyncDiskService#removeVolume. Contributed by Shashikant Banerjee.

  • HDFS-13463: Fix javadoc in FsDatasetImpl#checkAndUpdate. Contributed by Shashikant Banerjee.

  • HADOOP-15391: Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules.

  • HDFS-12828: OIV ReverseXML Processor fails with escaped characters

  • BUG-100895. Ignore UTs failing due to environmental issues.

  • HDFS-13439: Add test case for read block operation when it is moved. Contributed by Ajay Kumar.

  • HDFS-13435: RBF: Improve the error loggings for printing the stack trace.

  • HDFS-13330: ShortCircuitCache#fetchOrCreate never retries. Contributed by Gabor Bota.

  • HDFS-13311: RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows. Contributed by Inigo Goiri

  • HDFS-13056: Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo.

  • HDFS-13466: RBF: Add more router-related information to the UI.

  • HDFS-13462: Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers. Contributed by Lukas Majercak.

  • HDFS-13129: Add a test for DfsAdmin refreshSuperUserGroupsConfiguration. Contributed by Mukul Kumar Singh

  • HDFS-13433: webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal.

  • HDFS-11900: Hedged reads thread pool creation not synchronized. Contributed by John Zhuge.

  • HDFS-11043: TestWebHdfsTimeouts fails. Contributed by Xiaoyu Yao and Chao Sun.

  • HDFS-13315: Add a test for the issue reported in HDFS-11481 which is fixed by HDFS-10997. Contributed by Yongjun Zhang.

  • HADOOP-15332: Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota.

  • HADOOP-15331: Fix a race condition causing parsing error of java.io.BufferedInputStream in class org.apache.hadoop.conf.Configuration. Contributed by Miklos Szegedi.

  • HADOOP-15330 Remove jdk1.7 profile from hadoop-annotations module

  • HADOOP-15062: TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9. Contributed by Miklos Szegedi.

  • HADOOP-14667: Flexible Visual Studio support. Contributed by Allen Wittenauer

  • HDFS-13438: Fix javadoc in FsVolumeList#removeVolume. Contributed by Shashikant Banerjee.

  • HDFS-13418: NetworkTopology should be configurable when enable DFSNetworkTopology. Contributed by Tao Jie.

  • HADOOP-14970: MiniHadoopClusterManager doesn't respect lack of format option. Contributed by Erik Krogen

  • HADOOP-15379: Make IrqHandler.bind() public. Contributed by Ajay Kumar

  • HDFS-13436: Fix javadoc of package-info.java

  • HDFS-7101: Potential null dereference in DFSck#doWork(). Contributed by skrho and Ted Yu.

  • HDFS-13427: Fix the section titles of transparent encryption document.

  • HDFS-13386: RBF: Wrong date information in list file(-ls) result. Contributed by Dibyendu Karmakar.

  • HDFS-13428: RBF: Remove LinkedList From StateStoreFileImpl.java. Contributed by BELUGA BEHR.

  • HDFS-13045: RBF: Improve error message returned from subcluster. Contributed by Inigo Goiri.

  • HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran.

  • HADOOP-15313: TestKMS should close providers.

  • HADOOP-15357: Configuration.getPropsWithPrefix no longer does variable substitution. Contributed by Jim Brennan

  • HDFS-13328: Abstract ReencryptionHandler recursive logic in separate class. Contributed by Surendra Singh Lilhore.

  • HADOOP-15376: Remove double semi colons on imports that make Clover fall over.

  • HDFS-13420: License header is displayed in ArchivalStorage/MemoryStorage html pages. Contributed by Akira Ajisaka.

  • HADOOP-15374: Add links of the new features of 3.1.0 to the top page

  • HDFS-13384: RBF: Improve timeout RPC call mechanism. Contributed by Inigo Goiri.

  • HDFS-13410: RBF: Support federation with no subclusters. Contributed by Inigo Goiri.

  • HDFS-13380: RBF: mv/rm fail after the directory exceeded the quota limit. Contributed by Yiqun Lin.

  • HDFS-13388: RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun.

  • HDFS-13402: RBF: Fix java doc for StateStoreFileSystemImpl. Contributed by Yiran Wu.

  • Added CHANGES/RELEASES/Jdiff for 3.1.0 release

  • HDFS-13353: RBF: TestRouterWebHDFSContractCreate failed. Contributed by Takanobu Asanuma.

  • HDFS-13350: Negative legacy block ID will confuse Erasure Coding to be considered as striped block. (Contributed by Lei (Eddy) Xu).

  • HDFS-13364: RBF: Support NamenodeProtocol in the Router. Contributed by Inigo Goiri.

  • HADOOP-15355: TestCommonConfigurationFields is broken by HADOOP-15312. Contributed by LiXin Ge.

  • HADOOP-15317: Improve NetworkTopology chooseRandom's loop.

  • HDFS-13289: RBF: TestConnectionManager#testCleanup() test case need correction. Contributed by Dibyendu Karmakar.

  • HADOOP-15253: Should update maxQueueSize when refresh call queue. Contributed by Tao Jie.

  • HADOOP-14999: AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu.

  • HADOOP-12862: LDAP Group Mapping over SSL can not specify trust store. Contributed by Wei-Chiu Chuang and Konstantin Shvachko.

  • HDFS-13087: Snapshotted encryption zone information should be immutable. Contributed by LiXin Ge.

  • HADOOP-15342: Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria.