Hadoop

HADOOP-15571: Multiple FileContexts created with the same configuration object should be allowed to have different umask. Contributed by Vinod Kumar Vavilapalli.

HDFS-13712: BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas.

HDFS-13715:diskbalancer does not work if one of the blockpools are empty on a Federated cluster. Contributed by Bharat Viswanadham

HDFS-13715: Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak.

HDFS-13635: Incorrect message when block is not found. Contributed by Gabor Bota.

HADOOP-15554: Improve JIT performance for Configuration parsing. Contributed by Todd Lipcon.

HDFS-13702: Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.

HDFS-13703: Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.

HADOOP-15574: Suppress build error if there are no docs after excluding private annotations. Contributed by Takanobu Asanuma.

HADOOP-15548: Randomize local dirs. Contributed by Jim Brennan.

HDFS-13705:The native ISA-L library loading failure should be made warning rather than an error message. Contributed by Shashikant Banerjee.

HADOOP-15458: TestLocalFileSystem#testFSOutputStreamBuilder fails on Windows. Contributed by Xiao Liang.

HDFS-13692: StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. Contributed by Bharat Viswanadham.

HDFS-13682: Cannot create encryption zone after KMS auth token expires.

HADOOP-15527: Improve delay check for stopping processes. Fixed script location. Contributed by Vinod Kumar Vavilapalli

HADOOP-15527: Improve delay check for stopping processes. Contributed by Vinod Kumar Vavilapalli

HADOOP-15533: Make WASB listStatus messages consistent. Contributed by Esfandiar Manii

HADOOP-15523: Shell command timeout given is in seconds whereas it is taken as millisec while scheduling. Contributed by Bilwa S T.

HDFS-13681: Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang.

HDFS-13686: Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak.

HDFS-13174: hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth.

HDFS-13676: TestEditLogRace fails on Windows. Contributed by Zuoming Zhang.

HDFS-13673: TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang.

HDFS-13675 Speed up TestDFSAdminWithHA. Contributed by Lukas Majercak.

HDFS-13563: TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak.

HDFS-13641: Add metrics for edit log tailing. Contributed by Chao Sun.

BUG-103209:Dependency upgrades related to blackduck scan.

HADOOP-15527: Improve delay check for stopping processes. Contributed by Vinod Kumar Vavilapalli

HADOOP-15532: TestBasicDiskValidator fails with NoSuchFileException. Contributed by Giovanni Matteo Fumarola.

HDFS-13511: Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota.

HADOOP-15529: ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows. Contributed by Giovanni Matteo Fumarola.

HADOOP-15506: Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. Contributed by Esfandiar Manii.

HDFS-13653: Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.

HADOOP-15499: Performance severe drops when running RawErasureCoderBenchmark with NativeRSRawErasureCoder. Contributed by Sammi Chen.

HDFS-13667:Typo: Marking all datandoes as stale. Contributed by Nanda Kumar

HDFS-12670: can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal.

HDFS-13664: Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun.

HADOOP-15516: Add test cases to cover FileUtil#readLink. Contributed by Giovanni Matteo Fumarola.

HDFS-13659: Add more test coverage for contentSummary for snapshottable path. Contributed by Wei-Chiu Chuang.

HDFS-12950: [oiv] ls will fail in secure cluster. Contributed by Wei-Chiu Chuang.

HADOOP-15217: FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel.

HDFS-13545: "guarded" is misspelled as "gaurded" in FSPermissionChecker.java. Contributed by Jianchao Jia.

HDFS-13650: Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy. Contributed by Anbang Hu.

HDFS-13649: Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy. Contributed by Anbang Hu.

HDFS-13652: Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu.

HADOOP-15137: ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster. Contributed by Bharat Viswanadham

HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah

HDFS-13155: BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL. Contributed by Zsolt Venczel.

HDFS-13339: Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by liaoyuxiangqin and Zsolt Venczel.

HDFS-13648: Fix TestGetConf#testGetJournalNodes on Windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola.

HDFS-13651: TestReencryptionHandler fails on Windows. Contributed by Anbang Hu.

MapReduce JHS should honor yarn.webapp.filter-entity-list-by-user(addendum). Contributed by Sunil Govindan.

HDFS-13637: RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE. Contributed by CR Hota.

HDFS-13636: Cross-Site Scripting vulnerability in HttpServer2 (Contributed by Haibo Yan via Daniel Templeton)

HDFS-13582: Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel.

HDFS-13602: Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun.

HDFS-12978: Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko. (cherry picked from commit 442dd87dcd3e9913a1d97793fde879ce51d701e7)

HDFS-13631: TestDFSAdmin#testCheckNumOfBlocksInReportCommand should use a separate MiniDFSCluster path. Contributed by Anbang Hu.

BUG-104243:Dependency jackson-databind dependency related to blackduck scan.

HADOOP-15406: hadoop-nfs dependencies for mockito and junit are not test scope

HDFS-13611: Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient.

HDFS-13629: Some tests in TestDiskBalancerCommand fail on Windows due to MiniDFSCluster path conflict and improper path usage. Contributed by Anbang Hu.

HDFS-13632: Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu.

Additional check when unpacking archives. Contributed by Wilfred Spiegelenburg.

HADOOP-15497: TestTrash should use proper test path to avoid failing on Windows. Contributed by Anbang Hu.

HADOOP-15498: TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu.

HDFS-13591: TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu.

HDFS-13627: TestErasureCodingExerciseAPIs fails on Windows. Contributed by Anbang Hu.

HADOOP-15449: Increase default timeout of ZK session to avoid frequent NameNode failover

HDFS-13628: Update Archival Storage doc for Provided Storage

HDFS-13620: Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu.

HDFS-13619: TestAuditLoggerWithCommands fails on Windows. Contributed by Anbang Hu.

HADOOP-15473: Configure serialFilter in KeyProvider to avoid UnrecoverableKeyException caused by JDK-8189997. Contributed by Gabor Bota.

HDFS-13618: Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang.

HADOOP-15486: Make NetworkTopology#netLock fair. Contributed by Nanda kumar.

Skip the proxy user check if the ugi has not been initialized. Contributed by Daryn Sharp

HDFS-13540: DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen.

HADOOP-15478: WASB: hflush() and hsync() regression. Contributed by Thomas Marquardt.

HDFS-13493: Reduce the HttpServer2 thread count on DataNodes. Contributed by Erik Krogen.

Additional check when unpacking archives. Contributed by Jason Lowe and Akira Ajisaka.

HDFS-13587: TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu.

HDFS-13588: Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang.

HADOOP-15450: Avoid fsync storm triggered by DiskChecker and handle disk full situation. Contributed by Arpit Agarwal.

HDFS-13601: Optimize ByteString conversions in PBHelper.

HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru

HADOOP-15299: Bump Jackson 2 version to Jackson 2.9.x.

HDFS-13593: TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu.

HDFS-13592: TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu.

HDFS-13586: Fsync fails on directories on Windows. Contributed by Lukas Majercack.

HDFS-13560: Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu.

HDFS-13556: TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu.

HDFS-13554: TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu.

HDFS-13558: TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu.

BUG-103397. HDFS-13589. Add dfsAdmin command to query if upgrade is finalized.

HADOOP-15469: S3A directory committer commit job fails if _temporary directory created under dest. Contributed by Steve Loughran.

HADOOP-15250: Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong Contributed by Ajay Kumar

HDFS-13570: TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu.

HDFS-13581: DN UI logs link is broken when https is enabled. Contributed by Shashikant Banerjee.

HDFS-13559: TestBlockScanner does not close TestContext properly. Contributed by Anbang Hu.

HDFS-13512: WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar.

HDFS-13550:TestDebugAdmin#testComputeMetaCommand fails on Windows. Contributed by Anbang Hu.

HDFS-13557: TestDFSAdmin#testListOpenFiles fails on Windows. Contributed by Anbang Hu.

BUG-103055 Remove aiyun, openstack from hadoop-cloud-project

HDFS-13567: TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir. Contributed by Anbang Hu.

HDFS-13548: TestResolveHdfsSymlink#testFcResolveAfs fails on Windows. Contributed by Anbang Hu.

HDFS-11700. TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows. Contributed by Anbang Hu.

HDFS-13551: TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster. Contributed by Anbang Hu.

HADOOP-15442: ITestS3AMetrics.testMetricsRegister can't know metrics source's name. Contributed by Sean Mackrory.

HDFS-13544: Improve logging for JournalNode in federated cluster.

HDFS-13539 DFSStripedInputStream NPE when reportCheckSumFailure.

Add 2.9.1 release notes and changes documents

BUG-102169 Include Google Connector in Hadoop

HADOOP-15441: Log kms url and token service at debug level. Contributed by Gabor Bota

HDFS-13542: TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows. Contributed by Anbang Hu.

HDFS-13346: RBF: Fix synchronization of router quota and nameservice quota.

HADOOP-15454: TestRollingFileSystemSinkWithLocal fails on Windows. Contributed by Xiao Liang.

HDFS-13537: TestHdfsHelper does not generate jceks path properly for relative path in Windows. Contributed by Xiao Liang.

HDFS-12981: renameSnapshot a Non-Existent snapshot to itself should throw error. Contributed by Kitti Nanasi.

HADOOP-15446: WASB: PageBlobInputStream.skip breaks HBASE replication. Contributed by Thomas Marquardt

HADOOP-14841 Kms client should disconnect if unable to get output stream from connection. Contributed by Rushabh S Shah

HDFS-13525: RBF: Add unit test TestStateStoreDisabledNameservice. Contributed by Yiqun Lin.

HDFS-13481: TestRollingFileSystemSinkWithHdfs#testFlushThread: test failed intermittently (Contributed by Gabor Bota via Daniel Templeton)

HDFS-13512 . WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar.

HDFS-13488: RBF: Reject requests when a Router is overloaded. Contributed by Inigo Goiri.

HDFS-13503: Fix TestFsck test failures on Windows. Contributed by Xiao Liang.

HADOOP-15390: Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens (xiaochen via rkanter)

HDFS-13283: Percentage based Reserved Space Calculation for DataNode. Contributed by Lukas Majercak.

HDFS-13509: Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows. Contributed by Xiao Liang.

HDFS-13508: RBF: Normalize paths (automatically) when adding, updating, removing or listing mount table entries. Contributed by Ekanth S.

HDFS-13484: RBF: Disable Nameservices from the federation. Contributed by Inigo Goiri.

HDFS-13434: RBF: Fix dead links in RBF document. Contributed by Chetna Chaudhari.

HDFS-13490: RBF: Fix setSafeMode in the Router. Contributed by Inigo Goiri.

HDFS-13499: RBF: Show disabled name services in the UI. Contributed by Inigo Goiri.

HDFS-13326: RBF: Improve the interfaces to modify and view mount tables. Contributed by Gang Li.

HADOOP-15411: AuthenticationFilter should use Configuration.getPropsWithPrefix instead of iterator. (Suma Shivaprasad via wangda)

HDFS-13336: Test cases of TestWriteToReplica failed in windows. Contributed by Xiao Liang.

HDFS-13356: Balancer:Set default value of minBlockSize to 10mb. Contributed by Bharat Viswanadham

BUG-101422. Disable HDFS and Hadoop unit tests that are flaky in Mawo.

HDFS-13408: MiniDFSCluster to support being built on randomized base directory. Contributed by Xiao Liang

HDFS-13433: webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal.

HDFS-13478: RBF: Disabled Nameservice store API. Contributed by Inigo Goiri.

HDFS-10183: Prevent race condition during class initialization. Contributed by Pavel Avgustinov.

HDFS-13453: RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount table. Contributed by Dibyendu Karmakar.

HADOOP-15396: Some java source files are executable

HDFS-13464: Fix javadoc in FsVolumeList#handleVolumeFailures. Contributed by Shashikant Banerjee

HADOOP-15366: Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown. Contributed by Shashikant Banerjee.

HDFS-13426: Fix javadoc in FsDatasetAsyncDiskService#removeVolume. Contributed by Shashikant Banerjee.

HDFS-13463: Fix javadoc in FsDatasetImpl#checkAndUpdate. Contributed by Shashikant Banerjee.

HADOOP-15391: Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules.

HDFS-12828: OIV ReverseXML Processor fails with escaped characters

BUG-100895. Ignore UTs failing due to environmental issues.

HDFS-13439: Add test case for read block operation when it is moved. Contributed by Ajay Kumar.

HDFS-13435: RBF: Improve the error loggings for printing the stack trace.

HDFS-13330: ShortCircuitCache#fetchOrCreate never retries. Contributed by Gabor Bota.

HDFS-13311: RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows. Contributed by Inigo Goiri

HDFS-13056: Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo.

HDFS-13466: RBF: Add more router-related information to the UI.

HDFS-13462: Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers. Contributed by Lukas Majercak.

HDFS-13129: Add a test for DfsAdmin refreshSuperUserGroupsConfiguration. Contributed by Mukul Kumar Singh

HDFS-13433: webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal.

HDFS-11900: Hedged reads thread pool creation not synchronized. Contributed by John Zhuge.

HDFS-11043: TestWebHdfsTimeouts fails. Contributed by Xiaoyu Yao and Chao Sun.

HDFS-13315: Add a test for the issue reported in HDFS-11481 which is fixed by HDFS-10997. Contributed by Yongjun Zhang.

HADOOP-15332: Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota.

HADOOP-15331: Fix a race condition causing parsing error of java.io.BufferedInputStream in class org.apache.hadoop.conf.Configuration. Contributed by Miklos Szegedi.

HADOOP-15330 Remove jdk1.7 profile from hadoop-annotations module

HADOOP-15062: TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9. Contributed by Miklos Szegedi.

HADOOP-14667: Flexible Visual Studio support. Contributed by Allen Wittenauer

HDFS-13438: Fix javadoc in FsVolumeList#removeVolume. Contributed by Shashikant Banerjee.

HDFS-13418: NetworkTopology should be configurable when enable DFSNetworkTopology. Contributed by Tao Jie.

HADOOP-14970: MiniHadoopClusterManager doesn't respect lack of format option. Contributed by Erik Krogen

HADOOP-15379: Make IrqHandler.bind() public. Contributed by Ajay Kumar

HDFS-13436: Fix javadoc of package-info.java

HDFS-7101: Potential null dereference in DFSck#doWork(). Contributed by skrho and Ted Yu.

HDFS-13427: Fix the section titles of transparent encryption document.

HDFS-13386: RBF: Wrong date information in list file(-ls) result. Contributed by Dibyendu Karmakar.

HDFS-13428: RBF: Remove LinkedList From StateStoreFileImpl.java. Contributed by BELUGA BEHR.

HDFS-13045: RBF: Improve error message returned from subcluster. Contributed by Inigo Goiri.

HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran.

HADOOP-15313: TestKMS should close providers.

HADOOP-15357: Configuration.getPropsWithPrefix no longer does variable substitution. Contributed by Jim Brennan

HDFS-13328: Abstract ReencryptionHandler recursive logic in separate class. Contributed by Surendra Singh Lilhore.

HADOOP-15376: Remove double semi colons on imports that make Clover fall over.

HDFS-13420: License header is displayed in ArchivalStorage/MemoryStorage html pages. Contributed by Akira Ajisaka.

HADOOP-15374: Add links of the new features of 3.1.0 to the top page

HDFS-13384: RBF: Improve timeout RPC call mechanism. Contributed by Inigo Goiri.

HDFS-13410: RBF: Support federation with no subclusters. Contributed by Inigo Goiri.

HDFS-13380: RBF: mv/rm fail after the directory exceeded the quota limit. Contributed by Yiqun Lin.

HDFS-13388: RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun.

HDFS-13402: RBF: Fix java doc for StateStoreFileSystemImpl. Contributed by Yiran Wu.

Added CHANGES/RELEASES/Jdiff for 3.1.0 release

HDFS-13353: RBF: TestRouterWebHDFSContractCreate failed. Contributed by Takanobu Asanuma.

HDFS-13350: Negative legacy block ID will confuse Erasure Coding to be considered as striped block. (Contributed by Lei (Eddy) Xu).

HDFS-13364: RBF: Support NamenodeProtocol in the Router. Contributed by Inigo Goiri.

HADOOP-15355: TestCommonConfigurationFields is broken by HADOOP-15312. Contributed by LiXin Ge.

HADOOP-15317: Improve NetworkTopology chooseRandom's loop.

HDFS-13289: RBF: TestConnectionManager#testCleanup() test case need correction. Contributed by Dibyendu Karmakar.

HADOOP-15253: Should update maxQueueSize when refresh call queue. Contributed by Tao Jie.

HADOOP-14999: AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu.

HADOOP-12862: LDAP Group Mapping over SSL can not specify trust store. Contributed by Wei-Chiu Chuang and Konstantin Shvachko.

HDFS-13087: Snapshotted encryption zone information should be immutable. Contributed by LiXin Ge.

HADOOP-15342: Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria.