Hadoop

This release provides Hadoop Common 2.7.3 and the following Apache patches:

HADOOP-13190: Mention LoadBalancingKMSClientProvider in KMS HA documentation.
HADOOP-13227: AsyncCallHandler should use an event driven architecture to handle async calls.
HADOOP-14104: Client should always ask namenode for kms provider path.
HADOOP-14799: Update nimbus-jose-jwt to 4.41.1.
HADOOP-14814: Fix incompatible API change on FsServerDefaults to HADOOP-14104.
HADOOP-14903: Add json-smart explicitly to pom.xml.
HADOOP-15042: Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0.
HADOOP-15255: Upper/Lower case conversion support for group names in LdapGroupsMapping.
HADOOP-15265: exclude json-smart explicitly from hadoop-auth pom.xml.
HDFS-7922: ShortCircuitCache#close is not releasing ScheduledThreadPoolExecutors.
HDFS-8496: Calling stopWriter() with FSDatasetImpl lock held may block other threads (cmccabe).
HDFS-10267: Extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose.
HDFS-10489: Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones.
HDFS-11384: Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike.
HDFS-11689: New exception thrown by DFSClient%isHDFSEncryptionEnabled broke hacky hive code..
HDFS-11711: DN should not delete the block On "Too many open files" Exception.
HDFS-12347: TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently.
HDFS-12781: After Datanode down, In Namenode UI Datanode tab is throwing warning message..
HDFS-13054: Handling PathIsNotEmptyDirectoryException in DFSClient delete call..
HDFS-13120: Snapshot diff could be corrupted after concat.
YARN-3742: YARN RM will shut down if ZKClient creation times out.
YARN-6061: Add an UncaughtExceptionHandler for critical threads in RM.
YARN-7558: yarn logs command fails to get logs for running containers if UI authentication is enabled.
YARN-7697: Fetching logs for finished application fails even though log aggregation is complete.

HDP 2.6.4 provided Hadoop Common 2.7.3 and the following Apache patches:

HADOOP-13700: Remove unthrown IOException from TrashPolicy#initialize and #getInstance signatures.
HADOOP-13709: Ability to clean up subprocesses spawned by Shell when the process exits..
HADOOP-14059: typo in s3a rename(self, subdir) error message.
HADOOP-14542: Add IOUtils.cleanupWithLogger that accepts slf4j logger API.
HDFS-9887: WebHdfs socket timeouts should be configurable.
HDFS-9914: Fix configurable WebhDFS connect/read timeout.
MAPREDUCE-6698: Increase timeout on TestUnnecessaryBlockingOnHist oryFileInfo.testTwoThreadsQueryingDifferentJobOfSameUser.
YARN-4550: Some tests in TestContainerLanch fails on non-english locale environment.
YARN-4717: TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IllegalArgumentException from cleanup.
YARN-5042: Mount /sys/fs/cgroup into Docker containers as readonly mount.
YARN-5318: Fix intermittent test failure of TestRMAdminService#te stRefreshNodesResourceWithFileSystemBasedConfigurationProvider.
YARN-5641: Localizer leaves behind tarballs after container is complete..
YARN-6004: Refactor TestResourceLocalizationService#testDownloadingResourcesOnContainer so that it is less than 150 lines.
YARN-6078: Containers stuck in Localizing state.
YARN-6805: NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code..

HDP 2.6.3 provided Hadoop Common 2.7.3 and the following Apache patches:

HADOOP-1023: Add more logging in WASB page blob read function.
HADOOP-1026:. Logging bug in WASB causing out of index exception.
HADOOP-1540:. Support file exclusion list in distcp.
HADOOP-1540: breaks backward compatibilty.
HADOOP-3733: "s3x:" URLs break when Secret Key contains a slash, even if encoded.
HADOOP-5323:. Trash documentation should describe its directory structure and configurations.
HADOOP-5470: RunJar.unJar() should write the last modified time found in the jar entry to the uncompressed file.
HADOOP-5732: Add SFTPFileSystem.
HADOOP-7139:. Allow appending to existing SequenceFiles.
HADOOP-7256: Resource leak during failure scenario of closing of resources.
HADOOP-7713: dfs -count -q should label output column.
HADOOP-7713: dfs -count -q should label output column.
HADOOP-7817:. RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos().
HADOOP-7930: Kerberos relogin interval in UserGroupInformation should be configurable (xiaochen via rkanter) .
HADOOP-8151: Error handling in snappy decompressor throws invalid exceptions.
HADOOP-8436: NPE In getLocalPathForWrite - path, conf - when the required context item is not configured.
HADOOP-8437: getLocalPathForWrite should throw IOException for invalid paths.
HADOOP-8751: NPE in Token.toString() when Token is constructed using null identifier.
HADOOP-8818:. Use equals instead == in MD5MD5CRC32FileChecksum and TFileDumper.
HADOOP-8934: Shell command ls should include sort options.
HADOOP-9121:. InodeTree.java has redundant check for vName while throwing exception.
HADOOP-9242:. Duplicate surefire plugin config in hadoop-common.
HADOOP-9477:. Add posixGroups support for LDAP groups mapping service.
HADOOP-10035: Cleanup TestFilterFileSystem.
HADOOP-10048: LocalDirAllocator should avoid holding locks while accessing the filesystem.
HADOOP-10300:. Allowed deferred sending of call responses.
HADOOP-10365: BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block.
HADOOP-10406: TestIPC.testIpcWithReaderQueuing may fail.
HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.
HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.
HADOOP-10774: Update KerberosTestUtils for hadoop-auth tests when using IBM Java.
HADOOP-10775:. Shell operations to fail with meaningful errors on windows if winutils.exe not found.
HADOOP-10823: TestReloadingX509TrustManager is flaky.
HADOOP-10829: Iteration on CredentialProviderFactory.serviceLoader is thread-unsafe.
HADOOP-10965: Print fully qualified path in CommandWithDestination error messages.
HADOOP-10971: Add -C flag to make `hadoop fs -ls` print filenames only.
HADOOP-11031:. Design Document for Credential Provider API.
HADOOP-11098: [JDK8] Max Non Heap Memory default changed between JDK7 and 8.
HADOOP-11180: Change log message "token.Token: Cannot find class for token kind kms-dt" to debug.
HADOOP-11212: NetUtils.wrapException to handle SocketException explicitly.
HADOOP-11218: Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.
HADOOP-11252: RPC client does not time out by default.
HADOOP-11262: Enable YARN to use S3A.
HADOOP-11361: Fix a race condition in MetricsSourceAdapter.updateJmxCache.
HADOOP-11404: Clarify the "expected client Kerberos principal is null" authorization message.
HADOOP-11418: Property "io.compression.codec.lzo.class" does not work with other value besides default.
HADOOP-11491:. HarFs incorrectly declared as requiring an authority.
HADOOP-11572: s3a delete() operation fails during a concurrent delete of child entries.
HADOOP-11572: s3a delete() operation fails during a concurrent delete of child entries.
HADOOP-11581: Multithreaded correctness Warnings #org.apache.hadoop.fs.shell.Ls.
HADOOP-11613:. Remove commons-httpclient dependency from hadoop-azure.
HADOOP-11628: SPNEGO auth does not work with CNAMEs in JDK8. (Daryn Sharp via stevel).
HADOOP-11677: Add cookie flags for logs and static contexts.
HADOOP-11685: StorageException complaining "no lease ID" during HBase distributed log splitting.
HADOOP-11687: Ignore x-* and response headers when copying an Amazon S3 object.
HADOOP-11692:. Improve authentication failure WARN message to avoid user confusion.
HADOOP-11713: ViewFileSystem should support snapshot methods.
HADOOP-11717: Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth.
HADOOP-11772: RPC Invoker relies on static ClientCache which has synchronized(this) blocks.
HADOOP-11780: Prevent IPC reader thread death. .
HADOOP-11785:. Reduce the number of listStatus operation in distcp buildListing.
HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm.
HADOOP-11812:. Implement listLocatedStatus for ViewFileSystem to speed up split calculation.
HADOOP-11819: HttpServerFunctionalTest#prepareTestWebapp should create web app directory if it does not exist.
HADOOP-11827:. Speed-up distcp buildListing() using threadpool.
HADOOP-11846: TestCertificateUtil.testCorruptPEM failing on Jenkins JDK8.
HADOOP-11859: PseudoAuthenticationHandler fails with httpcomponents v4.4.
HADOOP-11864:. JWTRedirectAuthenticationHandler breaks java8 javadocs.
HADOOP-11876:. Refactor code to make it more readable, minor maybePrintStats bug.
HADOOP-11901: BytesWritable fails to support 2G chunks due to integer overflow.
HADOOP-11918: Listing an empty s3a root directory throws FileNotFound.
HADOOP-11924:. Tolerate JDK-8047340-related exceptions in Shell#isSetSidAvailable preventing class init.
HADOOP-11932: MetricsSinkAdapter may hang when being stopped.
HADOOP-11960: Enable Azure-Storage Client Side logging.
HADOOP-12001:. Fixed LdapGroupsMapping to include configurable Posix UID and GID attributes during the search.
HADOOP-12006:. Remove unimplemented option for from document in branch-2: 7.
HADOOP-12009: Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus. backport to 2.8.
HADOOP-12017: Hadoop archives command should use configurable replication factor when closing.
HADOOP-12043: Display warning if defaultFs is not set when running fs commands.
HADOOP-12045: Enable LocalFileSystem#setTimes to change atime.
HADOOP-12049: Control http authentication cookie persistence via configuration.
HADOOP-12052:IPC client downgrades all exception types to IOE, breaks callers trying to use them.
HADOOP-12054: RPC client should not retry for InvalidToken exceptions. .
HADOOP-12056: Use DirectoryStream in DiskChecker#checkDirs to detect errors when listing a directory.
HADOOP-12058:. Fix dead links to DistCp and Hadoop Archives pages.
HADOOP-12059:. S3Credentials should support use of CredentialProvider.
HADOOP-12061:. Incorrect command in single cluster setup document.
HADOOP-12073: Azure FileSystem PageBlobInputStream does not return -1 on EOF.
HADOOP-12074: in Shell.java#runCommand() rethrow InterruptedException as InterruptedIOException.
HADOOP-12076:. Incomplete Cache Mechanism in CredentialProvider API.
HADOOP-12087: [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel).'.
HADOOP-12089: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.
HADOOP-12095: org.apache.hadoop.fs.shell.TestCount fails.
HADOOP-12097: Allow port range to be specified while starting webapp.
HADOOP-12100:. ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermission.
HADOOP-12103: Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing.
HADOOP-12107: long running apps may have a huge number of StatisticsData instances under FileSystem.
HADOOP-12119:. hadoop fs -expunge does not work for federated namespace.
HADOOP-12158: Improve error message in TestCryptoStreamsWithOpensslAesCtrCryptoCodec when OpenSSL is not installed.
HADOOP-12159: Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces.
HADOOP-12161: Add getStoragePolicy API to the FileSystem interface.
HADOOP-12169:ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse.
HADOOP-12173: NetworkTopology::add calls toString always.
HADOOP-12185: NetworkTopology is not efficient adding/getting/removing nodes.
HADOOP-12186: ActiveStandbyElector shouldn't call monitorLockNodeAsync multiple times.
HADOOP-12189: Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.
HADOOP-12191: Bzip2Factory is not thread safe.
HADOOP-12213: Interrupted exception can occur when Client#stop is called.
HADOOP-12229:Fix inconsistent subsection titles in filesystem.md.
HADOOP-12230:. hadoop-project declares duplicate, conflicting curator dependencies.
HADOOP-12232:. Upgrade Tomcat dependency to 6.0.44.
HADOOP-12239: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.
HADOOP-12242: Add in-page TOC to filesystem specification pages.
HADOOP-12252: LocalDirAllocator should not throw NPE with empty string configuration.
HADOOP-12258: Need translate java.nio.file.NoSuchFileException to FileNotFoundException to avoid regression.
HADOOP-12259: Utility to Dynamic port allocation.
HADOOP-12269: Update aws-sdk dependency to 1.10.6.
HADOOP-12280:. Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded.
HADOOP-12284:. UserGroupInformation doAs can throw misleading exception.
HADOOP-12291:. Add support for nested groups in LdapGroupsMapping.
HADOOP-12292: Make use of DeleteObjects optional.
HADOOP-12296:. when setnetgrent returns 0 in linux, exception should be thrown.
HADOOP-12304: Applications using FileContext fail with the default file system configured to be wasb/s3/etc.
HADOOP-12317: Applications fail on NM restart on some linux distro because NM container recovery declares AM container as LOST.
HADOOP-12324: Better exception reporting in SaslPlainServer. .
HADOOP-12325: RPC Metrics : Add the ability track and log slow RPCs.
HADOOP-12334: Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries.
HADOOP-12345: Pad hostname correctly in CredentialsSys.java.
HADOOP-12346: Increase some default timeouts / retries for S3a connector.
HADOOP-12348: MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.
HADOOP-12350: WASB Logging: Improve WASB Logging around deletes, reads and writes.
HADOOP-12358: Add -safely flag to rm to prompt when deleting many files.
HADOOP-12359:. hadoop fs -getmerge doc is wrong.
HADOOP-12366:. expose calculated paths.
HADOOP-12374: Description of hdfs expunge command is confusing.
HADOOP-12386: RetryPolicies.RETRY_FOREVER should be able to specify a retry interval. (Sunil G via wangda) .
HADOOP-12404: Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class.
HADOOP-12406:. Fixed AbstractMapWritable.readFields to use the thread's ClassLoader to load class instead of System ClassLoader.
HADOOP-12407: Test failing: hadoop.ipc.TestSaslRPC.
HADOOP-12413: AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.
HADOOP-12415:. Add io.netty dependency to hadoop-nfs and to hadoop-hdfs, needed to build Bigtop successfuly, see BIGTOP-2049.
HADOOP-12416:. Trash messages should be handled by Logger instead of being delivered on System.out.
HADOOP-12416:. Trash messages should be handled by Logger instead of being delivered on System.out.".
HADOOP-12418: TestRPC.testRPCInterruptedSimple fails intermittently.
HADOOP-12423: Handle failure of registering shutdownhook by ShutdownHookManager in static block.
HADOOP-12426:. Add Entry point for Kerberos health check.
HADOOP-12437: Allow SecurityUtil to lookup alternate hostnames.
HADOOP-12438: TestLocalFileSystem tests can fail on Windows after HDFS-8767 fix for handling pipe.
HADOOP-12440: TestRPC#testRPCServerShutdown did not produce the desired thread states before shutting down.
HADOOP-12441: Fixed kill-command behavior to work correctly across OSes by using bash shell built-in.
HADOOP-12442: Display help if the command option to 'hdfs dfs' is not valid (Contributed by nijel) .
HADOOP-12444: Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel.
HADOOP-12449: TestDNS and TestNetUtils failing if no network.
HADOOP-12453: Support decoding KMS Delegation Token with its own Identifier. .
HADOOP-12457: [JDK8] Fix a failure of compiling common by javadoc.
HADOOP-12463: Fix TestShell.testGetSignalKillCommand failure on windows.
HADOOP-12464:. Interrupted client may try to fail-over and retry.
HADOOP-12465:. Incorrect javadoc in WritableUtils.java.
HADOOP-12469: distcp should not ignore the ignoreFailures option.
HADOOP-12472: Make GenericTestUtils.assertExceptionContains robust.
HADOOP-12478:. Shell.getWinUtilsPath() has been renamed Shell.getWinutilsPath().
HADOOP-12481:. JWTRedirectAuthenticationHandler doesn't Retain Original Query String.
HADOOP-12482:. Race condition in JMX cache update.
HADOOP-12483: Maintain wrapped SASL ordering for postponed IPC responses. (Daryn Sharp via yliu) .
HADOOP-12484: Single File Rename Throws Incorrectly In Potential Race Condition Scenarios.
HADOOP-12508: delete fails with exception when lease is held on blob.
HADOOP-12520:. Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests.
HADOOP-12526:. there are duplicate dependency definitions in pom's.
HADOOP-12533: Introduce FileNotFoundException in WASB for read and seek API.
HADOOP-12535:. Run FileSystem contract tests with hadoop-azure.
HADOOP-12537: S3A to support Amazon STS temporary credentials.
HADOOP-12540: TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown.
HADOOP-12542: TestDNS fails on Windows after HADOOP-12437.
HADOOP-12545:. Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp.
HADOOP-12548:. Read s3a creds from a Credential Provider.
HADOOP-12551: Introduce FileNotFoundException for WASB FileSystem API.
HADOOP-12555:. WASB to read credentials from a credential provider.
HADOOP-12559:. KMS connection failures should trigger TGT renewal.
HADOOP-12564:. Upgrade JUnit3 TestCase to JUnit 4 in org.apache.hadoop.io package.
HADOOP-12565:. Replace DSA with RSA for SSH key type in SingleCluster.md.
HADOOP-12568:. Update core-default.xml to describe posixGroups support.
HADOOP-12570:. HDFS Secure Mode Documentation updates.
HADOOP-12573: TestRPC.testClientBackOff failing.
HADOOP-12577: Bump up commons-collections version to 3.2.2 to address a security flaw.
HADOOP-12588:. Fix intermittent test failure of TestGangliaMetrics.
HADOOP-12589: Fix intermittent test failure of TestCopyPreserveFlag.
HADOOP-12590: TestCompressorDecompressor failing without stack traces.
HADOOP-12602:. TestMetricsSystemImpl#testQSize occasionally fails.
HADOOP-12603: TestSymlinkLocalFSFileContext#testSetTimesSymlinkToDir occasionally fail.
HADOOP-12604: Exception may be swallowed in KMSClientProvider.
HADOOP-12605: Fix intermittent failure of TestIPC.testIpcWithReaderQueuingi.
HADOOP-12608: Fix exception message in WASB when connecting with anonymous credential.
HADOOP-12609: Fix intermittent failure of TestDecayRpcScheduler.
HADOOP-12613:. TestFind.processArguments occasionally fails.
HADOOP-12617: SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal.
HADOOP-12622: Improve the loggings in RetryPolicies and RetryInvocationHandler.
HADOOP-12634: Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy.
HADOOP-12635:. Adding Append API support for WASB.
HADOOP-12636: Prevent ServiceLoader failure init for unused FileSystems.
HADOOP-12653: Use SO_REUSEADDR to avoid getting "Address already in use" when using kerberos and attempting to bind to any port on the local IP address.
HADOOP-12659:. Incorrect usage of config parameters in token manager of KMS.
HADOOP-12663: Remove Hard-Coded Values From FileSystem.java.
HADOOP-12666: Support Microsoft Azure Data Lake - as a file system in Hadoop.
HADOOP-12667:s3a to support createNonRecursive API.
HADOOP-12668: Support excluding weak Ciphers in HttpServer2 through ssl- server.conf.
HADOOP-12672: RPC timeout should not override IPC ping interval.
HADOOP-12675: Fix description about retention period in usage of expunge command.
HADOOP-12678: Handle empty rename pending metadata file during atomic rename in redo path.
HADOOP-12682:. Fix TestKMS#testKMSRestart* failure.
HADOOP-12688:. Fix deadlinks in Compatibility.md.
HADOOP-12689: S3 filesystem operations stopped working correctly.
HADOOP-12691:. Add CSRF Filter for REST APIs to Hadoop Common.
HADOOP-12691:. Move files to correct location.
HADOOP-12696: Add tests for S3Filesystem Contract.
HADOOP-12699: TestKMS#testKMSProvider intermittently fails during 'test rollover draining'.
HADOOP-12700: Remove unused import in TestCompressorDecompressor.java.
HADOOP-12706:. TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally.
HADOOP-12711:. Remove dependency on commons-httpclient for ServletUtil.
HADOOP-12715:. TestValueQueue#testgetAtMostPolicyALL fails intermittently.
HADOOP-12716: KerberosAuthenticator#doSpnegoSequence use incorrect class to determine isKeyTab in JDK8.
HADOOP-12717:. NPE when trying to rename a directory in Windows Azure Storage FileSystem.
HADOOP-12723:S3A: Add ability to plug in any AWSCredentialsProvider.
HADOOP-12731: Remove useless boxing/unboxing code.
HADOOP-12736:. TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out.
HADOOP-12738: Create unit test to automatically compare Common related classes and core-default.xml.
HADOOP-12751: While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple. (Bolke de Bruin via stevel).
HADOOP-12752:. Improve diagnostics/use of envvar/sysprop credential propagation.
HADOOP-12758:. Extend CSRF Filter with UserAgent Checks.
HADOOP-12761:. incremental maven build is not really incremental.
HADOOP-12765: HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections. .
HADOOP-12767: Update apache httpclient version to 4.5.2; httpcore to 4.4.4.
HADOOP-12772:. NetworkTopologyWithNodeGroup.getNodeGroup() can loop infinitely for invalid 'loc' values.
HADOOP-12773:. HBase classes fail to load with client/job classloader enabled.
HADOOP-12774: s3a should use UGI.getCurrentUser.getShortname() for username.
HADOOP-12776: Remove getaclstatus call for non-acl commands in getfacl.
HADOOP-12780:. During WASB atomic rename handle crash when one directory has been renamed but not file under it.
HADOOP-12782:. Faster LDAP group name resolution with ActiveDirectory.
HADOOP-12786:. "hadoop key" command usage is not documented.
HADOOP-12787: KMS SPNEGO sequence does not work with WEBHDFS.
HADOOP-12788: OpensslAesCtrCryptoCodec should log which random number generator is used.
HADOOP-12789:. log classpath of ApplicationClassLoader at INFO level.
HADOOP-12792:. TestUserGroupInformation#testGetServerSideGroups fails in chroot.
HADOOP-12793:. Write a new group mapping service guide.
HADOOP-12794: Support additional compression levels for GzipCodec.
HADOOP-12795:. KMS does not log detailed stack trace for unexpected errors.
HADOOP-12800:. Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs.
HADOOP-12801: Suppress obsolete S3FileSystem tests.
HADOOP-12804: Read Proxy Password from Credential Providers in S3 FileSystem.
HADOOP-12805:. Annotate CanUnbuffer with @InterfaceAudience.Public.
HADOOP-12807: S3AFileSystem should read AWS credentials from environment variables.
HADOOP-12810:. FileSystem#listLocatedStatus causes unnecessary RPC calls.
HADOOP-12817: Enable TLS v1.1 and 1.2.
HADOOP-12825: Log slow name resolutions.
HADOOP-12828: Print user when services are started.
HADOOP-12829: StatisticsDataReferenceCleaner swallows interrupt exceptions.
HADOOP-12831: LocalFS/FSOutputSummer NPEs in constructor if bytes per checksum set to 0.
HADOOP-12844:Recover when S3A fails on IOException in read().
HADOOP-12846:. Credential Provider Recursive Dependencies.
HADOOP-12847:. hadoop daemonlog should support https and SPNEGO for Kerberized cluster.
HADOOP-12849: TestSymlinkLocalFSFileSystem fails intermittently.
HADOOP-12851:. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders.
HADOOP-12853:. Change WASB documentation regarding page blob support.
HADOOP-12870:. Fix typo admininistration in CommandsManual.md.
HADOOP-12871:. Fix dead link to NativeLibraries.html in CommandsManual.md.
HADOOP-12872:. Fix formatting in ServiceLevelAuth.md.
HADOOP-12875: [Azure Data Lake] Support for contract test and unit test cases.
HADOOP-12878:. KMS SPNEGO sequence does not work with WEBHDFS.
HADOOP-12886: Exclude weak ciphers in SSLFactory through ssl-server.xml.
HADOOP-12891: S3AFileSystem should configure Multipart Copy threshold and chunk size.
HADOOP-12893:. Update LICENSE.txt and NOTICE.txt.
HADOOP-12893:. Update LICENSE.txt and NOTICE.txt.
HADOOP-12893:. Update LICENSE.txt and NOTICE.txt".
HADOOP-12895: SSLFactory#createSSLSocketFactory exception message is wrong.
HADOOP-12901:. Add warning log when KMSClientProvider cannot create a connection to the KMS server.
HADOOP-12902:. JavaDocs for SignerSecretProvider are out-of-date in AuthenticationFilter.
HADOOP-12903: IPC Server should allow suppressing exception logging by type,.
HADOOP-12903: IPC Server should allow suppressing exception logging by type, not log 'server too busy' messages.
HADOOP-12906: AuthenticatedURL should convert a 404/Not Found into an FileNotFoundException.
HADOOP-12916:. Allow RPC scheduler/callqueue backoff using response times.
HADOOP-12926: lz4.c does not detect 64-bit mode properly.
HADOOP-12929:. JWTRedirectAuthenticationHandler must accommodate null expiration time.
HADOOP-12943: Add -w -r options in dfs -test command.
HADOOP-12947:. Update documentation Hadoop Groups Mapping to add static group mapping, negative cache.
HADOOP-12950: ShutdownHookManager should have a timeout for each of the Registered shutdown hook.
HADOOP-12954: Add a way to change hadoop.security.token.service.use_ip (rkanter) .
HADOOP-12958: PhantomReference for filesystem statistics can trigger OOM. .
HADOOP-12962:. KMS key names are incorrectly encoded when creating key.
HADOOP-12963:Allow using path style addressing for accessing the s3 endpoint.
HADOOP-12964: Http server vulnerable to clickjacking.
HADOOP-12973: Make DU pluggable.
HADOOP-12975: Add jitter to CachingGetSpaceUsed's thread.
HADOOP-12977:s3a to handle delete("/", true) robustly.
HADOOP-12982:Document missing S3A and S3 properties.
HADOOP-12984:. Add GenericTestUtils.getTestDir method and use it for emporary directory in tests (Contributed by Steve Loughran and Vinayakumar B.
HADOOP-12985:. Support MetricsSource interface for DecayRpcScheduler Metrics.
HADOOP-12989:. Some tests in org.apache.hadoop.fs.shell.find occasionally time out.
HADOOP-12993: Change ShutdownHookManger complete shutdown log from INFO to DEBUG.
HADOOP-12994:. Specify PositionedReadable, add contract tests, fix problems.
HADOOP-13008:. Add XFS Filter for UIs to Hadoop Common.
HADOOP-13011: Clearly Document the Password Details for Keystore-based Credential Providers.
HADOOP-13018: Make Kdiag check whether hadoop.token.files points to existent and valid files.
HADOOP-13024: Distcp with -delete feature on raw data not implemented.
HADOOP-13026: Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator.
HADOOP-13028:add low level counter metrics for S3A; use in read performance tests. contributed by: stevel patch includes.
HADOOP-13030: Handle special characters in passwords in KMS startup script.
HADOOP-13034: Log message about input options in distcp lacks some items.
HADOOP-13037: Refactor Azure Data Lake Store as an independent FileSystem.
HADOOP-13039:. Add documentation for configuration property ipc.maximum.data.length.
HADOOP-13042:. Restore lost leveldbjni LICENSE and NOTICE changes.
HADOOP-13047:S3a Forward seek in stream length to be configurable.
HADOOP-13052:. ChecksumFileSystem mishandles crc file permissions.
HADOOP-13058:S3A FS fails during init against a read-only FS if multipart purge.
HADOOP-13065: Add a new interface for retrieving FS and FC Statistics.
HADOOP-13072: WindowsGetSpaceUsed constructor should be public.
HADOOP-13079: Add -q option to Ls to print ? instead of non-printable characters.
HADOOP-13084: Fix ASF License warnings in branch-2.7.
HADOOP-13098:. Dynamic LogLevel setting page should accept case-insensitive log level string.
HADOOP-13103: Group resolution from LDAP may fail on javax.naming.ServiceUnavailableException.
HADOOP-13105:. Support timeouts in LDAP queries in LdapGroupsMapping.
HADOOP-13113:Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel.
HADOOP-13116:Jets3tNativeS3FileSystemContractTest does not run.
HADOOP-13122:Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.
HADOOP-13130: s3a failures can surface as RTEs, not IOEs.
HADOOP-13131:. Add tests to verify that S3A supports SSE-S3 encryption.
HADOOP-13138: Unable to append to a SequenceFile with Compression.NONE.
HADOOP-13139:Branch-2: S3a to use thread pool that blocks clients.
HADOOP-13140: FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes.
HADOOP-13145:In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.
HADOOP-13146: Refactor RetryInvocationHandler.
HADOOP-13155:. Implement TokenRenewer to renew and cancel delegation tokens in KMS.
HADOOP-13158:S3AFileSystem#toString might throw NullPointerException due to null cannedACL.
HADOOP-13159:. Fix potential NPE in Metrics2 source for DecayRpcScheduler.
HADOOP-13162:. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs.
HADOOP-13163:Reuse pre-computed filestatus in Distcp-CopyMapper.
HADOOP-13164:Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories.
HADOOP-13168: Support Future.get with timeout in ipc async calls.
HADOOP-13169: Randomize file list in SimpleCopyListing.
HADOOP-13171: Add StorageStatistics to S3A; instrument some more operations.
HADOOP-13179: GenericOptionsParser is not thread-safe because commons-cli OptionBuilder is not thread-safe.
HADOOP-13183: S3A proxy tests fail after httpclient/httpcore upgrade.
HADOOP-13188:S3A file-create should throw error rather than overwrite directories.
HADOOP-13192:. org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly.
HADOOP-13197:. Add non-decayed call metrics for DecayRpcScheduler.
HADOOP-13202: Avoid possible overflow in org.apache.hadoop.util.bloom.BloomFilter#getNBytes. .
HADOOP-13203: S3a: Consider reducing the number of connection aborts by setting correct length in s3 request.
HADOOP-13204: Add support for SSE-KMS and SSE-C in s3a filesystem.
HADOOP-13207: Specify FileSystem listStatus, listFiles and RemoteIterator.
HADOOP-13208: S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories.
HADOOP-13212:Provide an option to set the socket buffers in S3AFileSystem.
HADOOP-13213: Small Documentation bug with AuthenticatedURL in hadoop-auth.
HADOOP-13237:: s3a initialization against public bucket fails if caller lacks any credentials.
HADOOP-13239:Deprecate s3:// in branch-2.
HADOOP-13241: document s3a better.
HADOOP-13242: Authenticate to Azure Data Lake using client ID and keys.
HADOOP-13244: o.a.h.ipc.Server#Server should honor handlerCount when queueSizePerHandler is specified in consturctor.
HADOOP-13249: RetryInvocationHandler need wrap InterruptedException in IOException when call Thread.sleep.
HADOOP-13251:. Authenticate with Kerberos credentials when renewing KMS delegation token.
HADOOP-13252: Tune S3A provider plugin mechanism.
HADOOP-13254: Create framework for configurable disk checkers (yufeigu via rkanter) .
HADOOP-13255:. KMSClientProvider should check and renew tgt when doing delegation token operations.
HADOOP-13257: Improve Azure Data Lake contract tests.
HADOOP-13263:. Reload cached groups in background after expiry.
HADOOP-13270:. BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks.
HADOOP-13280: FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps.
HADOOP-13283: Support reset operation for new global storage statistics and per FS storage stats.
HADOOP-13285:. DecayRpcScheduler MXBean should only report decayed CallVolumeSummary.
HADOOP-13287: TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +.
HADOOP-13288: Guard null stats key in FileSystemStorageStatistics.
HADOOP-13290: Appropriate use of generics in FairCallQueue.
HADOOP-13291: Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented.
HADOOP-13299: JMXJsonServlet is vulnerable to TRACE. (Haibo Chen via kasha) .
HADOOP-13305: Define common statistics names across schemes.
HADOOP-13309: Document S3A known limitations in file ownership and permission model.
HADOOP-13315: FileContext#umask is not initialized properly.
HADOOP-13316: Enforce Kerberos authentication for required ops in DelegationTokenAuthenticator.
HADOOP-13320: Fix arguments check in documentation for WordCount v2.0.
HADOOP-13323: Downgrade stack trace on FS load from Warn to debug.
HADOOP-13324: s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints).
HADOOP-13336:S3A to support per-bucket configuration.
HADOOP-13350:. Additional fix to LICENSE and NOTICE.
HADOOP-13351: TestDFSClientSocketSize buffer size tests are flaky.
HADOOP-13352: Make X-FRAME-OPTIONS configurable in HttpServer2.
HADOOP-13353: LdapGroupsMapping getPassward shouldn't return null when IOException throws. .
HADOOP-13354: Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients.
HADOOP-13368: DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation.
HADOOP-13381: KMS clients should use KMS Delegation Tokens from current UGI.
HADOOP-13382: Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Ported to Hortonworks HDP-2.5 by Matt Foley.
HADOOP-13387:users always get told off for using S3 —even when not using it.
HADOOP-13389:TestS3ATemporaryCredentials.testSTS error when using IAM credentials.
HADOOP-13392: [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured.
HADOOP-13393: Omit unsupported fs.defaultFS setting in ADLS documentation.
HADOOP-13403: AzureNativeFileSystem rename/delete performance improvements.
HADOOP-13405:doc for fs.s3a.acl.default indicates incorrect values.
HADOOP-13406:S3AFileSystem: Consider reusing filestatus in delete() and mkdirs().
HADOOP-13422: ZKDelegationTokenSecretManager JaasConfig does not work well with other ZK users in process.
HADOOP-13433:Race in UGI.reloginFromKeytab.
HADOOP-13434: Add quoting to Shell class.
HADOOP-13443: KMS should check the type of underlying keyprovider of KeyProviderExtension before falling back to default. .
HADOOP-13446: Support running isolated unit tests separate from AWS integration tests.
HADOOP-13447: Refactor S3AFileSystem to support introduction of separate metadata repository and tests.
HADOOP-13458: LoadBalancingKMSClientProvider#doOp should log IOException stacktrace. .
HADOOP-13459: hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time.
HADOOP-13461: NPE in KeyProvider.rollNewVersion. .
HADOOP-13466: Add an AutoCloseableLock class. (Chen Liang) .
HADOOP-13467: Shell#getSignalKillCommand should use the bash builtin on Linux.
HADOOP-13470: GenericTestUtils$LogCapturer is flaky.
HADOOP-13494: ReconfigurableBase can log sensitive information.
HADOOP-13502: Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests.
HADOOP-13512: ReloadingX509TrustManager should keep reloading in case of exception.
HADOOP-13513: Java 1.7 support for org.apache.hadoop.fs.azure testcases.
HADOOP-13535: Add jetty6 acceptor startup issue workaround to branch-2. .
HADOOP-13540:improve section on troubleshooting s3a auth problems.
HADOOP-13541:explicitly declare the Joda time version S3A depends on.
HADOOP-13546: Override equals and hashCode to avoid connection leakage.
HADOOP-13552: RetryInvocationHandler logs all remote exceptions.
HADOOP-13552: RetryInvocationHandler logs all remote exceptions.
HADOOP-13558: UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket.
HADOOP-13560: S3ABlockOutputStream to support huge (many GB) file writes.
HADOOP-13565: KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request.
HADOOP-13590: Retry until TGT expires even if the UGI renewal thread encountered exception. .
HADOOP-13599: s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown.
HADOOP-13614: Purge some superfluous/obsolete S3 FS tests that are slowing test runs down.
HADOOP-13638: KMS should set UGI's Configuration object properly.
HADOOP-13674: S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.
HADOOP-13675: Bug in return value for delete() calls in WASB.
HADOOP-13680: fs.s3a.readahead.range to use getLongBytes.
HADOOP-13686: Adding additional unit test for Trash (I).
HADOOP-13692: hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.
HADOOP-13707: If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed. .
HADOOP-13710: Supress CachingGetSpaceUsed from logging interrupted exception stacktrace. .
HADOOP-13716: Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup.
HADOOP-13727: S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. .
HADOOP-13735:ITestS3AFileContextStatistics.testStatistics() failing.
HADOOP-13737: Cleanup DiskChecker interface. .
HADOOP-13738: DiskChecker should perform some disk IO. .
HADOOP-13742: Expose NumOpenConnectionsPerUser as a metric. Brahma Reddy Battula. .
HADOOP-13749: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.
HADOOP-13782: Make MutableRates metrics thread-local write, aggregate-on-read. .
HADOOP-13792: Stackoverflow for schemeless defaultFS with trailing slash.
HADOOP-13801:regression: ITestS3AMiniYarnCluster failing.
HADOOP-13804: MutableStat mean loses accuracy if add(long, long) is used. .
HADOOP-13822: Use GlobalStorageStatistics.INSTANCE.reset() at FileSystem#clearStatistics(). Contribute by Brahma Reddy Battula .
HADOOP-13823: s3a rename: fail if dest file exists.
HADOOP-13824: FsShell can suppress the real error if no error message is present. .
HADOOP-13826: S3A Deadlock in multipart copy due to thread pool limits.
HADOOP-13830:Intermittent failure of ITestS3NContractRootDir.testRecursiveRootListing "Can not create a Path from an empty string". .
HADOOP-13831: Correct check for error code to detect Azure Storage Throttling and provide retries.
HADOOP-13838: KMSTokenRenewer should close providers.
HADOOP-13847: KMSWebApp should close KeyProviderCryptoExtension. .
HADOOP-13855: Fix a couple of the s3a statistic names to be consistent with the rest.
HADOOP-13857: S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs.
HADOOP-13863: Azure: Add a new SAS key mode for WASB.
HADOOP-13871: ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful.
HADOOP-13890: Maintain HTTP/host as SPNEGO SPN support and fix KerberosName parsing.
HADOOP-13928: TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime error.
HADOOP-13929: ADLS connector should not check in contract-test-options.xml.
HADOOP-13930: Azure: Add Authorization support to WASB.
HADOOP-13945: Azure: Add Kerberos and Delegation token support to WASB client.
HADOOP-13956: Read ADLS credentials from Credential Provider.
HADOOP-13962: Update ADLS SDK to 2.1.4.
HADOOP-13988: KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser.
HADOOP-14017: User friendly name for ADLS user and group.
HADOOP-14019: Fix some typos in the s3a docs.
HADOOP-14028: S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.
HADOOP-14029: Fix KMSClientProvider for non-secure proxyuser use case.
HADOOP-14038: Rename ADLS credential properties.
HADOOP-14048: REDO operation of WASB#AtomicRename should create placeholder blob for destination folder.
HADOOP-14049: Honour AclBit flag associated to file/folder permission for Azure datalake account.
HADOOP-14055: SwiftRestClient includes pass length in exception if auth fails.
HADOOP-14081:S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock).
HADOOP-14087: S3A typo in pom.xml test exclusions.
HADOOP-14091: AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari.
HADOOP-14092: [DOC] Typo in hadoop-aws index.md.
HADOOP-14092: Typo in hadoop-aws index.md.
HADOOP-14099: Split S3 testing documentation out into its own file.
HADOOP-14100: Upgrade Jsch jar to latest version to fix vulnerability in old versions.
HADOOP-14102: Relax error message assertion in S3A test ITestS3AEncryptionSSEC.
HADOOP-14113: Review ADL Docs.
HADOOP-14123: Remove misplaced ADL service provider config file for FileSystem.
HADOOP-14135: Remove URI parameter in AWSCredentialProvider constructors.
HADOOP-14138:Remove S3A ref from META-INF service discovery, rely on existing core-default entry.
HADOOP-14150: Implement getHomeDirectory() method in NativeAzureFileSystem.
HADOOP-14153: ADL module has messed doc structure.
HADOOP-14170: FileSystemContractBaseTest is not cleaning up test directory clearly.
HADOOP-14173: Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE.
HADOOP-14174: Set default ADLS access token provider type to ClientCredential.
HADOOP-14190: Add more on S3 regions to the s3a documentation.
HADOOP-14195: CredentialProviderFactory$getProviders is not thread-safe.
HADOOP-14196: Azure Data Lake doc is missing required config entry.
HADOOP-14197: Fix ADLS doc for credential provider.
HADOOP-14204: S3A multipart commit failing, "UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".
HADOOP-14204: S3A multipart commit failing, "UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".
HADOOP-14205: No FileSystem for scheme: adl.
HADOOP-14214: DomainSocketWatcher::add()/delete() should not self interrupt while looping await().
HADOOP-14230: TestAdlFileSystemContractLive fails to clean up.
HADOOP-14241: Add ADLS sensitive config keys to default list.
HADOOP-14255: HADOOP-14428. S3A to delete unnecessary fake directory objects in mkdirs().
HADOOP-14255: S3A to delete unnecessary fake directory objects in mkdirs().
HADOOP-14256: Correct the format for "Seoul" example.
HADOOP-14268: Fix markdown itemization in hadoop-aws documents.
HADOOP-14274: Azure: Simplify Ranger-WASB policy model.
HADOOP-14305: S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus.
HADOOP-14321: explicitly exclude s3a root dir ITests from parallel runs.
HADOOP-14324: Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics.
HADOOP-14349: Rename ADLS CONTRACT_ENABLE_KEY.
HADOOP-14351: Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache.
HADOOP-14361:Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently.
HADOOP-14369: NetworkTopology calls expensive toString() when logging.
HADOOP-14428: s3a: mkdir appears to be broken.
HADOOP-14442: Owner support for ranger-wasb integration.
HADOOP-14443: Azure: Support retry and client side failover for authorization, SASKey and delegation token generation.3.
HADOOP-14464: hadoop-aws doc header warning #5 line wrapped.
HADOOP-14465: LdapGroupsMapping - support user and group search base.
HADOOP-14472: Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently.
HADOOP-14478: Optimize NativeAzureFsInputStream for positional reads.
HADOOP-14500: Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails.
HADOOP-14512: WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename.
HADOOP-14520: WASB: Block compaction for Azure Block Blobs.
HADOOP-14535:wasb: implement high-performance random access and seek of block blobs.
HADOOP-14536: Update azure-storage sdk to version 5.3.0.
HADOOP-14543: ZKFC should use getAversion() while setting the zkacl.
HADOOP-14546: Azure: Concurrent I/O does not work when secure.mode is enabled.
HADOOP-14547: [WASB] the configured retry policy is not used for all storage operations.'.
HADOOP-14573: regression: Azure tests which capture logs failing with move to SLF4J.
HADOOP-14583: wasb throws an exception if you try to create a file and there's no parent directory.
HADOOP-14596: AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error.
HADOOP-14596: AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error.Contributed by Steve Loughran.
HADOOP-14609: NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code.
HADOOP-14640: Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation.
HADOOP-14642: wasb: add support for caching Authorization and SASKeys.
HADOOP-14658: branch-2 compilation is broken in hadoop-azure.
HADOOP-14660: wasb: improve throughput by 34% when account limit exceeded.
HADOOP-14662: Update azure-storage sdk to version 5.4.0.
HADOOP-14680: Azure: IndexOutOfBoundsException in BlockBlobInputStream.'.
HADOOP-14722: Azure: BlockBlobInputStream position incorrect after seek.
HADOOP-14768: Honoring sticky bit during Deletion when authorization is enabled in WASB.
HADOOP-14769: WASB: delete recursive should not fail if a file is deleted.
HADOOP-14802: Add support for using container saskeys for all accesses.
HADOOP-14820: Wasb mkdirs security checks inconsistent with HDFS.
HADOOP-14839: DistCp log output should contain copied and deleted files and directories.
HADOOP-14845: Azure wasb: getFileStatus not making any auth checks.
HADOOP-14899: Restrict Access to setPermission operation when authorization is enabled in WASB.
HADOOP-14910: Upgrade netty-all jar to latest 4.0.x.Final.
HADOOP-14913: Sticky bit implementation for rename() operation in Azure WASB.
HADOOP-14920: KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
HADOOP-14935: Azure: POSIX permissions are taking effect in access() method even when authorization is enabled.
HDFS-27: HDFS CLI with --config set to default config complains log file not found error.
HDFS-2580:. NameNode#main(...) can make use of GenericOptionsParser.
HDFS-2043:. TestHFlush failing intermittently.
HDFS-3677: dfs.namenode.edits.dir.required is missing from hdfs- default.xml.
HDFS-3918: EditLogTailer shouldn't log WARN when other node is in standby mode. .
HDFS-4015: Safemode should count and report orphaned blocks.
HDFS-4176: EditLogTailer should call rollEdits with a timeout. .
HDFS-4185: Add a metric for number of active leases.
HDFS-4366: Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks.
HDFS-4396: Add START_MSG/SHUTDOWN_MSG for ZKFC.
HDFS-4681: TestBlocksWithNotEnoughRacks#testCorruptBlockRereplicatedAcrossRacks fails using IBM java.
HDFS-4937: ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom.
HDFS-4946:. Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable.
HDFS-5640: Add snapshot methods to FileContext.
HDFS-6054: MiniQJMHACluster should not use static port to avoid binding failure in unit test.
HDFS-6101: TestReplaceDatanodeOnFailure fails occasionally.
HDFS-6184: Capture NN's thread dump when it fails over.
HDFS-6407: Add sorting and pagination in the datanode tab of the NN Web UI.
HDFS-6481: DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs.
HDFS-6533: TestBPOfferService#testBasicFunctionalitytest fails intermittently.
HDFS-6581: merge.15.patch to Champlain. This is a consolidated patch containing all feature changes and was applied to trunk.
HDFS-6581: Support for writing to single replica in RAM. Applied.
HDFS-6666: Abort NameNode and DataNode startup if security is enabled but block access token is not enabled.
HDFS-6763: Initialize file system-wide quota once on transitioning to active.
HDFS-6832: Fix the usage of 'hdfs namenode' command.
HDFS-6860: BlockStateChange logs are too noisy.
HDFS-6874: Add GETFILEBLOCKLOCATIONS operation to HttpFS.
HDFS-6888: Allow selectively audit logging ops.
HDFS-6945: BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed.
HDFS-7163:. WebHdfsFileSystem should retry reads according to the configured retry policy.
HDFS-7166:. SbNN Web UI shows #Under replicated blocks and #pending deletion blocks.
HDFS-7284: Add more debug info to BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas. (Wei-Chiu Chuang via Yongjun Zhang) .
HDFS-7314:. When the DFSClient lease cannot be renewed, abort open-for- write files rather than the entire DFSClient.
HDFS-7390: Provide JMX metrics per storage type.
HDFS-7452: skip StandbyException log for getCorruptFiles().
HDFS-7483: Display information per tier on the Namenode UI.
HDFS-7597: DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping.
HDFS-7608: hdfs dfsclient newConnectedPeer has no write timeout.
HDFS-7609: Avoid retry cache collision when Standby NameNode loading edits.
HDFS-7645: Fix CHANGES.txt.
HDFS-7645: Rolling upgrade is restoring blocks from trash multiple times.
HDFS-7701: Support reporting per storage type quota and usage with hadoop/hdfs shell.
HDFS-7725: Incorrect "nodes in service" metrics caused all writes to fail.
HDFS-7758: Retire FsDatasetSpi#getVolumes() and use FsDatasetSpi#getVolumeRefs() instead (Lei (Eddy) Xu via Colin P. McCabe) .
HDFS-7858: Improve HA Namenode Failover detection on the client.
HDFS-7890: Improve information on Top users for metrics in RollingWindowsManager and lower log level.
HDFS-7923: The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages.
HDFS-7928: Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy.
HDFS-7933: fsck should also report decommissioning replicas.
HDFS-7964: Add support for async edit logging. .
HDFS-7978:. Add LOG.isDebugEnabled() guard for some LOG.debug(..).
HDFS-7990: IBR delete ack should not be delayed.
HDFS-8008: Support client-side back off when the datanodes are congested.
HDFS-8008: Support client-side back off when the datanodes are congested.
HDFS-8009: Signal congestion on the DataNode.
HDFS-8009: Signal congestion on the DataNode.
HDFS-8046: Allow better control of getContentSummary.
HDFS-8055: NullPointerException when topology script is missing.
HDFS-8071:.(Contributed by Tsz Wo Nicholas Sze).
HDFS-8099: Change "DFSInputStream has been closed already" message to debug log level.
HDFS-8101:. DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS classes at runtime.
HDFS-8113: Add check for null BlockCollection pointers in BlockInfoContiguous structures.
HDFS-8131: Implement a space balanced block placement policy.
HDFS-8144: Split TestLazyPersistFiles into multiple tests.
HDFS-8152: Refactoring of lazy persist storage cases.
HDFS-8155: Support OAuth2 in WebHDFS.
HDFS-8180: AbstractFileSystem Implementation for WebHdfs.
HDFS-8205: CommandFormat#parse() should not parse option as value of option.
HDFS-8209: Support different number of datanode directories in MiniDFSCluster.
HDFS-8211: DataNode UUID is always null in the JMX counter.
HDFS-8219: setStoragePolicy with folder behavior is different after cluster restart.
HDFS-8224: Schedule a block for scanning if its metadata file is corrupt. .
HDFS-8229: LAZY_PERSIST file gets deleted after NameNode restart.
HDFS-8232: Missing datanode counters when using Metrics2 sink interface.
HDFS-8247: TestDiskspaceQuotaUpdate#testAppendOverTypeQuota is failing. fixing BUG-71395 by backporting HDFS-8231 .
HDFS-8276: LazyPersistFileScrubber should be disabled if scrubber interval configured zero.
HDFS-8278: When computing max-size-to-move in Balancer, count only the storage with remaining >= default block size. .
HDFS-8307: Spurious DNS Queries from hdfs shell.
HDFS-8309: Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows.
HDFS-8310: Fix TestCLI.testAll "help: help for find" on Windows.
HDFS-8311: DataStreamer.transfer() should timeout the socket InputStream.
HDFS-8312: Added permission check for moving file to Trash.
HDFS-8337: Accessing httpfs via webhdfs doesn't work from a jar with kerberos.
HDFS-8345: Storage policy APIs must be exposed via the FileSystem interface.
HDFS-8384: Allow NN to startup if there are files having a lease but are not under construction.
HDFS-8429: Avoid stuck threads if there is an error in DomainSocketWatcher that stops the thread.
HDFS-8431: hdfs crypto class not found in Windows.
HDFS-8432: Introduce a minimum compatible layout version to allow downgrade in more rolling upgrade use cases.
HDFS-8435: Support CreateFlag in WebHDFS.
HDFS-8492: DN should notify NN when client requests a missing block (Contributed by Walter Su) .
HDFS-8539: Hdfs doesnt have class 'debug' in windows.
HDFS-8542: WebHDFS getHomeDirectory behavior does not match specification.
HDFS-8546: Use try with resources in DataStorage and Storage.
HDFS-8548:. Minicluster throws NPE on shutdown.
HDFS-8549: Abort the balancer if an upgrade is in progress.
HDFS-8554: TestDatanodeLayoutUpgrade fails on Windows.
HDFS-8573: Move creation of restartMeta file logic from BlockReceiver to ReplicaInPipeline. .
HDFS-8578:. On upgrade, Datanode should process all storage/data dirs in parallel. .
HDFS-8581:. ContentSummary on / skips further counts on yielding lock.
HDFS-8593: Calculation of effective layout version mishandles comparison to current layout version in storage.
HDFS-8615:. Correct HTTP method in WebHDFS document.
HDFS-8633:. Fix setting of dfs.datanode.readahead.bytes in hdfs-default.xml to match DFSConfigKeys.
HDFS-8642: Make TestFileTruncate more reliable.
HDFS-8647: Abstract BlockManager's rack policy into BlockPlacementPolicy.
HDFS-8656: Preserve compatibility of ClientProtocol#rollingUpgrade after finalization.
HDFS-8659: Block scanner INFO message is spamming logs.
HDFS-8674: Improve performance of postponed block scans. .
HDFS-8676: Delayed rolling upgrade finalization can cause heartbeat expiration.
HDFS-8696: Make the lower and higher watermark in the DN Netty server configurable.
HDFS-8709:. Clarify automatic sync in FSEditLog#logEdit.
HDFS-8711: setSpaceQuota command should print the available storage type when input storage type is wrong.
HDFS-8722: Optimize datanode writes for small writes and flushes.
HDFS-8729: Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately which occasionally failed.
HDFS-8767: RawLocalFileSystem.listStatus() returns null for UNIX pipefile.
HDFS-8772: Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails. (Walter Su via wang).
HDFS-8778: TestBlockReportRateLimiting#testLeaseExpiration can deadlock.
HDFS-8785: TestDistributedFileSystem is failing in trunk.
HDFS-8792: BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) .
HDFS-8797: WebHdfsFileSystem creates too many connections for pread.
HDFS-8805: Archival Storage: getStoragePolicy should not need superuser privilege.
HDFS-8806: Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared.
HDFS-8809: HDFS fsck reports under construction blocks as CORRUPT.
HDFS-8815: DFS getStoragePolicy implementation using single RPC call.
HDFS-8816: Improve visualization for the Datanode tab in the NN UI.
HDFS-8818: Changes the global moveExecutor to per datanode executors and changes MAX_SIZE_TO_MOVE to be configurable. .
HDFS-8824: Do not use small blocks for balancing the cluster. .
HDFS-8826: In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. .
HDFS-8828:. Utilize Snapshot diff report to build diff copy list in distcp.
HDFS-8829: Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning.
HDFS-8831:. Trash Support for deletion in HDFS encryption zone.
HDFS-8844: TestHDFSCLI does not cleanup the test directory.
HDFS-8845:. DiskChecker should not traverse the entire tree.
HDFS-8846: Add a unit test for INotify functionality across a layout version upgrade.
HDFS-8850: VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks.
HDFS-8852:. HDFS architecture documentation of version 2.x is outdated about append write support.
HDFS-8855: Webhdfs client leaks active NameNode connections.
HDFS-8856: Make LeaseManager#countPath O(1).
HDFS-8859: Improve DataNode ReplicaMap memory footprint to save about 45%. .
HDFS-8860: Remove unused Replica copyOnWrite code (Lei (Eddy) Xu via Colin P. McCabe) .
HDFS-8863: The remaining space check in BlockPlacementPolicyDefault is flawed.
HDFS-8867: Enable optimized block reports.
HDFS-8879: Quota by storage type usage incorrectly initialized upon namenode restart.
HDFS-8880: NameNode metrics logging.
HDFS-8883: NameNode Metrics : Add FSNameSystem lock Queue Length.
HDFS-8884: Fail-fast check in BlockPlacementPolicyDefault#chooseTarget. .
HDFS-8885: ByteRangeInputStream used in webhdfs does not override available().
HDFS-8887: Expose storage type and storage ID in BlockLocation.
HDFS-8891: HDFS concat should keep srcs order.
HDFS-8896: DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager.
HDFS-8898: Create API and command-line argument to get quota and quota usage without detailed content summary.
HDFS-8911: NameNode Metric : Add Editlog counters as a JMX metric.
HDFS-8914:. Document HA support in the HDFS HdfsDesign.md.
HDFS-8923: Add -source flag to balancer usage message. .
HDFS-8930: Block report lease may leak if the 2nd full block report comes when NN is still in safemode.
HDFS-8932: NPE thrown in NameNode when try to get TotalSyncCount metric before editLogStream initialization.
HDFS-8939: Test(S)WebHdfsFileContextMainOperations failing on branch-2.
HDFS-8946: Improve choosing datanode storage for block placement. .
HDFS-8950: NameNode refresh doesn't remove DataNodes that are no longer in the allowed list.
HDFS-8964:. When validating the edit log, do not read at or beyond the file offset that is being written.
HDFS-8965: Harden edit log reading code against out of memory errors.
HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.
HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932. (Contributed by Anu Engineer).
HDFS-8983: NameNode support for protected directories.
HDFS-8983: NameNode support for protected directories.
HDFS-8995: Flaw in registration bookeeping can make DN die on reconnect.
HDFS-9008: Balancer#Parameters class could use a builder pattern.
HDFS-9008:was not backported correctly.
HDFS-9009: Send metrics logs to NullAppender by default.
HDFS-9019: Adding informative message to sticky bit permission denied exception.
HDFS-9033:. dfsadmin -metasave prints "NaN" for cache used%.
HDFS-9034: StorageTypeStats Metric should not count failed storage.'.
HDFS-9038: DFS reserved space is erroneously counted towards non-DFS used. (Brahma Reddy Battula) .
HDFS-9042:. Update document for the Storage policy name.
HDFS-9043:. Doc updation for commands in HDFS Federation.
HDFS-9047:. Retire libwebhdfs.
HDFS-9048:. DistCp documentation is out-of-dated.
HDFS-9063: Correctly handle snapshot path for getContentSummary.
HDFS-9072: Fix random failures in TestJMXGet.
HDFS-9082: Change the log level in WebHdfsFileSystem.initialize() from INFO to DEBUG.
HDFS-9083: Replication violates block placement policy.
HDFS-9106: Transfer failure during pipeline recovery causes permanent write failures.
HDFS-9107: Prevent NNs unrecoverable death spiral after full GC.
HDFS-9112: Improve error message for Haadmin when multiple name service IDs are configured.
HDFS-9128: TestWebHdfsFileContextMainOperations and TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows.
HDFS-9130: Use GenericTestUtils#setLogLevel to the logging level. .
HDFS-9142: Separating Configuration object for namenode(s) in MiniDFSCluster.
HDFS-9145: Tracking methods that hold FSNamesytemLock for too long. .
HDFS-9148: Incorrect assert message in TestWriteToReplica#testWriteToTemporary (Tony Wu via Lei (Eddy) Xu) .
HDFS-9175: Change scope of 'AccessTokenProvider.getAccessToken()' and 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to public.
HDFS-9178: Slow datanode I/O can cause a wrong node to be marked bad.
HDFS-9184: Logging HDFS operation's caller context into audit logs.
HDFS-9188: Make block corruption related tests FsDataset-agnostic.
HDFS-9193: Fix incorrect references the usages of the DN in dfshealth.js.
HDFS-9198:. Coalesce IBR processing in the NN.
HDFS-9205: Do not schedule corrupt blocks for replication. .
HDFS-9210: Fix some misuse of %n in VolumeScanner#printStats.
HDFS-9220: Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum.
HDFS-9221: HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array.
HDFS-9226: Unit Tests failing due to mockito version issue.
HDFS-9239: DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness.
HDFS-9251: Refactor TestWriteToReplica and TestFsDatasetImpl to avoid explicitly creating Files in the tests code. .
HDFS-9259:. Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario.
HDFS-9264: Minor cleanup of operations on FsVolumeList#volumes. (Walter Su via lei) .
HDFS-9273: ACLs on root directory may be lost after NN restart.
HDFS-9276: Failed to Update HDFS Delegation Token for long running application in HA mode.
HDFS-9289: Make DataStreamer#block thread safe and verify genStamp in commitBlock.
HDFS-9290: DFSClient#callAppend() is not backward compatible for slightly older NameNodes.
HDFS-9294: DFSClient deadlock when close file and failed to renew lease.
HDFS-9305: Delayed heartbeat processing causes storm of subsequent heartbeats.
HDFS-9311: Support optional offload of NameNode HA service health checks to a separate RPC server.
HDFS-9313: Possible NullPointerException in BlockManager if no excess replica can be chosen.
HDFS-9314: Improve BlockPlacementPolicyDefault's picking of excess replicas.
HDFS-9317:. Document fsck -blockId and -storagepolicy options in branch-2: 7.
HDFS-9343: Empty caller context considered invalid.
HDFS-9347: Invariant assumption in TestQuorumJournalManager.shutdown() is wrong.
HDFS-9350: Avoid creating temprorary strings in Block.toString() and getBlockName() (Staffan Friberg via cmccabe) .
HDFS-9354: Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows.
HDFS-9357: NN UI renders icons of decommissioned DN incorrectly.
HDFS-9358: TestNodeCount#testNodeCount timed out.
HDFS-9362: TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
HDFS-9364: Unnecessary DNS resolution attempts when creating NameNodeProxies.
HDFS-9383: TestByteArrayManager#testByteArrayManager fails.
HDFS-9384: TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server.
HDFS-9395:. Make HDFS audit logging consistant.
HDFS-9397: Fix typo for readChecksum() LOG.warn in BlockSender.java.
HDFS-9402: Switch DataNode.LOG to use slf4j.
HDFS-9406: FSImage may get corrupted after deleting snapshot.
HDFS-9412: getBlocks occupies FSLock and takes too long to complete.
HDFS-9413: getContentSummary() on standby should throw StandbyException.
HDFS-9415:. Document dfs.cluster.administrators and dfs.permissions.superusergroup.
HDFS-9426: Rollingupgrade finalization is not backward compatible.
HDFS-9428: Fix intermittent failure of TestDNFencing.testQueueingWithAppend.
HDFS-9431: DistributedFileSystem#concat fails if the target path is relative.
HDFS-9434: Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messags.
HDFS-9445: Datanode may deadlock while handling a bad volume.
HDFS-9466: TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky.
HDFS-9467: Fix data race accessing writeLockHeldTimeStamp in FSNamesystem. .
HDFS-9470:. Encryption zone on root not loaded from fsimage after NN restart. Xiao Chen via wang.
HDFS-9476: TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail.
HDFS-9478: Reason for failing ipc.FairCallQueue contruction should be thrown.
HDFS-9484: NNThroughputBenchmark$BlockReportStats should not send empty block reports.
HDFS-9493: Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk.
HDFS-9500: Fix software version counts for DataNodes during rolling upgrade. .
HDFS-9505:. HDFS Architecture documentation needs to be refreshed.
HDFS-9516:. Truncate file fails with data dirs on multiple disks.
HDFS-9517: Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes.
HDFS-9521: TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk.
HDFS-9525: hadoop utilities need to support provided delegation tokens.
HDFS-9530:. ReservedSpace is not cleared for abandoned Blocks.
HDFS-9533:. seen_txid in the shared edits directory is modified during bootstrapping.
HDFS-9534: Add CLI command to clear storage policy from a path.
HDFS-9549: TestCacheDirectives#testExceedsCapacity is flaky.
HDFS-9555:. LazyPersistFileScrubber should still sleep if there are errors in the clear progress.
HDFS-9557: Reduce object allocation in PB conversion. (Daryn Sharp via cnauroth).
HDFS-9566: Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method.
HDFS-9569:. Log the name of the fsimage being loaded for better supportability.
HDFS-9569:'s message change.
HDFS-9572: Prevent DataNode log spam if a client connects on the data transfer port but sends no data.
HDFS-9574: Reduce client failures during datanode restart.
HDFS-9584: NPE in distcp when ssl configuration file does not exist in class path.
HDFS-9589: Block files which have been hardlinked should be duplicated before the DataNode appends to the them (cmccabe) .
HDFS-9600: Do not check replication if the block is under construction.
HDFS-9601: NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block.
HDFS-9605: Add links to failed volumes to explorer.html in HDFS Web UI.
HDFS-9608: Disk IO imbalance in HDFS with heterogeneous storages.
HDFS-9612:. DistCp worker threads are not terminated after jobs are done.
HDFS-9618: Fix mismatch between log level and guard in BlockManager#computeRecoveryWorkForBlocks.
HDFS-9619: SimulatedFSDataset sometimes can not find blockpool for the correct namenode.
HDFS-9623: Update example configuration of block state change log in log4j.properties.
HDFS-9624: DataNode start slowly due to the initial DU command operations.
HDFS-9625: set replication for empty file failed when set storage policy.
HDFS-9626: TestBlockReplacement#testBlockReplacement fails occasionally.
HDFS-9629:. Update the footer of Web UI to show year 2016.
HDFS-9630: DistCp minor refactoring and clean up.
HDFS-9634:. webhdfs client side exceptions don't provide enough details.
HDFS-9638: to branch-2.7.
HDFS-9644:. Update encryption documentation to reflect nested EZs.
HDFS-9655: NN should start JVM pause monitor before loading fsimage.
HDFS-9661: Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage moveBlockAcrossStorage and createRbw.
HDFS-9669:. TcpPeerServer should respect ipc.server.listen.queue.size.
HDFS-9670: DistCp throws NPE when source is root.
HDFS-9672: o.a.h.hdfs.TestLeaseRecovery2 fails intermittently.
HDFS-9688:. Test the effect of nested encryption zones in HDFS downgrade.
HDFS-9696: Garbage snapshot records linger forever.
HDFS-9700: BUG-60868. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets for DataTransferProtocol.
HDFS-9701: DN may deadlock when hot-swapping under load. (Xiao Chen via lei) .
HDFS-9706: Log more details in debug logs in BlockReceiver's constructor.
HDFS-9710: DN can be configured to send block receipt IBRs in batches.
HDFS-9711:. Integrate CSRF prevention filter in WebHDFS.
HDFS-9713: DataXceiver#copyBlock should return if block is pinned.
HDFS-9715: Check storage ID uniqueness on datanode startup.
HDFS-9721: Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference.
HDFS-9724: Degraded performance in WebHDFS listing as it does not reuse ObjectMapper.
HDFS-9726: Refactor IBR code to a new class. .
HDFS-9730:. Storage ID update does not happen when there is a layout change.
HDFS-9739: DatanodeStorage.isValidStorageId() is broken.
HDFS-9740: Use a reasonable limit in DFSTestUtil.waitForMetric().
HDFS-9743: Fix TestLazyPersistFiles#testFallbackToDiskFull in branch-2: 7.
HDFS-9748: Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice.
HDFS-9752: Permanent write failures may happen to slow writers during datanode rolling upgrades.
HDFS-9760: WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler.
HDFS-9764: DistCp doesn't print value for several arguments including -numListstatusThreads.
HDFS-9765:. TestBlockScanner#testVolumeIteratorWithCaching fails intermittently.
HDFS-9766:. TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently.
HDFS-9768: Reuse ObjectMapper instance in HDFS to improve the performance.
HDFS-9772:. TestBlockReplacement#testThrottler doesn't work as expected.
HDFS-9777: Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh).
HDFS-9779: . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value.
HDFS-9781: FsDatasetImpl#getBlockReports can occasionally throw NullPointerException. .
HDFS-9784:. Example usage is not correct in Transparent Encryption document.
HDFS-9790: HDFS Balancer should exit with a proper message if upgrade is not finalized.
HDFS-9797:. Log Standby exceptions thrown by RequestHedgingProxyProvider at DEBUG Level.
HDFS-9799:. Reimplement getCurrentTrashDir to remove incompatibility.
HDFS-9812:. Streamer threads leak if failure happens when closing DFSOutputStream.
HDFS-9839: Reduce verbosity of processReport logging.
HDFS-9842: dfs.datanode.balance.bandwidthPerSec should accept friendly size units.
HDFS-9844:. Correct path creation in getTrashRoot to handle root dir.
HDFS-9851: NameNode throws NPE when setPermission is called on a path that does not exist.
HDFS-9854:. Log cipher suite negotiation more verbosely.
HDFS-9864:. Correct reference for RENEWDELEGATIONTOKEN and CANCELDELEGATIONTOKEN in webhdfs doc.
HDFS-9865:. TestBlockReplacement fails intermittently in trunk.
HDFS-9871: "Bytes Being Moved" -ve(-1 B) when cluster was already balanced.
HDFS-9874:. Long living DataXceiver threads cause volume shutdown to block.
HDFS-9880:. TestDatanodeRegistration fails occasionally.
HDFS-9881:. DistributedFileSystem#getTrashRoot returns incorrect path for encryption zones.
HDFS-9882: Add heartbeatsTotal in Datanode metrics.
HDFS-9902: Support different values of dfs.datanode.du.reserved per storage type.
HDFS-9904:. testCheckpointCancellationDuringUpload occasionally fails.
HDFS-9905:. WebHdfsFileSystem#runWithRetry should display original stack trace on error.
HDFS-9906: Remove spammy log spew when a datanode is restarted.
HDFS-9917: IBR accumulate more objects when SNN was down for sometime.
HDFS-9926:. MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils.
HDFS-9941: Do not log StandbyException on NN, other minor logging fixes.
HDFS-9947: Block#toString should not output information from derived classes (cmccabe) .
HDFS-9949: Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared.
HDFS-9958:. BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages.
HFDS-9137:should be backported correctly. .
HDFS-10178:. Permanent write failures can happen if pipeline recoveries occur for the first packet.
HDFS-10182:. Hedged read might overwrite user's buf.
HDFS-10186:. DirectoryScanner: Improve logs by adding full path of both actual and expected block directories. .
HDFS-10189: PacketResponder#toString should include the downstreams for PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE.
HDFS-10199: Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing.
HDFS-10216:. Distcp -diff throws exception when handling relative path.
HDFS-10217: show 'blockScheduled' tooltip in datanodes table.
HDFS-10220: A large number of expired leases can make namenode unresponsive and cause failover.
HDFS-10223: peerFromSocketAndKey performs SASL exchange before setting connection timeouts.
HDFS-10225: DataNode hot swap drives should disallow storage type changes. .
HDFS-10228: TestHDFSCLI fails.
HDFS-10235: Last contact for Live Nodes should be relative time.
HDFS-10239:. Fsshell mv fails if port usage doesn't match in src and destination paths.
HDFS-10242: Cannot create space quota of zero.
HDFS-10245:. Fix the findbugs warnings in branch-2.7.
HDFS-10253:. Fix TestRefreshCallQueue failure.
HDFS-10261:. TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports.
HDFS-10264:. Logging improvements in FSImageFormatProtobuf.Saver.
HDFS-10270: TestJMXGet:testNameNode() fails.
HDFS-10271:. Extra bytes are getting released from reservedSpace for append.
HDFS-10275:. TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly.
HDFS-10277: PositionedReadable test testReadFullyZeroByteFile failing in HDFS.
HDFS-10279: Improve validation of the configured number of tolerated failed volumes. .
HDFS-10281: TestPendingCorruptDnMessages fails intermittently.
HDFS-10287: MiniDFSCluster should implement AutoCloseable.
HDFS-10291: TestShortCircuitLocalRead failing.
HDFS-10301: BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order .
HDFS-10301: Remove FBR tracking state to fix false zombie storage detection for interleaving block reports. .
HDFS-10309: Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga).
HDFS-10312: Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.
HDFS-10313:. Distcp need to enforce the order of snapshot names passed to -diff.
HDFS-10319:. Balancer should not try to pair storages with different types.
HDFS-10320: Rack failures may result in NN terminate. (Xiao Chen via mingma) .
HDFS-10324:. Trash directory in an encryption zone should be pre-created with correct permissions.
HDFS-10326: Disable setting tcp socket send/receive buffers for write pipelines.
HDFS-10329: Bad initialisation of StringBuffer in RequestHedgingProxyProvider.
HDFS-10330: Add Corrupt Blocks Information in Metasave output.
HDFS-10335:Mover$Processor#chooseTarget() always chooses the first matching target storage group. .
HDFS-10341: Add a metric to expose the timeout number of pending replication blocks.
HDFS-10342: BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt.
HDFS-10343: BlockManager#createLocatedBlocks may return blocks on failed storages.
HDFS-10344:. DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash.
HDFS-10347:. Namenode report bad block method doesn't log the bad block or datanode.
HDFS-10360: DataNode may format directory and lose blocks if current/VERSION is missing.
HDFS-10367: TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception.
HDFS-10372:. Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume.
HDFS-10377:. CacheReplicationMonitor shutdown log message should use INFO level.
HDFS-10390: Implement asynchronous setAcl/getAclStatus for DistributedFileSystem.
HDFS-10396: Using -diff option with DistCp may get "Comparison method violates its general contract" exception.
HDFS-10397: Distcp should ignore -delete option if -diff option is provided instead of exiting.
HDFS-10410: RedundantEditLogInputStream.LOG is set to wrong class.
HDFS-10415: TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called.
HDFS-10417: Improve error message from checkBlockLocalPathAccess.
HDFS-10423: Increase default value of httpfs maxHttpHeaderSize. .
HDFS-10424: DatanodeLifelineProtocol not able to use under security cluster.
HDFS-10431: Refactor and speedup TestAsyncDFSRename.
HDFS-10437: ReconfigurationProtocol not covered by HDFSPolicyProvider.
HDFS-10438: When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.
HDFS-10440: Improve DataNode web UI.
HDFS-10448: CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1.
HDFS-10455: Logging the username when deny the setOwner operation.
HDFS-10457: DataNode should not auto-format block pool directory if VERSION is missing. (Wei-Chiu Chuang via lei) .
HDFS-10458:. getFileEncryptionInfo should return quickly for non-encrypted cluster.
HDFS-10468:. HDFS read ends up ignoring an interrupt.
HDFS-10469:. Add number of active xceivers to datanode metrics.
HDFS-10471: DFSAdmin#SetQuotaCommand's help msg is not correct.
HDFS-10474:. hftp copy fails when file name with Chinese+special char in branch-2.
HDFS-10481: HTTPFS server should correctly impersonate as end user to open file.
HDFS-10485: Fix findbugs warning in FSEditLog.java.
HDFS-10488:. Update WebHDFS documentation regarding CREATE and MKDIR default permissions.
HDFS-10493: Add links to datanode web UI in namenode datanodes page.
HDFS-10499: TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails Intermittently.
HDFS-10508:. DFSInputStream should set thread's interrupt status after catching InterruptException from sleep.
HDFS-10512: VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks. .
HDFS-10525: Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap.
HDFS-10544: Balancer doesn't work with IPFailoverProxyProvider. .
HDFS-10556: DistCpOptions should be validated automatically.
HDFS-10569: A bug causes OutOfIndex error in BlockListAsLongs.
HDFS-10579: HDFS web interfaces lack configs for X-FRAME-OPTIONS protection.
HDFS-10589: Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params.
HDFS-10609: Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications. .
HDFS-10617: PendingReconstructionBlocks.size() should be synchronized.
HDFS-10623: Remove unused import of httpclient.HttpConnection from TestWebHdfsTokens.
HDFS-10625: VolumeScanner to report why a block is found bad. .
HDFS-10627: Volume Scanner marks a block as "suspect" even if the exception is network-related. .
HDFS-10642: TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently.
HDFS-10643: [Unit Test] Namenode should use loginUser(hdfs) to generateEncryptedKey.
HDFS-10643: HDFS namenode should always use service user (hdfs) to generateEncryptedKey.
HDFS-10653: Optimize conversion from path string to components. .
HDFS-10655: Fix path related byte array conversion bugs. .
HDFS-10660: Expose storage policy apis via HDFSAdmin interface.
HDFS-10684: WebHDFS DataNode calls fail without parameter createparent. .
HDFS-10688: BPServiceActor may run into a tight loop for sending block report when hitting IOException.
HDFS-10703: HA NameNode Web UI should show last checkpoint time. (John Zhuge via Yongjun Zhang) .
HDFS-10710: In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui. .
HDFS-10713: Throttle FsNameSystem lock warnings. .
HDFS-10715: NPE when applying AvailableSpaceBlockPlacementPolicy. .
HDFS-10716: In Balancer, the target task should be removed when its size < 0. .
HDFS-10735:Distcp using webhdfs on secure HA clusters fails with StandbyException .
HDFS-10752: Several log refactoring/improvement suggestion in HDFS. .
HDFS-10757: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used. .
HDFS-10763: Open files can leak permanently due to inconsistent lease update.
HDFS-10763: Open files can leak permanently due to inconsistent lease update .
HDFS-10764: Fix INodeFile#getBlocks to not return null. .
HDFS-10793: Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184.
HDFS-10798: Make the threshold of reporting FSNamesystem lock contention configurable. .
HDFS-10817: Add Logging for Long-held NN Read Locks. .
HDFS-10822: Log DataNodes in the write pipeline. John Zhuge via Lei Xu.
HDFS-10838: Last full block report received time for each DN should be easily discoverable.
HDFS-10892: Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'.
HDFS-10893: Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test.
HDFS-10906: Add unit tests for Trash with HDFS encryption zones.
HDFS-10913: Introduce fault injectors to simulate slow mirrors. .
HDFS-10917: Collect peer performance statistics on DataNode. .
HDFS-10922: Adding additional unit tests for Trash (II).
HDFS-10930: Refactor: Wrap Datanode IO related operations. .
HDFS-10934: TestDFSShell#testStat fails intermittently.
HDFS-10956: Remove rename/delete performance penalty when not using snapshots. .
HDFS-10959: Update DataNode to use DatasetVolumeChecker. .
HDFS-10962: TestRequestHedgingProxyProvider is flaky. .
HDFS-10963: Reduce log level when network topology cannot find enough datanodes.
HDFS-10965: Add unit test for HDFS command 'dfsadmin -printTopology'.
HDFS-10966: Enhance Dispatcher logic on deciding when to give up a source DataNode. .
HDFS-10972: Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'.
HDFS-10984: Expose nntop output as metrics.
HDFS-10986: DFSAdmin should log detailed error message if any.
HDFS-10987: Make Decommission less expensive when lot of blocks present. .
HDFS-10998: Add unit tests for HDFS command 'dfsadmin -fetchImage' in HA.
HDFS-11003: Expose XmitsInProgress through DataNodeMXBean.
HDFS-11008: Change unit test for testing parsing "-source" parameter in Balancer CLI.
HDFS-11011: Add unit tests for HDFS command 'dfsadmin -set/clrSpaceQuota'.
HDFS-11015: Enforce timeout in balancer. .
HDFS-11018: Incorrect check and message in FsDatasetImpl#invalidate. .
HDFS-11030: TestDataNodeVolumeFailure#testVolumeFailure is flaky (though passing).
HDFS-11031: Add additional unit test for DataNode startup behavior when volumes fail.
HDFS-11047: Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode.
HDFS-11054: Suppress verbose log message in BlockPlacementPolicyDefault.
HDFS-11056: Addendum fix for self-healing branch. .
HDFS-11056: Concurrent append and read operations lead to checksum error. .
HDFS-11065: Add space quota tests for heterogenous storages.
HDFS-11069: Tighten the authorization of datanode RPC.
HDFS-11076: Add unit test for extended Acls.
HDFS-11083: Add unit test for DFSAdmin -report command.
HDFS-11085: Add unit test for NameNode failing to start when name dir is unwritable.
HDFS-11087: NamenodeFsck should check if the output writer is still writable. .
HDFS-11112: Journal Nodes should refuse to format non-empty directories. .
HDFS-11114: Support for running async disk checks in DataNode. .
HDFS-11119: Support for parallel checking of StorageLocations on DataNode startup. .
HDFS-11122: TestDFSAdmin#testReportCommand fails due to timed out.
HDFS-11128: CreateEditsLog throws NullPointerException. .
HDFS-11131: TestThrottledAsyncChecker#testCancellation is flaky.
HDFS-11148: Update DataNode to use StorageLocationChecker at startup. .
HDFS-11149: Fix build break. .
HDFS-11149: Support for parallel checking of FsVolumes. .
HDFS-11156: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API.
HDFS-11160: VolumeScanner reports write-in-progress replicas as corrupt incorrectly. .
HDFS-11180: Intermittent deadlock in NameNode when failover happens. .
HDFS-11182: Addendum fix for Fenton to fix a unit test. .
HDFS-11182: Update DataNode to use DatasetVolumeChecker. .
HDFS-11194: Maintain aggregated peer performance metrics on NameNode. .
HDFS-11195: Return error when appending files by webhdfs rest api fails. .
HDFS-11197: Listing encryption zones fails when deleting a EZ that is on a snapshotted directory. .
HDFS-11209: SNN can't checkpoint when rolling upgrade is not finalized.
HDFS-11224: Lifeline message should be ignored for dead nodes (Contributed by Vinayakumar B) .
HDFS-11229: HDFS-11056 failed to close meta file. .
HDFS-11240: Remove snapshot version of SDK dependency from Azure Data Lake Store File System.
HDFS-11251: ConcurrentModificationException during DataNode#refreshVolumes .
HDFS-11253: FileInputStream leak on failure path in BlockSender.
HDFS-11258: File mtime change could not save to editlog. .
HDFS-11263: ClassCastException when we use Bzipcodec for Fsimage compression. .
HDFS-11267: Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage. (Manoj Govindassamy via lei) .
HDFS-11274: Datanode should only check the failed volume upon IO errors. .
HDFS-11279: Cleanup unused DataNode#checkDiskErrorAsync(). .
HDFS-11280: Allow WebHDFS to reuse HTTP connections to NN. .
HDFS-11282: Document the missing metrics of DataNode Volume IO operations. .
HDFS-11292: log lastWrittenTxId etc info in logSyncAll. .
HDFS-11299: Support multiple Datanode File IO hooks. .
HDFS-11307: The rpc to portmap service for NFS has hardcoded timeout.
HDFS-11337: (HDFS-10958 backport). Add instrumentation hooks around Datanode disk IO. .
HDFS-11339: Support File IO sampling for Datanode IO profiling hooks. .
HDFS-11369: Change exception message in StorageLocationChecker. .
HDFS-11377: Balancer hung due to no available mover threads.
HDFS-11379: DFSInputStream may infinite loop requesting block locations. .
HDFS-11387: Socket reuse address option is not honored in PrivilegedNfsGatewayStarter. fixing BUG-66651 by backporting HDFS-4176 fixing BUG-66652 by backporting HDFS-5356 and HDFS-7553 .
HDFS-11391: Numeric usernames do no work with WebHDFS FS write access.
HDFS-11395: RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode. .
HDFS-11403: Zookeper ACLs on NN HA enabled clusters should be handled consistently. .
HDFS-11445: FSCK shows overall health stauts as corrupt even one replica is corrupt.
HDFS-11461: DataNode Disk Outlier Detection.
HDFS-11462: Fix occasional BindException in TestNameNodeMetricsLogger. .
HDFS-11476: Fix NPE in FsDatasetImpl#checkAndUpdate.
HDFS-11479: Socket re-use address option should be used in SimpleUdpServer. .
HDFS-11499: Decommissioning stuck because of failing recovery.
HDFS-11502: Datanode UI should display hostname based on JMX bean instead of window.location.hostname.
HDFS-11508: Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket is in TIME_WAIT state. .
HDFS-11511: Support Timeout when checking single disk.
HDFS-11517: Expose slow disks via DataNode JMX.
HDFS-11533: reuseAddress option should be used for child channels in Portmap and SimpleTcpServer.
HDFS-11534: Add counters for number of blocks in pending IBR.
HDFS-11545: Propagate DataNode's slow disks info to the NameNode via Heartbeat.
HDFS-11547: Add logs for slow BlockReceiver while writing data to disk.
HDFS-11551: Handle SlowDiskReport from DataNode at the NameNode. .
HDFS-11560: Expose slow disks via NameNode JMX.
HDFS-11570: Unit test for NameNodeStatusMXBean. .
HDFS-11592: Closing a file has a wasteful preconditions in NameNode.
HDFS-11603: Improve slow mirror/disk warnings in BlockReceiver.
HDFS-11608: HDFS write crashed with block size greater than 2 GB.
HDFS-11609: Some blocks can be permanently lost if nodes are decommissioned while dead.
HDFS-11630: TestThrottledAsyncCheckerTimeout fails intermittently in Jenkins builds.
HDFS-11634: Optimize BlockIterator when interating starts in the middle.
HDFS-11641: Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus.
HDFS-11674: reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered.
HDFS-11722: Change Datanode file IO profiling sampling to percentage. .
HDFS-11842: TestDataNodeOutlierDetectionViaMetrics UT fails intermittently. Controbuted by Hanisha Koneru. .
HDFS-11914: Add more diagnosis info for fsimage transfer failure.
HDFS-12043: Add counters for block re-replication.
HDFS-12130: Optimizing permission check for getContentSummary.
HDFS-12139: HTTPFS liststatus returns incorrect pathSuffix for path of file'.
HDFS-12158: Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection.'.
HDFS-12293: DataNode should log file name on disk error.
HDFS-12325: SFTPFileSystem operations should restore cwd.
HDFS-12407: Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or JournalNodeRpcServer fails to start.
HDFS-12683: DFSZKFailOverController re-order logic for logging Exception.
HDFS-14676: Wrong default value for fs.df.interval.'.
MAPREDUCE-5485: Allow repeating job commit by extending OutputCommitter API.
MAPREDUCE-5621: mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time.
MAPREDUCE-6165: [JDK8] TestCombineFileInputFormat failed on JDK8.
MAPREDUCE-6197: Cache MapOutputLocations in ShuffleHandler.
MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.
MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.
MAPREDUCE-6246: DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2.
MAPREDUCE-6273: HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state.
MAPREDUCE-6279: AM should explicity exit JVM after all services have stopped.
MAPREDUCE-6302: (BUG-47640) Backport preempt reducers after a configurable timeout irrespective of headroom.
MAPREDUCE-6304: Specifying node labels when submitting MR jobs.
MAPREDUCE-6338: MR AppMaster does not honor ephemeral port range.
MAPREDUCE-6350: JobHistory doesn't support fully-functional search.
MAPREDUCE-6404: Allow AM to specify a port range for starting its webapp.
MAPREDUCE-6436: JobHistory cache issue.
MAPREDUCE-6436: JobHistory cache issue.
MAPREDUCE-6460: TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails.
MAPREDUCE-6478: Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob.
MAPREDUCE-6485: Create a new task attempt with failed map task priority if in-progress attempts are unassigned.
MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.
MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.
MAPREDUCE-6514: Job hangs as ask is not updated after ramping down of all reducers.
MAPREDUCE-6543: Migrate MR Client test cases part 2.
MAPREDUCE-6549: multibyte delimiters with LineRecordReader cause duplicate records.
MAPREDUCE-6565: Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.
MAPREDUCE-6566: Add retry support to mapreduce CLI tool.
MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
MAPREDUCE-6579: TestNetworkedJob.testNetworkedJob failed due to.
MAPREDUCE-6593: TestJobHistoryEventHandler.testTimelineEventHandling fails because of NPE.
MAPREDUCE-6616: Fail to create jobhistory file if there are some multibyte characters in the job name.
MAPREDUCE-6618: YarnClientProtocolProvider leaking the YarnClient thread.(xgong).
MAPREDUCE-6621: Memory Leak in JobClient#submitJobInternal().
MAPREDUCE-6625: TestCLI#testGetJob fails occasionally (haibochen via rkanter).
MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.
MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.
MAPREDUCE-6639: Process hangs in LocatedFileStatusFetcher if FileSystem.get throws.
MAPREDUCE-6672: TestTeraSort fails on Windows.
MAPREDUCE-6684: High contention on scanning of user directory under immediate_done in Job History Server.
MAPREDUCE-6689: MapReduce job can infinitely increase number of reducer resource requests.
MAPREDUCE-6693: ArrayIndexOutOfBoundsException occurs when the length of the job name is equal to mapreduce.jobhistory.jobname.limit.
MAPREDUCE-6701: application master log can not be available when clicking jobhistory's am logs link.
MAPREDUCE-6702: Fixed failures in TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv.
MAPREDUCE-6711: After enabling preemption on cluster, jobs failed without making any attempts made after being preempted.
MAPREDUCE-6724: Single shuffle to memory must not exceed Integer#MAX_VALUE.
MAPREDUCE-6733: MapReduce JerseyTest tests failing with java.net.BindException: Address already in use.
MAPREDUCE-6762: ControlledJob#toString failed with NPE when job status is not successfully updated.
MAPREDUCE-6789: Fix TestAMWebApp failure. .
MAPREDUCE-6792: Allow user's full principal name as owner of MapReduce staging directory in JobSubmissionFiles#JobStagingDir().
MAPREDUCE-6797: Job history server scans can become blocked on a single, slow entry.
MAPREDUCE-6852: Job#updateStatus() failed with NPE due to race condition.
MAPREDUCE-6897: Add Unit Test to ensure Job end notification gets sent even when appMaster stop gets YarnRuntimeException.
YARN-857: Localization failures should be available in container diagnostics.
YARN-998: Keep NM resource updated through dynamic resource config for RM/NM restart.
YARN-1376: NM need to notify the log aggregation status to RM through heartbeat.
YARN-1402: Update related Web UI and CLI with exposing client API to check log aggregation status.
YARN-1462: AHS API and other AHS changes to handle tags for completed MR jobs. .
YARN-1556: NPE getting application report with a null appId.
YARN-1880: Cleanup TestApplicationClientProtocolOnHA.
YARN-1993: Cross-site scripting vulnerability in TextView.java. Contributed byKenji Kikushima.
YARN-2019: Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore.
YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.
YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.
YARN-2194: Fix bug causing CGroups functionality to fail on RHEL7.
YARN-2392: Add more diags about app retry limits on AM failures.
YARN-2495: Allow admin specify labels from each NM (Distributed configuration for node label).
YARN-2498: Respect labels in preemption policy of capacity scheduler for inter-queue preemption.
YARN-2513:Host framework UIs in YARN for use with the ATS Note: Patch is in progress.
YARN-2513:Host framework UIs in YARN for use with the ATS Note: Patch is in progress.
YARN-2571: RM to support YARN registry.
YARN-2619: Added NodeManager support for disk io isolation through cgroups.
YARN-2666: TestFairScheduler.testContinuousScheduling fails Intermittently.
YARN-2696: Queue sorting in CapacityScheduler should consider node label.
YARN-2740: Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled.
YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.
YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.
YARN-2868: FairScheduler: Metric for latency to allocate first container for an application.
YARN-2901: Add errors and warning metrics page to RM, NM web UI.
YARN-3021: YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp.
YARN-3026: Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp.
YARN-3102: Decommisioned Nodes not listed in Web UI.
YARN-3102: Decommisioned Nodes not listed in Web UI.
YARN-3110: Few issues in ApplicationHistory web ui.
YARN-3136: Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers.
YARN-3148: Allow CORS related headers to passthrough in WebAppProxyServlet.
YARN-3215: Respect labels in CapacityScheduler when computing headroom.
YARN-3216: Max-AM-Resource-Percentage should respect node labels.
YARN-3225: New parameter of CLI for decommissioning node gracefully in RMAdmin CLI.
YARN-3243: CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits.
YARN-3248: Display count of nodes blacklisted by apps in the web UI.
YARN-3254: HealthReport should include disk full information.
YARN-3266: RMContext#inactiveNodes should have NodeId as map key.
YARN-3269: Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path.
YARN-3293: Track and display capacity scheduler health metrics in web UI.
YARN-3294: Allow dumping of Capacity Scheduler debug logs via web UI for a fixed time period.
YARN-3305: Normalize AM resource request on app submission.
YARN-3318: Create Initial OrderingPolicy Framework and FifoOrderingPolicy.
YARN-3319: Implement a FairOrderingPolicy.
YARN-3326: Support RESTful API for getLabelsToNodes.
YARN-3343: Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate timeout.
YARN-3345: Add non-exclusive node label API.
YARN-3347: Improve YARN log command to get AMContainer logs as well as running containers logs.
YARN-3348: Add a 'yarn top' tool to help understand cluster usage.
YARN-3354: Add node label expression in ContainerTokenIdentifier to support RM recovery.
YARN-3356: Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label.
YARN-3357: Move TestFifoScheduler to FIFO package.
YARN-3360: Add JMX metrics to TimelineDataManager.
YARN-3360: Commits to fix the dependency.
YARN-3361: CapacityScheduler side changes to support non-exclusive node labels.
YARN-3362: Add node label usage in RM CapacityScheduler web UI.
YARN-3365: Enhanced NodeManager to support using the 'tc' tool via container-executor for outbound network traffic control.
YARN-3366: Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers.
YARN-3383: AdminService should use "warn" instead of "info" to log exception when operation fails.
YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.
YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.
YARN-3394: Enrich WebApplication proxy documentation.
YARN-3397: yarn rmadmin should skip -failover.
YARN-3404: Display queue name on application page.
YARN-3406: Display count of running containers in the RM's Web UI.
YARN-3410: YARN admin should be able to remove individual application records from RMStateStore.
YARN-3413: Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime.
YARN-3425: NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed.
YARN-3433: Jersey tests failing with Port in Use -again.
YARN-3435: AM container to be allocated Appattempt AM container shown as null.
YARN-3443: Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM.
YARN-3448: Added a rolling time-to-live LevelDB timeline store implementation.
YARN-3451: Display attempt start time and elapsed time on the web UI.
YARN-3453: Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing.
YARN-3459: Fix failiure of TestLog4jWarningErrorMetricsAppender.
YARN-3463: Integrate OrderingPolicy Framework with CapacityScheduler.
YARN-3480: Remove attempts that are beyond max-attempt limit from state store.
YARN-3494: Expose AM resource limit and usage in CS QueueMetrics.
YARN-3503: Expose disk utilization percentage and bad local and log dir counts in NM metrics.
YARN-3505: Node's Log Aggregation Report with SUCCEED should not cached in RMApps.
YARN-3511: Add errors and warnings page to ATS.
YARN-3517: RM web ui for dumping scheduler logs should be for admins only.
YARN-3521: Support return structured NodeLabel objects in REST API.
YARN-3530: ATS throws exception on trying to filter results without otherinfo.
YARN-3541: Add version info on timeline service / generic history web UI and REST API.
YARN-3543: ApplicationReport should be able to tell whether the Application is AM managed or not.
YARN-3552: RM Web UI shows -1 running containers for completed apps.
YARN-3565: NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String.
YARN-3579: CommonNodeLabelsManager should support NodeLabel instead of string label name when getting node-to-label/label-to-label mappings.
YARN-3580: [JDK8] TestClientRMService.testGetLabelsToNodes fails.
YARN-3581: Deprecate -directlyAccessNodeLabelStore in RMAdminCLI.
YARN-3583: Support of NodeLabel object instead of plain String in YarnClient side.
YARN-3589: RM and AH web UI display DOCTYPE wrongly.
YARN-3591: Resource localization on a bad disk causes subsequent containers failure.
YARN-3593: Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page.
YARN-3600: AM container link is broken - on a killed application, at least.
YARN-3604: removeApplication in ZKRMStateStore should also disable watch.
YARN-3623: Add a config to indicate the Timeline Service version.
YARN-3623: Addendum: Improve the description for Timeline Service Version configuration in yarn-default.xml.
YARN-3632: Ordering policy should be allowed to reorder an application when demand changes.
YARN-3635: Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework.
YARN-3654: ContainerLogsPage web UI should not have meta-refresh.
YARN-3684: Changed ContainerExecutor's primary lifecycle methods to use a more extensible mechanism of context objects.
YARN-3695: ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.
YARN-3695: ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.
YARN-3700: Made generic history service load a number of latest applications according to the parameter or the configuration.
YARN-3707: RM Web UI queue filter doesn't work.
YARN-3717: Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API. .
YARN-3722: Merge multiple TestWebAppUtils into o.a.h.yarn.webapp.util.TestWebAppUtils.
YARN-3727: For better error recovery, check if the directory exists before using it for localization.
YARN-3740: Fixed the typo in the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS.
YARN-3749: We should make a copy of configuration when init MiniYARNCluster with multiple RMs.
YARN-3751: Fixed AppInfo to check if used resources are null.
YARN-3762: FairScheduler: CME on FSParentQueue#getQueueUserAclInfo.
YARN-3766: Fixed the apps table column error of generic history web UI.
YARN-3768: ArrayIndexOutOfBoundsException with empty environment variables.
YARN-3769: Consider user limit when calculating total pending resource for preemption policy in Capacity Scheduler.
YARN-3785: Support for Resource as an argument during submitApp call in MockRM test class.
YARN-3844: Make hadoop-yarn-project Native code -Wall-clean.
YARN-3846: RM Web UI queue filter is not working for sub queue.
YARN-3849:(Too much of preemption activity causing continuos killing of containers across queues) to 2.3-maint.
YARN-3852: Add docker container support to container-executor.
YARN-3853: Add docker container runtime support to LinuxContainterExecutor.
YARN-3873: PendingApplications in LeafQueue should also use OrderingPolicy.
YARN-3885: ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level.
YARN-3894: RM startup should fail for wrong CS xml NodeLabel capacity configuration.
YARN-3896: RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously.
YARN-3920: FairScheduler container reservation on a node should be configurable to limit it to large containers.
YARN-3932: SchedulerApplicationAttempt#getResourceUsageReport and UserInfo should based on total-used-resources.
YARN-3940: Application moveToQueue should check NodeLabel permission.
YARN-3941: Proportional Preemption policy should try to avoid sending duplicate PREEMPT_CONTAINER event to scheduler.
YARN-3946: .
YARN-3946: Update exact reason as to why a submitted app is in ACCEPTED state to app's diagnostic message.
YARN-3948: Display Application Priority in RM Web UI.(Sunil G via rohithsharmaks).
YARN-3961: Expose pending, running and reserved containers of a queue in REST api and yarn top.
YARN-3963: AddNodeLabel on duplicate label addition shows success.
YARN-3966: Fix excessive loggings in CapacityScheduler.
YARN-3971: Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery.
YARN-3978: Configurably turn off the saving of container info in Generic AHS.
YARN-3982: container-executor parsing of container-executor.cfg broken in trunk and branch-2.
YARN-3983: Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic.
YARN-3986: getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface.
YARN-3987: am container complete msg ack to NM once RM receive it.
YARN-4000: RM crashes with NPE if leaf queue becomes parent queue during restart.
YARN-4004: container-executor should print output of docker logs if the docker container exits with non-0 exit status.
YARN-4007: Add support for different network setups when launching the docker container.
YARN-4017: container-executor overuses PATH_MAX.
YARN-4023: Publish Application Priority to TimelineServer.
YARN-4026: Refactored ContainerAllocator to accept a list of priorites rather than a single priority.
YARN-4082: Container shouldn't be killed when node's label updated.
YARN-4087: Followup fixes after YARN-2019 regarding RM behavior when state-store error occurs.
YARN-4092: Fixed UI redirection to print useful messages when both RMs are in standby mode.
YARN-4101: RM should print alert messages if ZooKeeper and Resourcemanager gets connection issue.
YARN-4108: CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request.
YARN-4109: Exception on RM scheduler page loading with labels.
YARN-4113: RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER.
YARN-4132: Separate configs for nodemanager to resourcemanager connection timeout and retries.
YARN-4140: RM container allocation delayed incase of app submitted to Nodelabel partition.
YARN-4149: yarn logs -am should provide an option to fetch all the log files.
YARN-4155: TestLogAggregationService.testLogAggregationServiceWithInterval failing.
YARN-4162: CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API.
YARN-4168: Fixed a failing test TestLogAggregationService.testLocalFileDeletionOnDiskFull.
YARN-4204: ConcurrentModificationException in FairSchedulerQueueInfo.
YARN-4215: RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request.
YARN-4225: Add preemption status to yarn queue -status for capacity scheduler.
YARN-4233: YARN Timeline Service plugin: ATS v1.5.
YARN-4243: Add retry on establishing ZooKeeper conenction in EmbeddedElectorService#serviceInit.
YARN-4245: Generalize config file handling in container-executor.
YARN-4252: Log container-executor invocation details when exit code is non-zero.
YARN-4253: Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor.
YARN-4255: container-executor does not clean up docker operation command files.
YARN-4258: Add support for controlling capabilities for docker containers.
YARN-4262: Allow whitelisted users to run privileged docker containers.
YARN-4267: Add additional logging to container launch implementations in container-executor.
YARN-4270: Limit application resource reservation on nodes for non-node/rack specific requests.
YARN-4285: Display resource usage as percentage of queue and cluster in the RM UI.
YARN-4287: Capacity Scheduler: Rack Locality improvement.
YARN-4288: Fixed RMProxy to retry on IOException from local host.
YARN-4290: Add -showDetails option to YARN Nodes CLI to print all nodes reports information.
YARN-4304: AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics.
YARN-4309: Add container launch related debug information to container logs when a container fails.
YARN-4313: Race condition in MiniMRYarnCluster when getting history server address.
YARN-4315: NaN in Queue percentage for cluster apps page.
YARN-4325: Nodemanager log handlers fail to send finished/failed events in some cases.
YARN-4329: [YARN-5437] Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler.
YARN-4345: yarn rmadmin -updateNodeResource doesn't work.
YARN-4347: Resource manager fails with Null pointer exception.
YARN-4349:YARN_APPLICATION call to ATS does not have YARN_APPLICATION_CALLER_CONTEXT.
YARN-4365: FileSystemNodeLabelStore should check for root dir existence on startup.
YARN-4384: updateNodeResource CLI should not accept negative values for resource.
YARN-4390: Do surgical preemption based on reserved container in CapacityScheduler.
YARN-4392: ApplicationCreatedEvent event time resets after RM restart/failover.
YARN-4393: TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently.
YARN-4403: (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period.
YARN-4405: Support node label store in non-appendable file system.
YARN-4411: RMAppAttemptImpl#createApplicationAttemptReport throws IllegalArgumentException.
YARN-4414: Nodemanager connection errors are retried at multiple levels.
YARN-4414: Nodemanager connection errors are retried at multiple levels.
YARN-4416: Deadlock due to synchronised get Methods in AbstractCSQueue.
YARN-4417: Make RM and Timeline-server REST APIs more consistent.
YARN-4418: AM Resource Limit per partition can be updated to ResourceUsage as well.
YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.
YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.
YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.
YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.
YARN-4439: Clarify NMContainerStatus#toString method.
YARN-4454: NM to nodelabel mapping going wrong after RM restart.
YARN-4465: SchedulerUtils#validateRequest for Label check should happen only when nodelabel enabled.
YARN-4477: FairScheduler: Handle condition which can result in an infinite loop in attemptScheduling.
YARN-4479: Change CS LeafQueue pendingOrderingPolicy to hornor recovered apps.
YARN-4496: Improve HA ResourceManager Failover detection on the client.
YARN-4497: RM might fail to restart when recovering apps whose attempts are missing.
YARN-4502: Fix two AM containers get allocated when AM restart. .
YARN-4522: Queue acl can be checked at app submission.
YARN-4535: Fix checkstyle error in CapacityScheduler.java.
YARN-4546: YARN-4546. ResourceManager crash due to scheduling opportunity overflow.
YARN-4546: YARN-4546. ResourceManager crash due to scheduling opportunity overflow.
YARN-4553: Add cgroups support for docker containers.
YARN-4556: TestFifoScheduler.testResourceOverCommit fails.
YARN-4557: Fix improper Queues sorting in PartitionedQueueComparator when accessible-node-labels=*.
YARN-4565: (BUG-50408) Sometimes when sizeBasedWeight FairOrderingPolicy is enabled, under stress appears that cluster is virtually in deadlock.
YARN-4571: Make app id/name available to the yarn authorizer provider for better auditing.
YARN-4573: Fix test failure in TestRMAppTransitions#testAppRunningKill and testAppKilledKilled.
YARN-4577: Enable aux services to have their own custom classpath/jar file.
YARN-4578: Directories that are mounted in docker containers need to be more restrictive/container-specific.
YARN-4582: Label-related invalid resource request exception should be able to properly handled by application.
YARN-4584: RM startup failure when AM attempts greater than max-attempts.
YARN-4591: YARN Web UIs should provide a robots.txt. Junping Du.
YARN-4595: Add support for configurable read-only mounts when launching Docker containers.
YARN-4596: SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities.
YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.
YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.
YARN-4607: Pagination support for AppAttempt page TotalOutstandingResource Requests table.
YARN-4610: Reservations continue looking for one app causes other apps to starve.
YARN-4617: LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy.
YARN-4623: TestSystemMetricsPublisher#testPublishAppAttemptMetricsForUnmanagedAM fails with NPE on branch-2.7.
YARN-4624: NPE in PartitionQueueCapacitiesInfo while accessing Scheduler UI.
YARN-4625: Make ApplicationSubmissionContext and ApplicationSubmissionContextInfo more consistent.
YARN-4633: TestRMRestart.testRMRestartAfterPreemption fails intermittently.
YARN-4634: Scheduler UI/Metrics need to consider cases like non-queue label mappings.
YARN-4643: Container recovery is broken with delegating container runtime.
YARN-4667: RM Admin CLI for refreshNodesResources throws NPE when nothing is configured.
YARN-4680: Fix TimerTasks leak in ATS V1.5 Writer.
YARN-4696: TimelineClient to add flush operation for deterministic writes (including testing) and Changes to EntityGroupFSTimelineStore for testability.
YARN-4697: NM aggregation thread pool is not bound by limits.
YARN-4699: Scheduler UI and REST o/p is not in sync when -replaceLabelsOnNode is used to change label of a node.
YARN-4709: NMWebServices produces incorrect JSON for containers.
YARN-4709: NMWebServices produces incorrect JSON for containers.
YARN-4723: NodesListManager$UnknownNodeId ClassCastException.
YARN-4737: Add CSRF filter support in YARN.
YARN-4737: Add CSRF filter support in YARN.
YARN-4744: Too many signal to container failure in case of LCE.
YARN-4746: yarn web services should convert parse failures of appId, appAttemptId and containerId to 400.
YARN-4748: ApplicationHistoryManagerOnTimelineStore should not swallow exceptions on generateApplicationReport.
YARN-4759: Fix signal handling for docker containers.
YARN-4762: Fixed CgroupHandler's creation and usage to avoid NodeManagers crashing when LinuxContainerExecutor is enabled.
YARN-4764: Application submission fails when submitted queue is not available in scheduler xml.
YARN-4767: Network issues can cause persistent RM UI outage.
YARN-4769: Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI.
YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.
YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.
YARN-4811: Generate histograms in ContainerMetrics for actual container resource usage.
YARN-4814: ATS 1.5 timelineclient impl call flush after every event write.
YARN-4815: ATS 1.5 timelineclinet impl try to create attempt directory for every event call.
YARN-4817: TimelineClient ATSv1.5 logging is very noisy.
YARN-4820: ResourceManager web redirects in HA mode drops query parameters.
YARN-4822: Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates.
YARN-4832: NM side resource value should get updated if change applied in RM side.
YARN-4833: For Queue AccessControlException client retries multiple times on both RM.
YARN-4842: Fixed "yarn logs" command to guess (and thus not require) the appOwner argument when viewing another user's logs.
YARN-4844: Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource.
YARN-4846: Fix random failures for TestCapacitySchedulerPreemption #testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers.
YARN-4851: Metric improvements for ATS v1.5 storage components.
YARN-4863: AHS Security login should be in serviceInit() instead of serviceStart().
YARN-4865: Track Reserved resources in ResourceUsage and QueueCapacities.
YARN-4890: Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate.
YARN-4893: Fix intermittent failures in TestRMAdminService.
YARN-4905: Improved "yarn logs" command-line to optionally show log metadata also.
YARN-4906: Capture container start/finish time in container metrics.
YARN-4909: Fix intermittent failures of TestRMWebServices And TestRMWithCSRFFilter.
YARN-4913: Yarn logs should take a -out option to write to a directory.
YARN-4916: TestNMProxy.tesNMProxyRPCRetry fails.
YARN-4920: ATS/NM should support a link to dowload/get the logs in text format.
YARN-4928: Some yarn.server.timeline.* tests fail on Windows attempting to use a test root path containing a colon.
YARN-4934: Reserved Resource for QueueMetrics needs to be handled correctly in few cases.
YARN-4955: Add retry for SocketTimeoutException in TimelineClient.
YARN-4956: findbug issue on LevelDBCacheTimelineStore.
YARN-4966: Improve yarn logs to fetch container logs without specifying nodeId.
YARN-4983: JVM and UGI metrics disappear after RM transitioned to standby mode.
YARN-4984: LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak.
YARN-4987: Read cache concurrency issue between read and evict in EntityGroupFS timeline store.
YARN-4990: Re-direction of a particular log file within in a container in NM UI does not redirect properly.
YARN-5002: getApplicationReport call may raise NPE for removed queues.
YARN-5023: TestAMRestart#testShouldNotCountFailureToMaxAttemptRetry random failure.
YARN-5029: RM needs to send update event with YarnApplicationState as Running to ATS/AHS.
YARN-5048: DelegationTokenRenewer#skipTokenRenewal may throw NPE.
YARN-5069: TestFifoScheduler.testResourceOverCommit race condition.
YARN-5076: YARN web interfaces lack XFS (Cross-Frame Script) protection.
YARN-5080: Cannot obtain logs using YARN CLI -am for either KILLED or RUNNING AM.
YARN-5083: YARN CLI for AM logs does not give any error message if entered invalid am value.
YARN-5088: Improve "yarn log" command-line to read the last K bytes for the log files.
YARN-5089: Improve "yarn log" command-line "logFiles" option to support regex.
YARN-5098: Fixed ResourceManager's DelegationTokenRenewer to replace expiring system-tokens if RM stops and only restarts after a long time.
YARN-5100: The YarnApplicationState is always running in ATS even application is finished.
YARN-5103: With NM recovery enabled, restarting NM multiple times results in AM restart.
YARN-5112: Excessive log warnings for directory permission issue on NM recovery.
YARN-5115: Avoid setting CONTENT-DISPOSITION header in the container-logs web-service.
YARN-5121: fix some container-executor portability issues.
YARN-5122: "yarn logs" for running containers should print an explicit footer saying that the log may be incomplete.
YARN-5131: Distributed shell AM fails when extra container arrives during finishing. Wangda Tan via junping_du.
YARN-5141: Get Container logs for the Running application from Yarn Logs CommandLine.
YARN-5190: Registering/unregistering container metrics triggered by ContainerEvent and ContainersMonitorEvent are conflict which cause uncaught exception in ContainerMonitorImpl.
YARN-5191: Renamed the newly added “download=true” option for getting logs via NMWebServices and AHSWebServices to be a better "format" option.
YARN-5199: Close LogReader in in AHSWebServices#getStreamingOutput and FileInputStream in NMWebServices#getLogs.
YARN-5200: Enhanced "yarn logs" to be able to get a list of containers whose logs are aggregated via a "show_container_log_info" option.
YARN-5206: RegistrySecurity includes id:pass in exception text if considered invalid.
YARN-5208: Run TestAMRMClient TestNMClient TestYarnClient TestClientRMTokens TestAMAuthorization tests with hadoop.security.token.service.use_ip enabled.
YARN-5213: Fix a bug in LogCLIHelpers which cause TestLogsCLI#testFetchApplictionLogs fails intermittently.
YARN-5214: Fixed locking in DirectoryCollection to avoid hanging NMs when various code-paths hit slow disks.
YARN-5223: Container line in yarn logs output for a live application should include the hostname for the container.
YARN-5224: Added new web-services /containers/{containerid}/logs & /containers/{containerid}/logs/{filename} and using them in "yarn logs" CLI to get logs of finished containers of a running application.
YARN-5226: Remove AHS enable check from LogsCLI#fetchAMContainerLogs.
YARN-5227: Yarn logs command: no need to specify applicationId when specifying containerId.
YARN-5233: Support for specifying a path for ATS plugin jars.
YARN-5237: Fix missing log files issue in rolling log aggregation.
YARN-5246: NMWebAppFilter web redirects drop query parameters.
YARN-5251: Yarn CLI to obtain App logs for last 'n' bytes fails.
YARN-5266: Wrong exit code while trying to get app logs using regex via CLI.
YARN-5270: Solve miscellaneous issues caused by YARN-4844.
YARN-5294: Pass remote ip address down to YarnAuthorizationProvider.
YARN-5296: NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl.
YARN-5297: Avoid printing a stack trace when recovering an app after the RM restarts.
YARN-5298: Mount usercache and NM filecache directories into Docker container.
YARN-5299: Log Docker run command when container fails.
YARN-5314: Fixed a ConcurrentModificationException in ATS v1.5 EntityGroupFSTimelineStore.
YARN-5337: Fix Dshell OOM issues.
YARN-5339: Passing file to -out for YARN log CLI doesnt give warning or error code.
YARN-5340: Fixed a race condition in RollingLevelDBTimelineStore that caused loss of Timeline events.
YARN-5342: Improve non-exclusive node partition resource allocation in Capacity Scheduler.
YARN-5353: ResourceManager can leak delegation tokens when they are shared across apps. (Jason Lowe via Varun Saxena).
YARN-5363: For AM containers, or for containers of running-apps, "yarn logs" incorrectly only (tries to) shows syslog file-type by default.
YARN-5368: Fixed a memory leak in Timeline server.
YARN-5432: Lock already held by another process while LevelDB cache store creation for dag.
YARN-5440: Use AHSClient in YarnClient when TimelineServer is running.
YARN-5456: container-executor support for FreeBSD, NetBSD, and others if conf path is absolute. .
YARN-5470: Differentiate exactly match with regex in yarn log CLI.
YARN-5476: Not existed application reported as ACCEPTED state by YarnClientImpl.
YARN-5523: Yarn running container log fetching causes OutOfMemoryError.
YARN-5524: Yarn live log aggregation does not throw if command line arg is wrong.
YARN-5539: TimelineClient failed to retry on java.net.SocketTimeoutException: Read timed out.
YARN-5549: AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately.
YARN-5551:: Ignore file backed pages from memory computation when smaps is enabled.
YARN-5659: getPathFromYarnURL should use standard methods.
YARN-5677: RM should transition to standby when connection is lost for an extended period.
YARN-5704: Provide config knobs to control enabling/disabling new/work in progress features in container-executor.
YARN-5711: Propogate exceptions back to client when using hedging RM failover provider.
YARN-5718: TimelineClient (and other places in YARN) shouldn't over-write HDFS client retry settings which could cause unexpected behavior.
YARN-5731: Preemption calculation is not accurate when reserved containers are present in queue.
YARN-5773: RM recovery too slow due to LeafQueue#activateApplications.
YARN-5864: Capacity Scheduler queue priority.
YARN-5873: RM crashes with NPE if generic application history is enabled.
YARN-5910: Support for multi-cluster delegation tokens.
YARN-5915: ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write.
YARN-5923: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled.
YARN-5923: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled.
YARN-5999: AMRMClientAsync will stop if any exceptions thrown on allocate call.
YARN-6011: Add a new web service to list the files on a container in AHSWebService.
YARN-6029: CapacityScheduler deadlock when ParentQueue#getQueueUserAclInfo is called by one thread and LeafQueue#assignContainers is releasing excessive reserved container by another thread.
YARN-6031: Application recovery has failed when node label feature is turned off during RM recovery.
YARN-6068: Log aggregation get failed when NM restart even with recovery. .
YARN-6081:LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container.
YARN-6082: Invalid REST api response for getApps since queueUsagePercentage is coming as INF.
YARN-6099: Improve webservice to list aggregated log files.
YARN-6100: Improve YARN webservice to output aggregated container logs.
YARN-6104: RegistrySecurity overrides zookeeper sasl system properties.
YARN-6108: Improve AHS webservice to accept NM address as a parameter to get container logs.
YARN-6113: Re-direct NM Web Service to get container logs for finished applications.
YARN-6123: Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed.
YARN-6126: Obtaining app logs for Running application fails with json parse error.
YARN-6137: Yarn client implicitly invoke ATS client which accesses HDFS. .
YARN-6145: Improve log message on fail over.
YARN-6174: Log files pattern should be same for both running and finished container.
YARN-6189: ApplicationNotFoundException due to RM failed over should throw more user friendly exception.
YARN-6228: EntityGroupFSTimelineStore should allow configurable cache stores.
YARN-6270: WebUtils.getRMWebAppURLWithScheme() needs to honor RM HA setting.
YARN-6271: yarn rmadin -getGroups returns information from standby RM.
YARN-6294: ATS client should better handle Socket closed case.
YARN-6313: yarn logs cli should provide logs for a completed container even when application is still running.
YARN-6314: Potential infinite redirection on YARN log redirection web service.
YARN-6332: Make RegistrySecurity use short user names for ZK ACLs.
YARN-6339: Improve performance for createAndGetApplicationReport.
YARN-6367: YARN logs CLI needs alway check containerLogsInfo/containerLogInfo before parse the JSON object from NMWebService.
YARN-6509: Maybe add a size threshold beyond which yarn logs will require a force option.
YARN-6550: Capture launch_container.sh logs to a separate log file.
YARN-6625: yarn application -list returns a tracking URL for AM that doesn't work in secured and HA environment.
YARN-6811: [ATS1.5] All history logs should be kept under its own User Directory.
YARN-6876: Create an abstract log writer for extendability.
YARN-6877: Create an abstract log reader for extendability.
YARN-6890: Not display killApp button on UI if UI is unsecured but cluster is secured.
YARN-6991: "Kill application" button does not show error if other user tries to kill the application for secure cluster.
YARN-6992: Kill application button is visible even if the application is FINISHED in RM UI.
YARN-7072: Add a new log aggregation file format controller.
YARN-7090: testRMRestartAfterNodeLabelDisabled[1] UT Fails.
YARN-7116: CapacityScheduler Web UI: Queue's AM usage is always show on per-user's AM usage.
YARN-7118: AHS REST API can return NullPointerException.
YARN-7144: Log Aggregation controller should not swallow the exceptions when it calls closeWriter and closeReader.
YARN-7148: TestLogsCLI fails in trunk and branch-2 and javadoc error.
YARN-7157: Add admin configuration to filter per-user's apps in secure cluster.
YARN-7163: RMContext need not to be injected to webapp and other Always Running services. .
YARN-7174: Add retry logic in LogsCLI when fetch running application logs.
YARN-7194: Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat.
YARN-7249: Fix CapacityScheduler NPE issue when a container preempted while the node is being removed.
YARN-7257:AggregatedLogsBlock reports a bad 'end' value as a bad 'start' value.
YARN-7259: Add size-based rolling policy to LogAggregationIndexedFileController.
YARN-7269: Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications.

HDP 2.6.2 provided Hadoop Common 2.7.3 and the following Apache patches:

HADOOP-1540: breaks backward compatibilty.
HADOOP-1540: Support file exclusion list in distcp.
HADOOP-3733: "s3x:" URLs break when Secret Key contains a slash, even if encoded.
HADOOP-5323: Trash documentation should describe its directory structure and configurations.
HADOOP-5470: RunJar.unJar() should write the last modified time found in the jar entry to the uncompressed file.
HADOOP-5732: Add SFTPFileSystem.
HADOOP-7139: Allow appending to existing SequenceFiles.
HADOOP-7256: Resource leak during failure scenario of closing of resources.
HADOOP-7713: dfs -count -q should label output column.
HADOOP-7713: dfs -count -q should label output column.
HADOOP-7817: RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos() .
HADOOP-7930: Kerberos relogin interval in UserGroupInformation should be configurable.
HADOOP-8151: Error handling in snappy decompressor throws invalid exceptions.
HADOOP-8436: NPE In getLocalPathForWrite ( path, conf ) when the required context item is not configured. .
HADOOP-8437: getLocalPathForWrite should throw IOException for invalid paths.
HADOOP-8751: NPE in Token.toString() when Token is constructed using null identifier.
HADOOP-8818: Use equals instead == in MD5MD5CRC32FileChecksum and TFileDumper.
HADOOP-8934: Shell command ls should include sort options.
HADOOP-9121: InodeTree.java has redundant check for vName while throwing exception.
HADOOP-9242: Duplicate surefire plugin config in hadoop-common.
HADOOP-9477: Add posixGroups support for LDAP groups mapping service.
HADOOP-10035: Cleanup TestFilterFileSystem.
HADOOP-10048: LocalDirAllocator should avoid holding locks while accessing the filesystem.
HADOOP-1023: Add more logging in WASB page blob read function.
HADOOP-1026: Logging bug in WASB causing out of index exception.
HADOOP-10300: Allowed deferred sending of call responses.
HADOOP-10365: BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block.
HADOOP-10406: TestIPC.testIpcWithReaderQueuing may fail.
HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.
HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.
HADOOP-1062: added hadoop-azure-datalake and azure-data-lake-store-sdk to all the wasb locations.
HADOOP-1062: added hadoop-azure-datalake and azure-data-lake-store-sdk to all the wasb locations.
HADOOP-10774: Update KerberosTestUtils for hadoop-auth tests when using IBM Java.
HADOOP-10775: Shell operations to fail with meaningful errors on windows if winutils.exe not found.
HADOOP-10823: TestReloadingX509TrustManager is flaky.
HADOOP-10829: Iteration on CredentialProviderFactory.serviceLoader is thread-unsafe.
HADOOP-10965: Print fully qualified path in CommandWithDestination error messages.
HADOOP-10971: Add -C flag to make `hadoop fs -ls` print filenames only.
HADOOP-11031: Design Document for Credential Provider API.
HADOOP-11098: [JDK8] Max Non Heap Memory default changed between JDK7 and 8.
HADOOP-11180: Change log message "token.Token: Cannot find class for token kind kms-dt" to debug.
HADOOP-11212: NetUtils.wrapException to handle SocketException explicitly.
HADOOP-11218: Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.
HADOOP-11252: RPC client does not time out by default.
HADOOP-11262: Enable YARN to use S3A.
HADOOP-11361: Fix a race condition in MetricsSourceAdapter.updateJmxCache.
HADOOP-11404: Clarify the "expected client Kerberos principal is null" authorization message.
HADOOP-11418: Property "io.compression.codec.lzo.class" does not work with other value besides default.
HADOOP-11491: HarFs incorrectly declared as requiring an authority.
HADOOP-11581: Multithreaded correctness Warnings #org.apache.hadoop.fs.shell.Ls.
HADOOP-11613: Remove commons-httpclient dependency from hadoop-azure.
HADOOP-11628: SPNEGO auth does not work with CNAMEs in JDK8. (Daryn Sharp via stevel).
HADOOP-11677: Add cookie flags for logs and static contexts.
HADOOP-11685: StorageException complaining "no lease ID" during HBase distributed log splitting.
HADOOP-11687: Ignore x-* and response headers when copying an Amazon S3 object.
HADOOP-11692: Improve authentication failure WARN message to avoid user confusion.
HADOOP-11713: ViewFileSystem should support snapshot methods.
HADOOP-11717: Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth.
HADOOP-11772: RPC Invoker relies on static ClientCache which has synchronized(this) blocks.
HADOOP-11780: Prevent IPC reader thread death.
HADOOP-11785: Reduce the number of listStatus operation in distcp buildListing.
HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm.
HADOOP-11812: Implement listLocatedStatus for ViewFileSystem to speed up split calculation.
HADOOP-11819: HttpServerFunctionalTest#prepareTestWebapp should create web app directory if it does not exist.
HADOOP-11827: Speed-up distcp buildListing() using threadpool.
HADOOP-11859: PseudoAuthenticationHandler fails with httpcomponents v4.4.
HADOOP-11864: JWTRedirectAuthenticationHandler breaks java8 javadocs.
HADOOP-11876: Refactor code to make it more readable, minor maybePrintStats bug.
HADOOP-11901: BytesWritable fails to support 2G chunks due to integer overflow.
HADOOP-11918: Listing an empty s3a root directory throws FileNotFound.
HADOOP-11924: Tolerate JDK-8047340-related exceptions in Shell#isSetSidAvailable preventing class init.
HADOOP-11932: MetricsSinkAdapter may hang when being stopped.
HADOOP-11960: Enable Azure-Storage Client Side logging.
HADOOP-12001: Fixed LdapGroupsMapping to include configurable Posix UID and GID attributes during the search.
HADOOP-12006: Remove unimplemented option for from document in branch-2.7.
HADOOP-12009: Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus. backport to 2.8.
HADOOP-12017: Hadoop archives command should use configurable replication factor when closing.
HADOOP-12043: Display warning if defaultFs is not set when running fs commands.
HADOOP-12045: Enable LocalFileSystem#setTimes to change atime.
HADOOP-12049: Control http authentication cookie persistence via configuration.
HADOOP-12052: IPC client downgrades all exception types to IOE, breaks callers trying to use them.
HADOOP-12054: RPC client should not retry for InvalidToken exceptions.
HADOOP-12056: Use DirectoryStream in DiskChecker#checkDirs to detect errors when listing a directory.
HADOOP-12058: Fix dead links to DistCp and Hadoop Archives pages.
HADOOP-12059: S3Credentials should support use of CredentialProvider.
HADOOP-12061: Incorrect command in single cluster setup document.
HADOOP-12073: Azure FileSystem PageBlobInputStream does not return -1 on EOF.
HADOOP-12074: in Shell.java#runCommand() rethrow InterruptedException as InterruptedIOException.
HADOOP-12076: Incomplete Cache Mechanism in CredentialProvider API.
HADOOP-12087: [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel).'.
HADOOP-12089: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.
HADOOP-12095: org.apache.hadoop.fs.shell.TestCount fails.
HADOOP-12097: Allow port range to be specified while starting webapp.
HADOOP-12100: ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermission.
HADOOP-12103: Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing.
HADOOP-12107: long running apps may have a huge number of StatisticsData instances under FileSystem.
HADOOP-12119: hadoop fs -expunge does not work for federated namespace.
HADOOP-12158: Improve error message in TestCryptoStreamsWithOpensslAesCtrCryptoCodec when OpenSSL is not installed.
HADOOP-12159: Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces.
HADOOP-12161: Add getStoragePolicy API to the FileSystem interface.
HADOOP-12169: ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse.
HADOOP-12173: NetworkTopology::add calls toString always.
HADOOP-12185: NetworkTopology is not efficient adding/getting/removing nodes.
HADOOP-12186: ActiveStandbyElector shouldn't call monitorLockNodeAsync multiple times.
HADOOP-12189: Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.
HADOOP-12191: Bzip2Factory is not thread safe.
HADOOP-12213: Interrupted exception can occur when Client#stop is called.
HADOOP-12229: Fix inconsistent subsection titles in filesystem.md.
HADOOP-12230: hadoop-project declares duplicate, conflicting curator dependencies.
HADOOP-12232: Upgrade Tomcat dependency to 6.0.44.
HADOOP-12239: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.
HADOOP-12242: Add in-page TOC to filesystem specification pages.
HADOOP-12252: LocalDirAllocator should not throw NPE with empty string configuration.
HADOOP-12258: Need translate java.nio.file.NoSuchFileException to FileNotFoundException to avoid regression.
HADOOP-12259: Utility to Dynamic port allocation.
HADOOP-12269: Update aws-sdk dependency to 1.10.6.
HADOOP-12280: Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded.
HADOOP-12284: UserGroupInformation doAs can throw misleading exception.
HADOOP-12291: Add support for nested groups in LdapGroupsMapping.
HADOOP-12292: Make use of DeleteObjects optional.
HADOOP-12296: when setnetgrent returns 0 in linux, exception should be thrown.
HADOOP-12304: Applications using FileContext fail with the default file system configured to be wasb/s3/etc.
HADOOP-12317: Applications fail on NM restart on some linux distro because NM container recovery declares AM container as LOST.
HADOOP-12324: Better exception reporting in SaslPlainServer. (Mike Yoder via stevel).
HADOOP-12325: RPC Metrics : Add the ability track and log slow RPCs.
HADOOP-12334: Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries.
HADOOP-12345: Pad hostname correctly in CredentialsSys.java.
HADOOP-12346: Increase some default timeouts / retries for S3a connector.
HADOOP-12348: MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.
HADOOP-12350: WASB Logging: Improve WASB Logging around deletes, reads and writes.
HADOOP-12358: Add -safely flag to rm to prompt when deleting many files.
HADOOP-12359: hadoop fs -getmerge doc is wrong.
HADOOP-12366: expose calculated paths.
HADOOP-12374: Description of hdfs expunge command is confusing.
HADOOP-12386: RetryPolicies.RETRY_FOREVER should be able to specify a retry interval.
HADOOP-12404: Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class.
HADOOP-12406: Fixed AbstractMapWritable.readFields to use the thread's ClassLoader to load class instead of System ClassLoader.
HADOOP-12407: Test failing: hadoop.ipc.TestSaslRPC.
HADOOP-12413: AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.
HADOOP-12415: Add io.netty dependency to hadoop-nfs and to hadoop-hdfs, needed to build Bigtop successfuly, see BIGTOP-2049.
HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.
HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.".
HADOOP-12418: TestRPC.testRPCInterruptedSimple fails intermittently.
HADOOP-12423: Handle failure of registering shutdownhook by ShutdownHookManager in static block.
HADOOP-12426: Add Entry point for Kerberos health check.
HADOOP-12437: Allow SecurityUtil to lookup alternate hostnames.
HADOOP-12438: TestLocalFileSystem tests can fail on Windows after HDFS-8767 fix for handling pipe.
HADOOP-12440: TestRPC#testRPCServerShutdown did not produce the desired thread states before shutting down.
HADOOP-12441: Fixed kill-command behavior to work correctly across OSes by using bash shell built-in.
HADOOP-12442: Display help if the command option to 'hdfs dfs' is not valid.
HADOOP-12444: Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel.
HADOOP-12449: TestDNS and TestNetUtils failing if no network.
HADOOP-12453: Support decoding KMS Delegation Token with its own Identifier.
HADOOP-12457: [JDK8] Fix a failure of compiling common by javadoc.
HADOOP-12463: Fix TestShell.testGetSignalKillCommand failure on windows.
HADOOP-12464: Interrupted client may try to fail-over and retry.
HADOOP-12465: Incorrect javadoc in WritableUtils.java.
HADOOP-12469: distcp should not ignore the ignoreFailures option.
HADOOP-12472: Make GenericTestUtils.assertExceptionContains robust.
HADOOP-12478: Shell.getWinUtilsPath() has been renamed Shell.getWinutilsPath().
HADOOP-12481: JWTRedirectAuthenticationHandler doesn't Retain Original Query String.
HADOOP-12482: Race condition in JMX cache update.
HADOOP-12483: Maintain wrapped SASL ordering for postponed IPC responses.
HADOOP-12484: Single File Rename Throws Incorrectly In Potential Race Condition Scenarios.
HADOOP-12508: delete fails with exception when lease is held on blob.
HADOOP-12520: Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests.
HADOOP-12526: there are duplicate dependency definitions in pom's.
HADOOP-12533: Introduce FileNotFoundException in WASB for read and seek API.
HADOOP-12535: Run FileSystem contract tests with hadoop-azure.
HADOOP-12537: S3A to support Amazon STS temporary credentials.
HADOOP-12540: TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown.
HADOOP-12542: TestDNS fails on Windows after HADOOP-12437.
HADOOP-12545: Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp.
HADOOP-12548: Read s3a creds from a Credential Provider.
HADOOP-12551: Introduce FileNotFoundException for WASB FileSystem API.
HADOOP-12555: WASB to read credentials from a credential provider.
HADOOP-12559: KMS connection failures should trigger TGT renewal.
HADOOP-12564: Upgrade JUnit3 TestCase to JUnit 4 in org.apache.hadoop.io package.
HADOOP-12565: Replace DSA with RSA for SSH key type in SingleCluster.md.
HADOOP-12568: Update core-default.xml to describe posixGroups support.
HADOOP-12570: HDFS Secure Mode Documentation updates.
HADOOP-12573: TestRPC.testClientBackOff failing.
HADOOP-12577: Bump up commons-collections version to 3.2.2 to address a security flaw.
HADOOP-12588: Fix intermittent test failure of TestGangliaMetrics.
HADOOP-12589: Fix intermittent test failure of TestCopyPreserveFlag.
HADOOP-12590: TestCompressorDecompressor failing without stack traces.
HADOOP-12602: TestMetricsSystemImpl#testQSize occasionally fails.
HADOOP-12603: TestSymlinkLocalFSFileContext#testSetTimesSymlinkToDir occasionally fail.
HADOOP-12604: Exception may be swallowed in KMSClientProvider.
HADOOP-12605: Fix intermittent failure of TestIPC.testIpcWithReaderQueuingi.
HADOOP-12608: Fix exception message in WASB when connecting with anonymous credential.
HADOOP-12609: Fix intermittent failure of TestDecayRpcScheduler.
HADOOP-12613: TestFind.processArguments occasionally fails.
HADOOP-12617: SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal.
HADOOP-12622: Improve the loggings in RetryPolicies and RetryInvocationHandler.
HADOOP-12634: Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy.
HADOOP-12635: Adding Append API support for WASB.
HADOOP-12636: Prevent ServiceLoader failure init for unused FileSystems.
HADOOP-12653: Use SO_REUSEADDR to avoid getting "Address already in use" when using kerberos and attempting to bind to any port on the local IP address.
HADOOP-12659: Incorrect usage of config parameters in token manager of KMS.
HADOOP-12663: Remove Hard-Coded Values From FileSystem.java.
HADOOP-12666: Support Microsoft Azure Data Lake - as a file system in Hadoop.
HADOOP-12667: s3a to support createNonRecursive API.
HADOOP-12668: Support excluding weak Ciphers in HttpServer2 through ssl- server.conf.
HADOOP-12672: RPC timeout should not override IPC ping interval.
HADOOP-12675: Fix description about retention period in usage of expunge command.
HADOOP-12678: Handle empty rename pending metadata file during atomic rename in redo path.
HADOOP-12682: Fix TestKMS#testKMSRestart* failure.
HADOOP-12688: Fix deadlinks in Compatibility.md.
HADOOP-12689: S3 filesystem operations stopped working correctly.
HADOOP-12691: Add CSRF Filter for REST APIs to Hadoop Common.
HADOOP-12691: Move files to correct location.
HADOOP-12696: Add tests for S3Filesystem Contract.
HADOOP-12699: TestKMS#testKMSProvider intermittently fails during 'test rollover draining'.
HADOOP-12700: Remove unused import in TestCompressorDecompressor.java.
HADOOP-12706: TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally.
HADOOP-12711: Remove dependency on commons-httpclient for ServletUtil.
HADOOP-12715: TestValueQueue#testgetAtMostPolicyALL fails intermittently.
HADOOP-12716: KerberosAuthenticator#doSpnegoSequence use incorrect class to determine isKeyTab in JDK8.
HADOOP-12717: NPE when trying to rename a directory in Windows Azure Storage FileSystem.
HADOOP-12723: S3A: Add ability to plug in any AWSCredentialsProvider.
HADOOP-12731: Remove useless boxing/unboxing code.
HADOOP-12736: TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out.
HADOOP-12738: Create unit test to automatically compare Common related classes and core-default.xml.
HADOOP-12751: While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple. (Bolke de Bruin via stevel).
HADOOP-12752: Improve diagnostics/use of envvar/sysprop credential propagation.
HADOOP-12758: Extend CSRF Filter with UserAgent Checks.
HADOOP-12761: incremental maven build is not really incremental.
HADOOP-12765: HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections.
HADOOP-12767: Update apache httpclient version to 4.5.2; httpcore to 4.4.4.
HADOOP-12772: NetworkTopologyWithNodeGroup.getNodeGroup() can loop infinitely for invalid 'loc' values.
HADOOP-12773: HBase classes fail to load with client/job classloader enabled.
HADOOP-12774: s3a should use UGI.getCurrentUser.getShortname() for username.
HADOOP-12776: Remove getaclstatus call for non-acl commands in getfacl.
HADOOP-12780: During WASB atomic rename handle crash when one directory has been renamed but not file under it.
HADOOP-12782: Faster LDAP group name resolution with ActiveDirectory.
HADOOP-12786: "hadoop key" command usage is not documented.
HADOOP-12787: KMS SPNEGO sequence does not work with WEBHDFS.
HADOOP-12788: OpensslAesCtrCryptoCodec should log which random number generator is used.
HADOOP-12789: log classpath of ApplicationClassLoader at INFO level.
HADOOP-12792: TestUserGroupInformation#testGetServerSideGroups fails in chroot.
HADOOP-12793: Write a new group mapping service guide.
HADOOP-12794: Support additional compression levels for GzipCodec.
HADOOP-12795: KMS does not log detailed stack trace for unexpected errors.
HADOOP-12800: Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs.
HADOOP-12801: Suppress obsolete S3FileSystem tests.
HADOOP-12804: Read Proxy Password from Credential Providers in S3 FileSystem.
HADOOP-12805: Annotate CanUnbuffer with @InterfaceAudience.Public.
HADOOP-12807: S3AFileSystem should read AWS credentials from environment variables.
HADOOP-12810: FileSystem#listLocatedStatus causes unnecessary RPC calls.
HADOOP-12817: Enable TLS v1.1 and 1.2.
HADOOP-12825: Log slow name resolutions.
HADOOP-12828: Print user when services are started.
HADOOP-12829: StatisticsDataReferenceCleaner swallows interrupt exceptions.
HADOOP-12831: LocalFS/FSOutputSummer NPEs in constructor if bytes per checksum set to 0.
HADOOP-12844: Recover when S3A fails on IOException in read().
HADOOP-12846: Credential Provider Recursive Dependencies.
HADOOP-12847: hadoop daemonlog should support https and SPNEGO for Kerberized cluster.
HADOOP-12849: TestSymlinkLocalFSFileSystem fails intermittently.
HADOOP-12851: S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders.
HADOOP-12853: Change WASB documentation regarding page blob support.
HADOOP-12870: Fix typo admininistration in CommandsManual.md.
HADOOP-12871: Fix dead link to NativeLibraries.html in CommandsManual.md.
HADOOP-12872: Fix formatting in ServiceLevelAuth.md.
HADOOP-12875: [Azure Data Lake] Support for contract test and unit test cases.
HADOOP-12878: KMS SPNEGO sequence does not work with WEBHDFS.
HADOOP-12886: Exclude weak ciphers in SSLFactory through ssl-server.xml.
HADOOP-12891: S3AFileSystem should configure Multipart Copy threshold and chunk size.
HADOOP-12893: Update LICENSE.txt and NOTICE.txt.
HADOOP-12893: Update LICENSE.txt and NOTICE.txt.
HADOOP-12893: Update LICENSE.txt and NOTICE.txt".
HADOOP-12895: SSLFactory#createSSLSocketFactory exception message is wrong.
HADOOP-12901: Add warning log when KMSClientProvider cannot create a connection to the KMS server.
HADOOP-12902: JavaDocs for SignerSecretProvider are out-of-date in AuthenticationFilter.
HADOOP-12903: IPC Server should allow suppressing exception logging by type,.
HADOOP-12903: IPC Server should allow suppressing exception logging by type, not log 'server too busy' messages.
HADOOP-12906: AuthenticatedURL should convert a 404/Not Found into an FileNotFoundException.
HADOOP-12909: Change ipc.Client to support asynchronous calls.
HADOOP-12916: Allow RPC scheduler/callqueue backoff using response times.
HADOOP-12926: lz4.c does not detect 64-bit mode properly.
HADOOP-12929: JWTRedirectAuthenticationHandler must accommodate null expiration time.
HADOOP-12943: Add -w -r options in dfs -test command.
HADOOP-12947: Update documentation Hadoop Groups Mapping to add static group mapping, negative cache.
HADOOP-12950: ShutdownHookManager should have a timeout for each of the Registered shutdown hook.
HADOOP-12954: Add a way to change hadoop.security.token.service.use_ip.
HADOOP-12957: Limit the number of outstanding async calls.
HADOOP-12958: PhantomReference for filesystem statistics can trigger OOM.
HADOOP-12962: KMS key names are incorrectly encoded when creating key.
HADOOP-12963: Allow using path style addressing for accessing the s3 endpoint.
HADOOP-12964: Http server vulnerable to clickjacking.
HADOOP-12973: Make DU pluggable.
HADOOP-12975: Add jitter to CachingGetSpaceUsed's thread.
HADOOP-12977: s3a to handle delete("/", true) robustly.
HADOOP-12982: Document missing S3A and S3 properties.
HADOOP-12984: Add GenericTestUtils.getTestDir method and use it for emporary directory in tests.
HADOOP-12985: Support MetricsSource interface for DecayRpcScheduler Metrics.
HADOOP-12989: Some tests in org.apache.hadoop.fs.shell.find occasionally time out.
HADOOP-12993: Change ShutdownHookManger complete shutdown log from INFO to DEBUG.
HADOOP-12994: Specify PositionedReadable, add contract tests, fix problems.
HADOOP-13008: Add XFS Filter for UIs to Hadoop Common.
HADOOP-13011: Clearly Document the Password Details for Keystore-based Credential Providers.
HADOOP-13018: Make Kdiag check whether hadoop.token.files points to existent and valid files.
HADOOP-13024: Distcp with -delete feature on raw data not implemented.
HADOOP-13026: Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator.
HADOOP-13028: add low level counter metrics for S3A; use in read performance tests.
HADOOP-13030: Handle special characters in passwords in KMS startup script.
HADOOP-13034: Log message about input options in distcp lacks some items.
HADOOP-13037: Refactor Azure Data Lake Store as an independent FileSystem.
HADOOP-13039: Add documentation for configuration property ipc.maximum.data.length.
HADOOP-13042: Restore lost leveldbjni LICENSE and NOTICE changes.
HADOOP-13047: S3a Forward seek in stream length to be configurable.
HADOOP-13052: ChecksumFileSystem mishandles crc file permissions.
HADOOP-13058: S3A FS fails during init against a read-only FS if multipart purge.
HADOOP-13065: Add a new interface for retrieving FS and FC Statistics.
HADOOP-13072: WindowsGetSpaceUsed constructor should be public.
HADOOP-13079: Add -q option to Ls to print ? instead of non-printable characters.
HADOOP-13084: Fix ASF License warnings in branch-2.7.
HADOOP-13098: Dynamic LogLevel setting page should accept case-insensitive log level string.
HADOOP-13103: Group resolution from LDAP may fail on javax.naming.ServiceUnavailableException.
HADOOP-13105: Support timeouts in LDAP queries in LdapGroupsMapping.
HADOOP-13113: Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel.
HADOOP-13116: Jets3tNativeS3FileSystemContractTest does not run.
HADOOP-13122: Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.
HADOOP-13130: s3a failures can surface as RTEs, not IOEs.
HADOOP-13131: Add tests to verify that S3A supports SSE-S3 encryption.
HADOOP-13138: Unable to append to a SequenceFile with Compression.NONE.
HADOOP-13139:: S3a to use thread pool that blocks clients.
HADOOP-13140: FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes.
HADOOP-13145: In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.
HADOOP-13146: Refactor RetryInvocationHandler.
HADOOP-13155: Implement TokenRenewer to renew and cancel delegation tokens in KMS.
HADOOP-13158: S3AFileSystem#toString might throw NullPointerException due to null cannedACL.
HADOOP-13159: Fix potential NPE in Metrics2 source for DecayRpcScheduler.
HADOOP-13162: Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs.
HADOOP-13163: Reuse pre-computed filestatus in Distcp-CopyMapper.
HADOOP-13164: Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories.
HADOOP-13168: Support Future.get with timeout in ipc async calls.
HADOOP-13169: Randomize file list in SimpleCopyListing.
HADOOP-13171: Add StorageStatistics to S3A; instrument some more operations.
HADOOP-13179: GenericOptionsParser is not thread-safe because commons-cli OptionBuilder is not thread-safe.
HADOOP-13183: S3A proxy tests fail after httpclient/httpcore upgrade.
HADOOP-13188: S3A file-create should throw error rather than overwrite directories.
HADOOP-13192: org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly.
HADOOP-13197: Add non-decayed call metrics for DecayRpcScheduler.
HADOOP-13202: Avoid possible overflow in org.apache.hadoop.util.bloom.BloomFilter#getNBytes.
HADOOP-13203: S3a: Consider reducing the number of connection aborts by setting correct length in s3 request.
HADOOP-13204: Add support for SSE-KMS and SSE-C in s3a filesystem.
HADOOP-13207: Specify FileSystem listStatus, listFiles and RemoteIterator.
HADOOP-13208: S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories.
HADOOP-13212: Provide an option to set the socket buffers in S3AFileSystem.
HADOOP-13213: Small Documentation bug with AuthenticatedURL in hadoop-auth.
HADOOP-13237: : s3a initialization against public bucket fails if caller lacks any credentials.
HADOOP-13239: Deprecate s3:// in branch-2.
HADOOP-13241: document s3a better.
HADOOP-13242: Authenticate to Azure Data Lake using client ID and keys.
HADOOP-13244: o.a.h.ipc.Server#Server should honor handlerCount when queueSizePerHandler is specified in consturctor.
HADOOP-13249: RetryInvocationHandler need wrap InterruptedException in IOException when call Thread.sleep.
HADOOP-13251: Authenticate with Kerberos credentials when renewing KMS delegation token.
HADOOP-13252: Tune S3A provider plugin mechanism.
HADOOP-13254: Create framework for configurable disk checkers.
HADOOP-13255: KMSClientProvider should check and renew tgt when doing delegation token operations.
HADOOP-13257: Improve Azure Data Lake contract tests.
HADOOP-13263: Reload cached groups in background after expiry.
HADOOP-13270: BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks.
HADOOP-13280: FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps.
HADOOP-13283: Support reset operation for new global storage statistics and per FS storage stats.
HADOOP-13285: DecayRpcScheduler MXBean should only report decayed CallVolumeSummary.
HADOOP-13287: TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +.
HADOOP-13288: Guard null stats key in FileSystemStorageStatistics.
HADOOP-13290: Appropriate use of generics in FairCallQueue.
HADOOP-13291: Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented.
HADOOP-13299: JMXJsonServlet is vulnerable to TRACE.
HADOOP-13305: Define common statistics names across schemes.
HADOOP-13309: Document S3A known limitations in file ownership and permission model.
HADOOP-13315: FileContext#umask is not initialized properly.
HADOOP-13316: Enforce Kerberos authentication for required ops in DelegationTokenAuthenticator.
HADOOP-13320: Fix arguments check in documentation for WordCount v2.0.
HADOOP-13323: Downgrade stack trace on FS load from Warn to debug.
HADOOP-13324: s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints).
HADOOP-13336: S3A to support per-bucket configuration.
HADOOP-13350: Additional fix to LICENSE and NOTICE.
HADOOP-13351: TestDFSClientSocketSize buffer size tests are flaky.
HADOOP-13352: Make X-FRAME-OPTIONS configurable in HttpServer2.
HADOOP-13353: LdapGroupsMapping getPassward shouldn't return null when IOException throws.
HADOOP-13354: Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients.
HADOOP-13368: DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation.
HADOOP-13381: KMS clients should use KMS Delegation Tokens from current UGI.
HADOOP-13382: Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Ported to Hortonworks HDP-2.5 by Matt Foley.
HADOOP-13387: users always get told off for using S3 —even when not using it.
HADOOP-13389: TestS3ATemporaryCredentials.testSTS error when using IAM credentials.
HADOOP-13392: [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured.
HADOOP-13393: Omit unsupported fs.defaultFS setting in ADLS documentation.
HADOOP-13403: AzureNativeFileSystem rename/delete performance improvements.
HADOOP-13405: doc for fs.s3a.acl.default indicates incorrect values.
HADOOP-13406: S3AFileSystem: Consider reusing filestatus in delete() and mkdirs().
HADOOP-13422: ZKDelegationTokenSecretManager JaasConfig does not work well with other ZK users in process.
HADOOP-13433: Race in UGI.reloginFromKeytab.
HADOOP-13434: Add quoting to Shell class.
HADOOP-13443: KMS should check the type of underlying keyprovider of KeyProviderExtension before falling back to default.
HADOOP-13446: Support running isolated unit tests separate from AWS integration tests.
HADOOP-13447: Refactor S3AFileSystem to support introduction of separate metadata repository and tests.
HADOOP-13458: LoadBalancingKMSClientProvider#doOp should log IOException stacktrace.
HADOOP-13459: hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time.
HADOOP-13461: NPE in KeyProvider.rollNewVersion.
HADOOP-13466: Add an AutoCloseableLock class.
HADOOP-13467: Shell#getSignalKillCommand should use the bash builtin on Linux.
HADOOP-13470: GenericTestUtils$LogCapturer is flaky.
HADOOP-13494: ReconfigurableBase can log sensitive information.
HADOOP-13502: Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests.
HADOOP-13512: ReloadingX509TrustManager should keep reloading in case of exception.
HADOOP-13513: Java 1.7 support for org.apache.hadoop.fs.azure testcases.
HADOOP-13535: Add jetty6 acceptor startup issue workaround to branch-2.
HADOOP-13540: improve section on troubleshooting s3a auth problems.
HADOOP-13541: explicitly declare the Joda time version S3A depends on.
HADOOP-13546: Override equals and hashCode to avoid connection leakage.
HADOOP-13552: RetryInvocationHandler logs all remote exceptions.
HADOOP-13552: RetryInvocationHandler logs all remote exceptions.
HADOOP-13558: UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket.
HADOOP-13560: S3ABlockOutputStream to support huge (many GB) file writes.
HADOOP-13565: KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request.
HADOOP-13590: Retry until TGT expires even if the UGI renewal thread encountered exception.
HADOOP-13599: s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown.
HADOOP-13614: Purge some superfluous/obsolete S3 FS tests that are slowing test runs down.
HADOOP-13638: KMS should set UGI's Configuration object properly.
HADOOP-13674: S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.
HADOOP-13675: Bug in return value for delete() calls in WASB.
HADOOP-13680: fs.s3a.readahead.range to use getLongBytes.
HADOOP-13686: Adding additional unit test for Trash (I).
HADOOP-13692: hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.
HADOOP-13707: If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed.
HADOOP-13710: Supress CachingGetSpaceUsed from logging interrupted exception stacktrace.
HADOOP-13716: Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup.
HADOOP-13727: S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider.
HADOOP-13735: ITestS3AFileContextStatistics.testStatistics() failing.
HADOOP-13737: Cleanup DiskChecker interface.
HADOOP-13738: DiskChecker should perform some disk IO.
HADOOP-13742: Expose NumOpenConnectionsPerUser as a metric. Brahma Reddy Battula.
HADOOP-13749: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.
HADOOP-13782: Make MutableRates metrics thread-local write, aggregate-on- read.
HADOOP-13792: Stackoverflow for schemeless defaultFS with trailing slash.
HADOOP-13801: regression: ITestS3AMiniYarnCluster failing.
HADOOP-13804: MutableStat mean loses accuracy if add(long, long) is used.
HADOOP-13822: Use GlobalStorageStatistics.INSTANCE.reset() at FileSystem#clearStatistics(). Contribute by Brahma Reddy Battula.
HADOOP-13823: s3a rename: fail if dest file exists.
HADOOP-13824: FsShell can suppress the real error if no error message is present.
HADOOP-13826: S3A Deadlock in multipart copy due to thread pool limits.
HADOOP-13830: Intermittent failure of ITestS3NContractRootDir.testRecursiveRootListing "Can not create a Path from an empty string". .
HADOOP-13831: Correct check for error code to detect Azure Storage Throttling and provide retries.
HADOOP-13838: KMSTokenRenewer should close providers.
HADOOP-13847: KMSWebApp should close KeyProviderCryptoExtension.
HADOOP-13855: Fix a couple of the s3a statistic names to be consistent with the rest.
HADOOP-13857: S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs.
HADOOP-13863: Azure: Add a new SAS key mode for WASB.
HADOOP-13871: ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful.
HADOOP-13890: Maintain HTTP/host as SPNEGO SPN support and fix KerberosName parsing.
HADOOP-13928: TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime error.
HADOOP-13929: ADLS connector should not check in contract-test- options.xml.
HADOOP-13930: Azure: Add Authorization support to WASB.
HADOOP-13945: Azure: Add Kerberos and Delegation token support to WASB client.
HADOOP-13956: Read ADLS credentials from Credential Provider.
HADOOP-13962: Update ADLS SDK to 2.1.4.
HADOOP-13988: KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser.
HADOOP-14017: User friendly name for ADLS user and group.
HADOOP-14019: Fix some typos in the s3a docs.
HADOOP-14028: S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.
HADOOP-14029: Fix KMSClientProvider for non-secure proxyuser use case.
HADOOP-14038: Rename ADLS credential properties.
HADOOP-14048: REDO operation of WASB#AtomicRename should create placeholder blob for destination folder.
HADOOP-14049: Honour AclBit flag associated to file/folder permission for Azure datalake account.
HADOOP-14055: SwiftRestClient includes pass length in exception if auth fails.
HADOOP-14081: S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock).
HADOOP-14091: AbstractFileSystem implementaion for 'wasbs' scheme.
HADOOP-14100: Upgrade Jsch jar to latest version to fix vulnerability in old versions.
HADOOP-14102: Relax error message assertion in S3A test ITestS3AEncryptionSSEC.
HADOOP-14113: Review ADL Docs.
HADOOP-14123: Remove misplaced ADL service provider config file for FileSystem.
HADOOP-14138: Remove S3A ref from META-INF service discovery, rely on existing core-default entry.
HADOOP-14150: Implement getHomeDirectory() method in NativeAzureFileSystem.
HADOOP-14153: ADL module has messed doc structure.
HADOOP-14170: FileSystemContractBaseTest is not cleaning up test directory clearly.
HADOOP-14173: Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE.
HADOOP-14174: Set default ADLS access token provider type to ClientCredential.
HADOOP-14195: CredentialProviderFactory$getProviders is not thread-safe.
HADOOP-14196: Azure Data Lake doc is missing required config entry.
HADOOP-14197: Fix ADLS doc for credential provider.
HADOOP-14205: No FileSystem for scheme: adl.
HADOOP-14214: DomainSocketWatcher::add()/delete() should not self interrupt while looping await().
HADOOP-14230: TestAdlFileSystemContractLive fails to clean up.
HADOOP-14241: Add ADLS sensitive config keys to default list.
HADOOP-14274: Azure: Simplify Ranger-WASB policy model.
HADOOP-14321: explicitly exclude s3a root dir ITests from parallel runs.
HADOOP-14349: Rename ADLS CONTRACT_ENABLE_KEY.
HADOOP-14351: Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache.
HADOOP-14361: Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently.
HADOOP-14416: Path starting with 'wasb///' not resolved correctly while authorizing with WASB-Ranger.
HADOOP-14442: Owner support for ranger-wasb integration.
HADOOP-14443: Azure: Support retry and client side failover for authorization, SASKey and delegation token generation.3.
HADOOP-14465: LdapGroupsMapping - support user and group search base.
HADOOP-14472: Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently.
HADOOP-14478: Optimize NativeAzureFsInputStream for positional reads.
HADOOP-14500: Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails.
HADOOP-14512: WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename.
HADOOP-14535: wasb: implement high-performance random access and seek of block blobs.
HADOOP-14536: Update azure-storage sdk to version 5.3.0.
HADOOP-14546: Azure: Concurrent I/O does not work when secure.mode is enabled.
HADOOP-14547: [WASB] the configured retry policy is not used for all storage operations.'.
HADOOP-14581: Restrict setOwner to list of user when security is enabled in wasb.
HADOOP-14640: Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation.
HADOOP-14642: wasb: add support for caching Authorization and SASKeys.
HADOOP-14658: branch-2 compilation is broken in hadoop-azure.
HADOOP-14680: Azure: IndexOutOfBoundsException in BlockBlobInputStream.'.
HADOOP-14722: Azure: BlockBlobInputStream position incorrect after seek.
HDFS-1477: Support reconfiguring dfs.heartbeat.interval and dfs.namenode.heartbeat.recheck-interval without NN restart.
HDFS-2043: TestHFlush failing intermittently.
HDFS-2580: NameNode#main(...) can make use of GenericOptionsParser.
HDFS-27: HDFS CLI with --config set to default config complains log file not found error.
HDFS-3677: dfs.namenode.edits.dir.required is missing from hdfs- default.xml.
HDFS-3918: EditLogTailer shouldn't log WARN when other node is in standby mode.
HDFS-4015: Safemode should count and report orphaned blocks.
HDFS-4176: EditLogTailer should call rollEdits with a timeout.
HDFS-4185: Add a metric for number of active leases.
HDFS-4366: Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks.
HDFS-4396: Add START_MSG/SHUTDOWN_MSG for ZKFC.
HDFS-4681: TestBlocksWithNotEnoughRacks#testCorruptBlockRereplicatedAcrossRacks fails using IBM java.
HDFS-4937: ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom.
HDFS-4946: Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable.
HDFS-5640: Add snapshot methods to FileContext.
HDFS-6054: MiniQJMHACluster should not use static port to avoid binding failure in unit test.
HDFS-6101: TestReplaceDatanodeOnFailure fails occasionally.
HDFS-6184: Capture NN's thread dump when it fails over.
HDFS-6407: Add sorting and pagination in the datanode tab of the NN Web UI.
HDFS-6481: DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs.
HDFS-6533: TestBPOfferService#testBasicFunctionalitytest fails intermittently.
HDFS-6581: Support for writing to single replica in RAM. Applied HDFS-6581.merge.15.patch to Champlain. This is a consolidated patch containing all feature changes and was applied to trunk.
HDFS-6666: Abort NameNode and DataNode startup if security is enabled but block access token is not enabled.
HDFS-6763: Initialize file system-wide quota once on transitioning to active.
HDFS-6832: Fix the usage of 'hdfs namenode' command.
HDFS-6860: BlockStateChange logs are too noisy.
HDFS-6874: Add GETFILEBLOCKLOCATIONS operation to HttpFS.
HDFS-6945: BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed.
HDFS-7163: WebHdfsFileSystem should retry reads according to the configured retry policy.
HDFS-7166: SbNN Web UI shows #Under replicated blocks and #pending deletion blocks.
HDFS-7284: Add more debug info to BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas.
HDFS-7314: When the DFSClient lease cannot be renewed, abort open-for- write files rather than the entire DFSClient.
HDFS-7390: Provide JMX metrics per storage type.
HDFS-7452: skip StandbyException log for getCorruptFiles() .
HDFS-7483: Display information per tier on the Namenode UI.
HDFS-7597: DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping.
HDFS-7608: hdfs dfsclient newConnectedPeer has no write timeout.
HDFS-7609: Avoid retry cache collision when Standby NameNode loading edits.
HDFS-7645: Fix CHANGES.txt.
HDFS-7645: Rolling upgrade is restoring blocks from trash multiple times.
HDFS-7701: Support reporting per storage type quota and usage with hadoop/hdfs shell.
HDFS-7725: Incorrect "nodes in service" metrics caused all writes to fail.
HDFS-7758: Retire FsDatasetSpi#getVolumes() and use FsDatasetSpi#getVolumeRefs() instead.
HDFS-7833: DataNode reconfiguration does not recalculate valid volumes required, based on configured failed volumes tolerated.
HDFS-7858: Improve HA Namenode Failover detection on the client.
HDFS-7890: Improve information on Top users for metrics in RollingWindowsManager and lower log level.
HDFS-7923: The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages.
HDFS-7928: Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy.
HDFS-7933: fsck should also report decommissioning replicas.
HDFS-7964: Add support for async edit logging.
HDFS-7978: Add LOG.isDebugEnabled() guard for some LOG.debug(..).
HDFS-7990: IBR delete ack should not be delayed.
HDFS-8008: Support client-side back off when the datanodes are congested.
HDFS-8008: Support client-side back off when the datanodes are congested.
HDFS-8009: Signal congestion on the DataNode.
HDFS-8009: Signal congestion on the DataNode.
HDFS-8046: Allow better control of getContentSummary.
HDFS-8055: NullPointerException when topology script is missing.
HDFS-8071:.
HDFS-8099: Change "DFSInputStream has been closed already" message to debug log level.
HDFS-8101: DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS classes at runtime.
HDFS-8113: Add check for null BlockCollection pointers in BlockInfoContiguous structures.
HDFS-8131: Implement a space balanced block placement policy.
HDFS-8144: Split TestLazyPersistFiles into multiple tests.
HDFS-8152: Refactoring of lazy persist storage cases.
HDFS-8155: Support OAuth2 in WebHDFS.
HDFS-8180: AbstractFileSystem Implementation for WebHdfs.
HDFS-8205: CommandFormat#parse() should not parse option as value of option.
HDFS-8209: Support different number of datanode directories in MiniDFSCluster.
HDFS-8211: DataNode UUID is always null in the JMX counter.
HDFS-8219: setStoragePolicy with folder behavior is different after cluster restart.
HDFS-8224: Schedule a block for scanning if its metadata file is corrupt.
HDFS-8229: LAZY_PERSIST file gets deleted after NameNode restart.
HDFS-8232: Missing datanode counters when using Metrics2 sink interface.
HDFS-8247: TestDiskspaceQuotaUpdate#testAppendOverTypeQuota is failing.
HDFS-8276: LazyPersistFileScrubber should be disabled if scrubber interval configured zero.
HDFS-8278: When computing max-size-to-move in Balancer, count only the storage with remaining >= default block size. (szetszwo).
HDFS-8307: Spurious DNS Queries from hdfs shell.
HDFS-8309: Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows.
HDFS-8310: Fix TestCLI.testAll "help: help for find" on Windows.
HDFS-8311: DataStreamer.transfer() should timeout the socket InputStream.
HDFS-8312: Added permission check for moving file to Trash.
HDFS-8337: Accessing httpfs via webhdfs doesn't work from a jar with kerberos.
HDFS-8345: Storage policy APIs must be exposed via the FileSystem interface.
HDFS-8384: Allow NN to startup if there are files having a lease but are not under construction.
HDFS-8431: hdfs crypto class not found in Windows.
HDFS-8432: Introduce a minimum compatible layout version to allow downgrade in more rolling upgrade use cases.
HDFS-8435: Support CreateFlag in WebHDFS.
HDFS-8492: DN should notify NN when client requests a missing block.
HDFS-8539: Hdfs doesnt have class 'debug' in windows.
HDFS-8542: WebHDFS getHomeDirectory behavior does not match specification.
HDFS-8546: Use try with resources in DataStorage and Storage.
HDFS-8548: Minicluster throws NPE on shutdown.
HDFS-8549: Abort the balancer if an upgrade is in progress.
HDFS-8554: TestDatanodeLayoutUpgrade fails on Windows.
HDFS-8573: Move creation of restartMeta file logic from BlockReceiver to ReplicaInPipeline.
HDFS-8578: On upgrade, Datanode should process all storage/data dirs in parallel. .
HDFS-8581: ContentSummary on / skips further counts on yielding lock.
HDFS-8582: Support getting a list of reconfigurable config properties and do not generate spurious reconfig warnings.
HDFS-8593: Calculation of effective layout version mishandles comparison to current layout version in storage.
HDFS-8615: Correct HTTP method in WebHDFS document.
HDFS-8633: Fix setting of dfs.datanode.readahead.bytes in hdfs-default.xml to match DFSConfigKeys.
HDFS-8642: Make TestFileTruncate more reliable.
HDFS-8647: Abstract BlockManager's rack policy into BlockPlacementPolicy.
HDFS-8656: Preserve compatibility of ClientProtocol#rollingUpgrade after finalization.
HDFS-8659: Block scanner INFO message is spamming logs.
HDFS-8674: Improve performance of postponed block scans.
HDFS-8676: Delayed rolling upgrade finalization can cause heartbeat expiration.
HDFS-8696: Make the lower and higher watermark in the DN Netty server configurable.
HDFS-8709: Clarify automatic sync in FSEditLog#logEdit.
HDFS-8711: setSpaceQuota command should print the available storage type when input storage type is wrong.
HDFS-8722: Optimize datanode writes for small writes and flushes.
HDFS-8729: Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately which occasionally failed.
HDFS-8767: RawLocalFileSystem.listStatus() returns null for UNIX pipefile.
HDFS-8772: Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails. (Walter Su via wang).
HDFS-8778: TestBlockReportRateLimiting#testLeaseExpiration can deadlock.
HDFS-8785: TestDistributedFileSystem is failing in trunk.
HDFS-8792: BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory.
HDFS-8797: WebHdfsFileSystem creates too many connections for pread.
HDFS-8806: Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared.
HDFS-8809: HDFS fsck reports under construction blocks as CORRUPT.
HDFS-8815: DFS getStoragePolicy implementation using single RPC call.
HDFS-8816: Improve visualization for the Datanode tab in the NN UI.
HDFS-8818: Changes the global moveExecutor to per datanode executors and changes MAX_SIZE_TO_MOVE to be configurable. (szetszwo).
HDFS-8824: Do not use small blocks for balancing the cluster. (szetszwo).
HDFS-8826: In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo).
HDFS-8828: Utilize Snapshot diff report to build diff copy list in distcp.
HDFS-8829: Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning.
HDFS-8831: Trash Support for deletion in HDFS encryption zone.
HDFS-8844: TestHDFSCLI does not cleanup the test directory.
HDFS-8845: DiskChecker should not traverse the entire tree.
HDFS-8846: Add a unit test for INotify functionality across a layout version upgrade.
HDFS-8850: VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks.
HDFS-8852: HDFS architecture documentation of version 2.x is outdated about append write support.
HDFS-8855: Webhdfs client leaks active NameNode connections.
HDFS-8856: Make LeaseManager#countPath O(1).
HDFS-8859: Improve DataNode ReplicaMap memory footprint to save about 45%.
HDFS-8860: Remove unused Replica copyOnWrite code.
HDFS-8863: The remaining space check in BlockPlacementPolicyDefault is flawed.
HDFS-8867: Enable optimized block reports.
HDFS-8879: Quota by storage type usage incorrectly initialized upon namenode restart.
HDFS-8880: NameNode metrics logging.
HDFS-8883: NameNode Metrics : Add FSNameSystem lock Queue Length.
HDFS-8884: Fail-fast check in BlockPlacementPolicyDefault#chooseTarget.
HDFS-8885: ByteRangeInputStream used in webhdfs does not override available().
HDFS-8887: Expose storage type and storage ID in BlockLocation.
HDFS-8891: HDFS concat should keep srcs order.
HDFS-8896: DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager.
HDFS-8898: Create API and command-line argument to get quota and quota usage without detailed content summary.
HDFS-8911: NameNode Metric : Add Editlog counters as a JMX metric.
HDFS-8914: Document HA support in the HDFS HdfsDesign.md.
HDFS-8923: Add -source flag to balancer usage message. .
HDFS-8930: Block report lease may leak if the 2nd full block report comes when NN is still in safemode.
HDFS-8932: NPE thrown in NameNode when try to get TotalSyncCount metric before editLogStream initialization.
HDFS-8939: Test(S)WebHdfsFileContextMainOperations failing on branch-2.
HDFS-8946: Improve choosing datanode storage for block placement.
HDFS-8950: NameNode refresh doesn't remove DataNodes that are no longer in the allowed list.
HDFS-8964: When validating the edit log, do not read at or beyond the file offset that is being written.
HDFS-8965: Harden edit log reading code against out of memory errors.
HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.
HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.
HDFS-8983: NameNode support for protected directories.
HDFS-8983: NameNode support for protected directories.
HDFS-8995: Flaw in registration bookeeping can make DN die on reconnect.
HDFS-9008: Balancer#Parameters class could use a builder pattern.
HDFS-9008: was not backported correctly.
HDFS-9009: Send metrics logs to NullAppender by default.
HDFS-9019: Adding informative message to sticky bit permission denied exception.
HDFS-9033: dfsadmin -metasave prints "NaN" for cache used%.
HDFS-9034: StorageTypeStats Metric should not count failed storage.'.
HDFS-9038: DFS reserved space is erroneously counted towards non-DFS used.
HDFS-9042: Update document for the Storage policy name.
HDFS-9043: Doc updation for commands in HDFS Federation.
HDFS-9047: Retire libwebhdfs.
HDFS-9048: DistCp documentation is out-of-dated.
HDFS-9063: Correctly handle snapshot path for getContentSummary.
HDFS-9072: Fix random failures in TestJMXGet.
HDFS-9082: Change the log level in WebHdfsFileSystem.initialize() from INFO to DEBUG.
HDFS-9083: Replication violates block placement policy.
HDFS-9094: Add command line option to ask NameNode reload configuration.
HDFS-9106: Transfer failure during pipeline recovery causes permanent write failures.
HDFS-9107: Prevent NNs unrecoverable death spiral after full GC.
HDFS-9112: Improve error message for Haadmin when multiple name service IDs are configured.
HDFS-9128: TestWebHdfsFileContextMainOperations and TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows.
HDFS-9130: Use GenericTestUtils#setLogLevel to the logging level.
HDFS-9142: Separating Configuration object for namenode(s) in MiniDFSCluster.
HDFS-9145: Tracking methods that hold FSNamesytemLock for too long.
HDFS-9148: Incorrect assert message in TestWriteToReplica#testWriteToTemporary.
HDFS-9175: Change scope of 'AccessTokenProvider.getAccessToken()' and 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to public.
HDFS-9178: Slow datanode I/O can cause a wrong node to be marked bad.
HDFS-9184: Logging HDFS operation's caller context into audit logs.
HDFS-9188: Make block corruption related tests FsDataset- agnostic.
HDFS-9193: Fix incorrect references the usages of the DN in dfshealth.js.
HDFS-9198: Coalesce IBR processing in the NN.
HDFS-9205: Do not schedule corrupt blocks for replication. (szetszwo).
HDFS-9210: Fix some misuse of %n in VolumeScanner#printStats.
HDFS-9214: Support reconfiguring dfs.datanode.balance.max.concurrent.moves without DN restart.
HDFS-9220: Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum.
HDFS-9221: HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array.
HDFS-9226: Unit Tests failing due to mockito version issue.
HDFS-9239: DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness.
HDFS-9251: Refactor TestWriteToReplica and TestFsDatasetImpl to avoid explicitly creating Files in the tests code.
HDFS-9259: Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario.
HDFS-9264: Minor cleanup of operations on FsVolumeList#volumes. (Walter Su via lei).
HDFS-9273: ACLs on root directory may be lost after NN restart.
HDFS-9276: Failed to Update HDFS Delegation Token for long running application in HA mode.
HDFS-9289: Make DataStreamer#block thread safe and verify genStamp in commitBlock.
HDFS-9290: DFSClient#callAppend() is not backward compatible for slightly older NameNodes.
HDFS-9294: DFSClient deadlock when close file and failed to renew lease.
HDFS-9305: Delayed heartbeat processing causes storm of subsequent heartbeats.
HDFS-9311: Support optional offload of NameNode HA service health checks to a separate RPC server.
HDFS-9313: Possible NullPointerException in BlockManager if no excess replica can be chosen.
HDFS-9314: Improve BlockPlacementPolicyDefault's picking of excess replicas.
HDFS-9317: Document fsck -blockId and -storagepolicy options in branch-2.7.
HDFS-9343: Empty caller context considered invalid.
HDFS-9347: Invariant assumption in TestQuorumJournalManager.shutdown() is wrong.
HDFS-9349: Support reconfiguring fs.protected.directories without NN restart.
HDFS-9350: Avoid creating temprorary strings in Block.toString() and getBlockName() (Staffan Friberg via cmccabe).
HDFS-9354: Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows.
HDFS-9357: NN UI renders icons of decommissioned DN incorrectly.
HDFS-9358: TestNodeCount#testNodeCount timed out.
HDFS-9362: TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
HDFS-9364: Unnecessary DNS resolution attempts when creating NameNodeProxies.
HDFS-9383: TestByteArrayManager#testByteArrayManager fails.
HDFS-9384: TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server.
HDFS-9395: Make HDFS audit logging consistant.
HDFS-9397: Fix typo for readChecksum() LOG.warn in BlockSender.java.
HDFS-9402: Switch DataNode.LOG to use slf4j.
HDFS-9406: FSImage may get corrupted after deleting snapshot.
HDFS-9412: getBlocks occupies FSLock and takes too long to complete.
HDFS-9413: getContentSummary() on standby should throw StandbyException.
HDFS-9414: Refactor reconfiguration of ClientDatanodeProtocol for reusability.
HDFS-9415: Document dfs.cluster.administrators and dfs.permissions.superusergroup.
HDFS-9426: Rollingupgrade finalization is not backward compatible.
HDFS-9428: Fix intermittent failure of TestDNFencing.testQueueingWithAppend.
HDFS-9431: DistributedFileSystem#concat fails if the target path is relative.
HDFS-9434: Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messags.
HDFS-9445: Datanode may deadlock while handling a bad volume.
HDFS-9466: TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky.
HDFS-9467: Fix data race accessing writeLockHeldTimeStamp in FSNamesystem.
HDFS-9470: Encryption zone on root not loaded from fsimage after NN restart. Xiao Chen via wang.
HDFS-9476: TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail.
HDFS-9478: Reason for failing ipc.FairCallQueue contruction should be thrown.
HDFS-9484: NNThroughputBenchmark$BlockReportStats should not send empty block reports.
HDFS-9493: Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk.
HDFS-9500: Fix software version counts for DataNodes during rolling upgrade.
HDFS-9505: HDFS Architecture documentation needs to be refreshed.
HDFS-9516: Truncate file fails with data dirs on multiple disks.
HDFS-9517: Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes.
HDFS-9521: TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk.
HDFS-9525: hadoop utilities need to support provided delegation tokens.
HDFS-9530: ReservedSpace is not cleared for abandoned Blocks.
HDFS-9533: seen_txid in the shared edits directory is modified during bootstrapping.
HDFS-9534: Add CLI command to clear storage policy from a path.
HDFS-9549: TestCacheDirectives#testExceedsCapacity is flaky.
HDFS-9555: LazyPersistFileScrubber should still sleep if there are errors in the clear progress.
HDFS-9557: Reduce object allocation in PB conversion. (Daryn Sharp via cnauroth).
HDFS-9566: Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method.
HDFS-9569: 's message change.
HDFS-9569: Log the name of the fsimage being loaded for better supportability.
HDFS-9572: Prevent DataNode log spam if a client connects on the data transfer port but sends no data.
HDFS-9574: Reduce client failures during datanode restart.
HDFS-9584: NPE in distcp when ssl configuration file does not exist in class path.
HDFS-9589: Block files which have been hardlinked should be duplicated before the DataNode appends to the them.
HDFS-9600: Do not check replication if the block is under construction.
HDFS-9601: NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block.
HDFS-9605: Add links to failed volumes to explorer.html in HDFS Web UI.
HDFS-9608: Disk IO imbalance in HDFS with heterogeneous storages.
HDFS-9612: DistCp worker threads are not terminated after jobs are done.
HDFS-9618: Fix mismatch between log level and guard in BlockManager#computeRecoveryWorkForBlocks.
HDFS-9619: SimulatedFSDataset sometimes can not find blockpool for the correct namenode.
HDFS-9623: Update example configuration of block state change log in log4j.properties.
HDFS-9624: DataNode start slowly due to the initial DU command operations.
HDFS-9625: set replication for empty file failed when set storage policy.
HDFS-9626: TestBlockReplacement#testBlockReplacement fails occasionally.
HDFS-9629: Update the footer of Web UI to show year 2016.
HDFS-9630: DistCp minor refactoring and clean up.
HDFS-9634: webhdfs client side exceptions don't provide enough details.
HDFS-9638: to branch-2.7.
HDFS-9644: Update encryption documentation to reflect nested EZs.
HDFS-9655: NN should start JVM pause monitor before loading fsimage.
HDFS-9661: Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage moveBlockAcrossStorage and createRbw.
HDFS-9669: TcpPeerServer should respect ipc.server.listen.queue.size.
HDFS-9670: DistCp throws NPE when source is root.
HDFS-9672: o.a.h.hdfs.TestLeaseRecovery2 fails intermittently.
HDFS-9688: Test the effect of nested encryption zones in HDFS downgrade.
HDFS-9696: Garbage snapshot records linger forever.
HDFS-9700: BUG-60868. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets for DataTransferProtocol.
HDFS-9701: DN may deadlock when hot-swapping under load.
HDFS-9706: Log more details in debug logs in BlockReceiver's constructor.
HDFS-9710: DN can be configured to send block receipt IBRs in batches.
HDFS-9711: Integrate CSRF prevention filter in WebHDFS.
HDFS-9713: DataXceiver#copyBlock should return if block is pinned.
HDFS-9715: Check storage ID uniqueness on datanode startup.
HDFS-9721: Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference.
HDFS-9724: Degraded performance in WebHDFS listing as it does not reuse ObjectMapper.
HDFS-9726: Refactor IBR code to a new class. (szetszwo).
HDFS-9730: Storage ID update does not happen when there is a layout change.
HDFS-9739: DatanodeStorage.isValidStorageId() is broken.
HDFS-9740: Use a reasonable limit in DFSTestUtil.waitForMetric().
HDFS-9743: Fix TestLazyPersistFiles#testFallbackToDiskFull in branch-2.7.
HDFS-9748: Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice.
HDFS-9752: Permanent write failures may happen to slow writers during datanode rolling upgrades.
HDFS-9760: WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler.
HDFS-9764: DistCp doesn't print value for several arguments including -numListstatusThreads.
HDFS-9765: TestBlockScanner#testVolumeIteratorWithCaching fails intermittently.
HDFS-9766: TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently.
HDFS-9768: Reuse ObjectMapper instance in HDFS to improve the performance.
HDFS-9772: TestBlockReplacement#testThrottler doesn't work as expected.
HDFS-9777: Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh).
HDFS-9779: . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value.
HDFS-9781: FsDatasetImpl#getBlockReports can occasionally throw NullPointerException.
HDFS-9784: Example usage is not correct in Transparent Encryption document.
HDFS-9790: HDFS Balancer should exit with a proper message if upgrade is not finalized.
HDFS-9797: Log Standby exceptions thrown by RequestHedgingProxyProvider at DEBUG Level.
HDFS-9799: Reimplement getCurrentTrashDir to remove incompatibility.
HDFS-9801: ReconfigurableBase should update the cached configuration.
HDFS-9812: Streamer threads leak if failure happens when closing DFSOutputStream.
HDFS-9839: Reduce verbosity of processReport logging.
HDFS-9842: dfs.datanode.balance.bandwidthPerSec should accept friendly size units.
HDFS-9844: Correct path creation in getTrashRoot to handle root dir.
HDFS-9851: NameNode throws NPE when setPermission is called on a path that does not exist.
HDFS-9854: Log cipher suite negotiation more verbosely.
HDFS-9864: Correct reference for RENEWDELEGATIONTOKEN and CANCELDELEGATIONTOKEN in webhdfs doc.
HDFS-9865: TestBlockReplacement fails intermittently in trunk.
HDFS-9871: "Bytes Being Moved" -ve(-1 B) when cluster was already balanced.
HDFS-9874: Long living DataXceiver threads cause volume shutdown to block.
HDFS-9880: TestDatanodeRegistration fails occasionally.
HDFS-9881: DistributedFileSystem#getTrashRoot returns incorrect path for encryption zones.
HDFS-9882: Add heartbeatsTotal in Datanode metrics.
HDFS-9902: Support different values of dfs.datanode.du.reserved per storage type.
HDFS-9904: testCheckpointCancellationDuringUpload occasionally fails.
HDFS-9905: WebHdfsFileSystem#runWithRetry should display original stack trace on error.
HDFS-9906: Remove spammy log spew when a datanode is restarted.
HDFS-9917: IBR accumulate more objects when SNN was down for sometime.
HDFS-9926: MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils.
HDFS-9941: Do not log StandbyException on NN, other minor logging fixes.
HDFS-9947: Block#toString should not output information from derived classes.
HDFS-9949: Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared.
HDFS-9958: BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages.
HDFS-10178: Permanent write failures can happen if pipeline recoveries occur for the first packet.
HDFS-10182: Hedged read might overwrite user's buf.
HDFS-10186: DirectoryScanner: Improve logs by adding full path of both actual and expected block directories. .
HDFS-10189: PacketResponder#toString should include the downstreams for PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE.
HDFS-10199: Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing.
HDFS-10207: Support enable Hadoop IPC backoff without namenode restart.
HDFS-10209: Support enable caller context in HDFS namenode audit log without restart namenode.
HDFS-10216: Distcp -diff throws exception when handling relative path.
HDFS-10217: show 'blockScheduled' tooltip in datanodes table.
HDFS-10223: peerFromSocketAndKey performs SASL exchange before setting connection timeouts.
HDFS-10224: Implement asynchronous rename for DistributedFileSystem.
HDFS-10225: DataNode hot swap drives should disallow storage type changes.
HDFS-10228: TestHDFSCLI fails.
HDFS-10235: Last contact for Live Nodes should be relative time.
HDFS-10239: Fsshell mv fails if port usage doesn't match in src and destination paths.
HDFS-10242: Cannot create space quota of zero.
HDFS-10245: Fix the findbugs warnings in branch-2.7.
HDFS-10253: Fix TestRefreshCallQueue failure.
HDFS-10261: TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports.
HDFS-10264: Logging improvements in FSImageFormatProtobuf.Saver.
HDFS-10270: TestJMXGet:testNameNode() fails.
HDFS-10271: Extra bytes are getting released from reservedSpace for append.
HDFS-10275: TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly.
HDFS-10277: PositionedReadable test testReadFullyZeroByteFile failing in HDFS.
HDFS-10279: Improve validation of the configured number of tolerated failed volumes.
HDFS-10281: TestPendingCorruptDnMessages fails intermittently.
HDFS-10287: MiniDFSCluster should implement AutoCloseable.
HDFS-10291: TestShortCircuitLocalRead failing.
HDFS-10301: BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order.
HDFS-10301: Remove FBR tracking state to fix false zombie storage detection for interleaving block reports.
HDFS-10309: Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga) (Amit Anand).
HDFS-10312: Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.
HDFS-10313: Distcp need to enforce the order of snapshot names passed to -diff.
HDFS-10319: Balancer should not try to pair storages with different types.
HDFS-10320: Rack failures may result in NN terminate.
HDFS-10324: Trash directory in an encryption zone should be pre-created with correct permissions.
HDFS-10329: Bad initialisation of StringBuffer in RequestHedgingProxyProvider.
HDFS-10330: Add Corrupt Blocks Information in Metasave output.
HDFS-10335: Mover$Processor#chooseTarget() always chooses the first matching target storage group. .
HDFS-10341: Add a metric to expose the timeout number of pending replication blocks.
HDFS-10342: BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt.
HDFS-10343: BlockManager#createLocatedBlocks may return blocks on failed storages.
HDFS-10344: DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash.
HDFS-10346: Implement asynchronous setPermission/setOwner for DistributedFileSystem.
HDFS-10347: Namenode report bad block method doesn't log the bad block or datanode.
HDFS-10360: DataNode may format directory and lose blocks if current/VERSION is missing.
HDFS-10367: TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception.
HDFS-10372: Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume.
HDFS-10377: CacheReplicationMonitor shutdown log message should use INFO level.
HDFS-10390: Implement asynchronous setAcl/getAclStatus for DistributedFileSystem.
HDFS-10396: Using -diff option with DistCp may get "Comparison method violates its general contract" exception.
HDFS-10397: Distcp should ignore -delete option if -diff option is provided instead of exiting.
HDFS-10410: RedundantEditLogInputStream.LOG is set to wrong class.
HDFS-10415: TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called.
HDFS-10417: Improve error message from checkBlockLocalPathAccess.
HDFS-10423: Increase default value of httpfs maxHttpHeaderSize.
HDFS-10424: DatanodeLifelineProtocol not able to use under security cluster.
HDFS-10431: Refactor and speedup TestAsyncDFSRename.
HDFS-10437: ReconfigurationProtocol not covered by HDFSPolicyProvider.
HDFS-10438: When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.
HDFS-10440: Improve DataNode web UI.
HDFS-10448: CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1.
HDFS-10455: Logging the username when deny the setOwner operation.
HDFS-10457: DataNode should not auto-format block pool directory if VERSION is missing.
HDFS-10458: getFileEncryptionInfo should return quickly for non-encrypted cluster.
HDFS-10468: HDFS read ends up ignoring an interrupt.
HDFS-10469: Add number of active xceivers to datanode metrics.
HDFS-10471: DFSAdmin#SetQuotaCommand's help msg is not correct.
HDFS-10474: hftp copy fails when file name with Chinese+special char in branch-2.
HDFS-10481: HTTPFS server should correctly impersonate as end user to open file.
HDFS-10485: Fix findbugs warning in FSEditLog.java.
HDFS-10488: Update WebHDFS documentation regarding CREATE and MKDIR default permissions.
HDFS-10493: Add links to datanode web UI in namenode datanodes page.
HDFS-10499: TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails Intermittently.
HDFS-10508: DFSInputStream should set thread's interrupt status after catching InterruptException from sleep.
HDFS-10512: VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks.
HDFS-10525: Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap.
HDFS-10544: Balancer doesn't work with IPFailoverProxyProvider.
HDFS-10556: DistCpOptions should be validated automatically.
HDFS-10569: A bug causes OutOfIndex error in BlockListAsLongs.
HDFS-10579: HDFS web interfaces lack configs for X-FRAME-OPTIONS protection.
HDFS-10589: Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params.
HDFS-10609: Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications.
HDFS-10617: PendingReconstructionBlocks.size() should be synchronized.
HDFS-10623: Remove unused import of httpclient.HttpConnection from TestWebHdfsTokens.
HDFS-10625: VolumeScanner to report why a block is found bad.
HDFS-10627: Volume Scanner marks a block as "suspect" even if the exception is network-related.
HDFS-10642: TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently.
HDFS-10643: [Unit Test] Namenode should use loginUser(hdfs) to generateEncryptedKey.
HDFS-10643: HDFS namenode should always use service user (hdfs) to generateEncryptedKey.
HDFS-10653: Optimize conversion from path string to components.
HDFS-10655: Fix path related byte array conversion bugs.
HDFS-10660: Expose storage policy apis via HDFSAdmin interface.
HDFS-10684: WebHDFS DataNode calls fail without parameter createparent.
HDFS-10688: BPServiceActor may run into a tight loop for sending block report when hitting IOException.
HDFS-10703: HA NameNode Web UI should show last checkpoint time.
HDFS-10710: In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.
HDFS-10713: Throttle FsNameSystem lock warnings.
HDFS-10715: NPE when applying AvailableSpaceBlockPlacementPolicy.
HDFS-10716: In Balancer, the target task should be removed when its size < 0. .
HDFS-10735: Distcp using webhdfs on secure HA clusters fails with StandbyException.
HDFS-10752: Several log refactoring/improvement suggestion in HDFS.
HDFS-10757: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.
HDFS-10763: Open files can leak permanently due to inconsistent lease update.
HDFS-10763: Open files can leak permanently due to inconsistent lease update.
HDFS-10764: Fix INodeFile#getBlocks to not return null.
HDFS-10793: Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184.
HDFS-10798: Make the threshold of reporting FSNamesystem lock contention configurable.
HDFS-10817: Add Logging for Long-held NN Read Locks.
HDFS-10822: Log DataNodes in the write pipeline. John Zhuge via Lei Xu.
HDFS-10892: Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'.
HDFS-10893: Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test.
HDFS-10906: Add unit tests for Trash with HDFS encryption zones.
HDFS-10913: Introduce fault injectors to simulate slow mirrors.
HDFS-10917: Collect peer performance statistics on DataNode.
HDFS-10922: Adding additional unit tests for Trash (II).
HDFS-10930: Refactor: Wrap Datanode IO related operations.
HDFS-10934: TestDFSShell#testStat fails intermittently.
HDFS-10956: Remove rename/delete performance penalty when not using snapshots.
HDFS-10959: Update DataNode to use DatasetVolumeChecker.
HDFS-10962: TestRequestHedgingProxyProvider is flaky.
HDFS-10963: Reduce log level when network topology cannot find enough datanodes.
HDFS-10965: Add unit test for HDFS command 'dfsadmin -printTopology'.
HDFS-10966: Enhance Dispatcher logic on deciding when to give up a source DataNode.
HDFS-10972: Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'.
HDFS-10984: Expose nntop output as metrics.
HDFS-10986: DFSAdmin should log detailed error message if any.
HDFS-10987: Make Decommission less expensive when lot of blocks present.
HDFS-10998: Add unit tests for HDFS command 'dfsadmin -fetchImage' in HA.
HDFS-11003: Expose XmitsInProgress through DataNodeMXBean.
HDFS-11008: Change unit test for testing parsing "-source" parameter in Balancer CLI.
HDFS-11011: Add unit tests for HDFS command 'dfsadmin -set/clrSpaceQuota'.
HDFS-11015: Enforce timeout in balancer.
HDFS-11018: Incorrect check and message in FsDatasetImpl#invalidate.
HDFS-11030: TestDataNodeVolumeFailure#testVolumeFailure is flaky (though passing).
HDFS-11031: Add additional unit test for DataNode startup behavior when volumes fail.
HDFS-11047: Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode.
HDFS-11054: Suppress verbose log message in BlockPlacementPolicyDefault.
HDFS-11056: Addendum fix for self-healing branch.
HDFS-11056: Concurrent append and read operations lead to checksum error.
HDFS-11065: Add space quota tests for heterogenous storages.
HDFS-11069: Tighten the authorization of datanode RPC.
HDFS-11076: Add unit test for extended Acls.
HDFS-11083: Add unit test for DFSAdmin -report command.
HDFS-11085: Add unit test for NameNode failing to start when name dir is unwritable.
HDFS-11087: NamenodeFsck should check if the output writer is still writable.
HDFS-11112: Journal Nodes should refuse to format non-empty directories.
HDFS-11114: Support for running async disk checks in DataNode.
HDFS-11119: Support for parallel checking of StorageLocations on DataNode startup.
HDFS-11122: TestDFSAdmin#testReportCommand fails due to timed out.
HDFS-11128: CreateEditsLog throws NullPointerException.
HDFS-11148: Update DataNode to use StorageLocationChecker at startup.
HDFS-11149: Fix build break.
HDFS-11149: Support for parallel checking of FsVolumes.
HDFS-11156: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API.
HDFS-11160: VolumeScanner reports write-in-progress replicas as corrupt incorrectly.
HDFS-11180: Intermittent deadlock in NameNode when failover happens.
HDFS-11182: Addendum fix for Fenton to fix a unit test.
HDFS-11182: Update DataNode to use DatasetVolumeChecker.
HDFS-11194: Maintain aggregated peer performance metrics on NameNode.
HDFS-11195: Return error when appending files by webhdfs rest api fails.
HDFS-11197: Listing encryption zones fails when deleting a EZ that is on a snapshotted directory.
HDFS-11209: SNN can't checkpoint when rolling upgrade is not finalized.
HDFS-11224: Lifeline message should be ignored for dead nodes.
HDFS-11229: HDFS-11056 failed to close meta file.
HDFS-11240: Remove snapshot version of SDK dependency from Azure Data Lake Store File System.
HDFS-11251: ConcurrentModificationException during DataNode#refreshVolumes.
HDFS-11253: FileInputStream leak on failure path in BlockSender.
HDFS-11258: File mtime change could not save to editlog.
HDFS-11263: ClassCastException when we use Bzipcodec for Fsimage compression.
HDFS-11267: Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage.
HDFS-11274: Datanode should only check the failed volume upon IO errors.
HDFS-11279: Cleanup unused DataNode#checkDiskErrorAsync().
HDFS-11280: Allow WebHDFS to reuse HTTP connections to NN.
HDFS-11282: Document the missing metrics of DataNode Volume IO operations.
HDFS-11292: log lastWrittenTxId etc info in logSyncAll.
HDFS-11299: Support multiple Datanode File IO hooks.
HDFS-11307: The rpc to portmap service for NFS has hardcoded timeout.
HDFS-11337: (HDFS-10958 backport). Add instrumentation hooks around Datanode disk IO.
HDFS-11339: Support File IO sampling for Datanode IO profiling hooks.
HDFS-11369: Change exception message in StorageLocationChecker.
HDFS-11379: DFSInputStream may infinite loop requesting block locations.
HDFS-11387: Socket reuse address option is not honored in PrivilegedNfsGatewayStarter.
HDFS-11391: Numeric usernames do no work with WebHDFS FS write access.
HDFS-11395: RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode.
HDFS-11403: Zookeper ACLs on NN HA enabled clusters should be handled consistently.
HDFS-11445: FSCK shows overall health stauts as corrupt even one replica is corrupt.
HDFS-11461: DataNode Disk Outlier Detection.
HDFS-11462: Fix occasional BindException in TestNameNodeMetricsLogger.
HDFS-11476: Fix NPE in FsDatasetImpl#checkAndUpdate.
HDFS-11479: Socket re-use address option should be used in SimpleUdpServer.
HDFS-11499: Decommissioning stuck because of failing recovery.
HDFS-11502: Datanode UI should display hostname based on JMX bean instead of window.location.hostname.
HDFS-11508: Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket is in TIME_WAIT state.
HDFS-11517: Expose slow disks via DataNode JMX.
HDFS-11533: reuseAddress option should be used for child channels in Portmap and SimpleTcpServer.
HDFS-11534: Add counters for number of blocks in pending IBR.
HDFS-11545: Propagate DataNode's slow disks info to the NameNode via Heartbeat.
HDFS-11547: Add logs for slow BlockReceiver while writing data to disk.
HDFS-11551: Handle SlowDiskReport from DataNode at the NameNode.
HDFS-11560: Expose slow disks via NameNode JMX.
HDFS-11570: Unit test for NameNodeStatusMXBean.
HDFS-11603: Improve slow mirror/disk warnings in BlockReceiver.
HDFS-11608: HDFS write crashed with block size greater than 2 GB.
HDFS-11634: Optimize BlockIterator when interating starts in the middle.
HDFS-11722: Change Datanode file IO profiling sampling to percentage.
HDFS-11842: TestDataNodeOutlierDetectionViaMetrics UT fails intermittently. Controbuted by Hanisha Koneru.
HDFS-12139: HTTPFS liststatus returns incorrect pathSuffix for path of file'.
HDFS-12158: Secondary Namenode's web interface lack configs for X-FRAME- OPTIONS protection.'.
HDFS-14676: Wrong default value for fs.df.interval.'.
MAPREDUCE-5485: Allow repeating job commit by extending OutputCommitter API.
MAPREDUCE-5621: mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time.
MAPREDUCE-6121: JobResourceUpdater#compareFs() doesn't handle HA namespaces.
MAPREDUCE-6165: [JDK8] TestCombineFileInputFormat failed on JDK8.
MAPREDUCE-6197: Cache MapOutputLocations in ShuffleHandler.
MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.
MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.
MAPREDUCE-6246: DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2.
MAPREDUCE-6273: HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state.
MAPREDUCE-6279: AM should explicity exit JVM after all services have stopped.
MAPREDUCE-6302:.
MAPREDUCE-6304: Specifying node labels when submitting MR jobs.
MAPREDUCE-6338: MR AppMaster does not honor ephemeral port range.
MAPREDUCE-6350: JobHistory doesn't support fully-functional search.
MAPREDUCE-6404: Allow AM to specify a port range for starting its webapp.
MAPREDUCE-6436: JobHistory cache issue.
MAPREDUCE-6436: JobHistory cache issue.
MAPREDUCE-6460: TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails.
MAPREDUCE-6478: Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob.
MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.
MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.
MAPREDUCE-6514: Job hangs as ask is not updated after ramping down of all reducers.
MAPREDUCE-6543: Migrate MR Client test cases part 2.
MAPREDUCE-6549: multibyte delimiters with LineRecordReader cause duplicate records.
MAPREDUCE-6565: Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.
MAPREDUCE-6566: Add retry support to mapreduce CLI tool.
MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
MAPREDUCE-6579: TestNetworkedJob.testNetworkedJob failed due to YARN-3946.
MAPREDUCE-6593: TestJobHistoryEventHandler.testTimelineEventHandling fails because of NPE.
MAPREDUCE-6616: Fail to create jobhistory file if there are some multibyte characters in the job name.
MAPREDUCE-6618: YarnClientProtocolProvider leaking the YarnClient thread.(xgong).
MAPREDUCE-6621: Memory Leak in JobClient#submitJobInternal().
MAPREDUCE-6625: TestCLI#testGetJob fails occasionally (haibochen via rkanter).
MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.
MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.
MAPREDUCE-6639: Process hangs in LocatedFileStatusFetcher if FileSystem.get throws.
MAPREDUCE-6672: TestTeraSort fails on Windows.
MAPREDUCE-6684: High contention on scanning of user directory under immediate_done in Job History Server.
MAPREDUCE-6689: MapReduce job can infinitely increase number of reducer resource requests.
MAPREDUCE-6693: ArrayIndexOutOfBoundsException occurs when the length of the job name is equal to mapreduce.jobhistory.jobname.limit.
MAPREDUCE-6701: application master log can not be available when clicking jobhistory's am logs link.
MAPREDUCE-6702: Fixed failures in TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv.
MAPREDUCE-6711: After enabling preemption on cluster, jobs failed without making any attempts made after being preempted.
MAPREDUCE-6724: Single shuffle to memory must not exceed Integer#MAX_VALUE.
MAPREDUCE-6733: MapReduce JerseyTest tests failing with java.net.BindException: Address already in use.
MAPREDUCE-6762: ControlledJob#toString failed with NPE when job status is not successfully updated.
MAPREDUCE-6789: Fix TestAMWebApp failure. .
MAPREDUCE-6792: Allow user's full principal name as owner of MapReduce staging directory in JobSubmissionFiles#JobStagingDir().
MAPREDUCE-6797: Job history server scans can become blocked on a single, slow entry.
MAPREDUCE-6852: Job#updateStatus() failed with NPE due to race condition.txt.
MAPREDUCE-6895: Job end notification not send due to YarnRuntimeException.
MAPREDUCE-6897: Add Unit Test to ensure Job end notification gets sent even when appMaster stop gets YarnRuntimeException.
YARN-857: Localization failures should be available in container diagnostics.
YARN-998: Keep NM resource updated through dynamic resource config for RM/NM restart.
YARN-1376: NM need to notify the log aggregation status to RM through heartbeat.
YARN-1402: Update related Web UI and CLI with exposing client API to check log aggregation status.
YARN-1462: AHS API and other AHS changes to handle tags for completed MR jobs. .
YARN-1556: NPE getting application report with a null appId.
YARN-1728: Workaround guice3x-undecoded pathInfo in YARN WebApp.
YARN-1880: Cleanup TestApplicationClientProtocolOnHA.
YARN-1993: Cross-site scripting vulnerability in TextView.java.
YARN-2019: Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore.
YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.
YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.
YARN-2194: Fix bug causing CGroups functionality to fail on RHEL7.
YARN-2392: Add more diags about app retry limits on AM failures.
YARN-2495: Allow admin specify labels from each NM (Distributed configuration for node label).
YARN-2498: Respect labels in preemption policy of capacity scheduler for inter-queue preemption.
YARN-2513: Host framework UIs in YARN for use with the ATS.
YARN-2513: Host framework UIs in YARN for use with the ATS Note: Patch is in progress.
YARN-2571: RM to support YARN registry.
YARN-2619: Added NodeManager support for disk io isolation through cgroups.
YARN-2666: TestFairScheduler.testContinuousScheduling fails Intermittently.
YARN-2696: Queue sorting in CapacityScheduler should consider node label.
YARN-2740: Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled.
YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.
YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.
YARN-2868: FairScheduler: Metric for latency to allocate first container for an application.
YARN-2901: Add errors and warning metrics page to RM, NM web UI.
YARN-3021: YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp.
YARN-3026: Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp.
YARN-3102: Decommisioned Nodes not listed in Web UI.
YARN-3110: Few issues in ApplicationHistory web ui.
YARN-3136: Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers.
YARN-3148: Allow CORS related headers to passthrough in WebAppProxyServlet.
YARN-3215: Respect labels in CapacityScheduler when computing headroom.
YARN-3216: Max-AM-Resource-Percentage should respect node labels.
YARN-3225: New parameter of CLI for decommissioning node gracefully in RMAdmin CLI.
YARN-3243: CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits.
YARN-3248: Display count of nodes blacklisted by apps in the web UI.
YARN-3266: RMContext#inactiveNodes should have NodeId as map key.
YARN-3269: Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path.
YARN-3293: Track and display capacity scheduler health metrics in web UI.
YARN-3294: Allow dumping of Capacity Scheduler debug logs via web UI for a fixed time period.
YARN-3305: Normalize AM resource request on app submission.
YARN-3318: Create Initial OrderingPolicy Framework and FifoOrderingPolicy.
YARN-3319: Implement a FairOrderingPolicy.
YARN-3326: Support RESTful API for getLabelsToNodes.
YARN-3343: Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate timeout.
YARN-3345: Add non-exclusive node label API.
YARN-3347: Improve YARN log command to get AMContainer logs as well as running containers logs.
YARN-3348: Add a 'yarn top' tool to help understand cluster usage.
YARN-3354: Add node label expression in ContainerTokenIdentifier to support RM recovery.
YARN-3356: Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label.
YARN-3357: Move TestFifoScheduler to FIFO package.
YARN-3360: Add JMX metrics to TimelineDataManager.
YARN-3361: CapacityScheduler side changes to support non-exclusive node labels.
YARN-3362: Add node label usage in RM CapacityScheduler web UI.
YARN-3365: Enhanced NodeManager to support using the 'tc' tool via container-executor for outbound network traffic control.
YARN-3366: Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers.
YARN-3383: AdminService should use "warn" instead of "info" to log exception when operation fails.
YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.
YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.
YARN-3394: Enrich WebApplication proxy documentation.
YARN-3397: yarn rmadmin should skip -failover.
YARN-3404: Display queue name on application page.
YARN-3406: Display count of running containers in the RM's Web UI.
YARN-3410: YARN admin should be able to remove individual application records from RMStateStore.
YARN-3413: Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime.
YARN-3425: NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed.
YARN-3433: Jersey tests failing with Port in Use -again.
YARN-3435: AM container to be allocated Appattempt AM container shown as null.
YARN-3443: Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM.
YARN-3448: Added a rolling time-to-live LevelDB timeline store implementation.
YARN-3451: Display attempt start time and elapsed time on the web UI.
YARN-3453: Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing.
YARN-3459: Fix failiure of TestLog4jWarningErrorMetricsAppender.
YARN-3463: Integrate OrderingPolicy Framework with CapacityScheduler.
YARN-3467: Expose allocatedMB, allocatedVCores, and runningContainers metrics on running Applications in RM Web UI.
YARN-3480: Remove attempts that are beyond max-attempt limit from state store.
YARN-3494: Expose AM resource limit and usage in CS QueueMetrics.
YARN-3503: Expose disk utilization percentage and bad local and log dir counts in NM metrics.
YARN-3505: Node's Log Aggregation Report with SUCCEED should not cached in RMApps.
YARN-3511: Add errors and warnings page to ATS.
YARN-3517: RM web ui for dumping scheduler logs should be for admins only.
YARN-3521: Support return structured NodeLabel objects in REST API.
YARN-3530: ATS throws exception on trying to filter results without otherinfo.
YARN-3541: Add version info on timeline service / generic history web UI and REST API.
YARN-3543: ApplicationReport should be able to tell whether the Application is AM managed or not.
YARN-3552: RM Web UI shows -1 running containers for completed apps.
YARN-3565: NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String.
YARN-3579: CommonNodeLabelsManager should support NodeLabel instead of string label name when getting node-to-label/label-to-label mappings.
YARN-3580: [JDK8] TestClientRMService.testGetLabelsToNodes fails.
YARN-3581: Deprecate -directlyAccessNodeLabelStore in RMAdminCLI.
YARN-3583: Support of NodeLabel object instead of plain String in YarnClient side.
YARN-3589: RM and AH web UI display DOCTYPE wrongly.
YARN-3591: Resource localization on a bad disk causes subsequent containers failure.
YARN-3593: Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page.
YARN-3600: AM container link is broken -- on a killed application, at least.
YARN-3604: removeApplication in ZKRMStateStore should also disable watch.
YARN-3623: Add a config to indicate the Timeline Service version.
YARN-3623: Addendum: Improve the description for Timeline Service Version configuration in yarn-default.xml.
YARN-3632: Ordering policy should be allowed to reorder an application when demand changes.
YARN-3635: Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework.
YARN-3654: ContainerLogsPage web UI should not have meta-refresh.
YARN-3684: Changed ContainerExecutor's primary lifecycle methods to use a more extensible mechanism of context objects.
YARN-3695: ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.
YARN-3695: ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.
YARN-3700: Made generic history service load a number of latest applications according to the parameter or the configuration.
YARN-3707: RM Web UI queue filter doesn't work.
YARN-3717: Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API. (Naganarasimha G R via wangda).
YARN-3722: Merge multiple TestWebAppUtils into o.a.h.yarn.webapp.util.TestWebAppUtils.
YARN-3727: For better error recovery, check if the directory exists before using it for localization.
YARN-3740: Fixed the typo in the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS.
YARN-3749: We should make a copy of configuration when init MiniYARNCluster with multiple RMs.
YARN-3751: Fixed AppInfo to check if used resources are null.
YARN-3762: FairScheduler: CME on FSParentQueue#getQueueUserAclInfo.
YARN-3766: Fixed the apps table column error of generic history web UI.
YARN-3768: ArrayIndexOutOfBoundsException with empty environment variables.
YARN-3769: Consider user limit when calculating total pending resource for preemption policy in Capacity Scheduler.
YARN-3785: Support for Resource as an argument during submitApp call in MockRM test class.
YARN-3844: Make hadoop-yarn-project Native code -Wall-clean.
YARN-3846: RM Web UI queue filter is not working for sub queue.
YARN-3849: (Too much of preemption activity causing continuos killing of containers across queues) to 2.3-maint.
YARN-3852: Add docker container support to container-executor.
YARN-3853: Add docker container runtime support to LinuxContainterExecutor.
YARN-3873: PendingApplications in LeafQueue should also use OrderingPolicy.
YARN-3877: YarnClientImpl.submitApplication swallows exceptions.
YARN-3885: ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level.
YARN-3894: RM startup should fail for wrong CS xml NodeLabel capacity configuration.
YARN-3896: RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously.
YARN-3920: FairScheduler container reservation on a node should be configurable to limit it to large containers.
YARN-3932: SchedulerApplicationAttempt#getResourceUsageReport and UserInfo should based on total-used-resources.
YARN-3940: Application moveToQueue should check NodeLabel permission.
YARN-3941: Proportional Preemption policy should try to avoid sending duplicate PREEMPT_CONTAINER event to scheduler.
YARN-3946: Update exact reason as to why a submitted app is in ACCEPTED state to app's diagnostic message.
YARN-3948: Display Application Priority in RM Web UI.(Sunil G via rohithsharmaks).
YARN-3961: Expose pending, running and reserved containers of a queue in REST api and yarn top.
YARN-3963: AddNodeLabel on duplicate label addition shows success.
YARN-3966: Fix excessive loggings in CapacityScheduler.
YARN-3971: Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery.
YARN-3978: Configurably turn off the saving of container info in Generic AHS.
YARN-3982: container-executor parsing of container-executor.cfg broken in trunk and branch-2.
YARN-3983: Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic.
YARN-3986: getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface.
YARN-3987: am container complete msg ack to NM once RM receive it.
YARN-4000: RM crashes with NPE if leaf queue becomes parent queue during restart.
YARN-4004: container-executor should print output of docker logs if the docker container exits with non-0 exit status.
YARN-4007: Add support for different network setups when launching the docker container.
YARN-4017: container-executor overuses PATH_MAX.
YARN-4023: Publish Application Priority to TimelineServer.
YARN-4026: Refactored ContainerAllocator to accept a list of priorites rather than a single priority.
YARN-4082: Container shouldn't be killed when node's label updated.
YARN-4087: Followup fixes after YARN-2019 regarding RM behavior when state- store error occurs.
YARN-4092: Fixed UI redirection to print useful messages when both RMs are in standby mode.
YARN-4101: RM should print alert messages if ZooKeeper and Resourcemanager gets connection issue.
YARN-4108: CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request.
YARN-4109: Exception on RM scheduler page loading with labels.
YARN-4113: RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER.
YARN-4132: Separate configs for nodemanager to resourcemanager connection timeout and retries.
YARN-4140: RM container allocation delayed incase of app submitted to Nodelabel partition.
YARN-4149: yarn logs -am should provide an option to fetch all the log files.
YARN-4155: TestLogAggregationService.testLogAggregationServiceWithInterval failing.
YARN-4162: CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API.
YARN-4168: Fixed a failing test TestLogAggregationService.testLocalFileDeletionOnDiskFull.
YARN-4204: ConcurrentModificationException in FairSchedulerQueueInfo.
YARN-4215: RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request.
YARN-4225: Add preemption status to yarn queue -status for capacity scheduler.
YARN-4233: YARN Timeline Service plugin: ATS v1.5.
YARN-4243: Add retry on establishing ZooKeeper conenction in EmbeddedElectorService#serviceInit.
YARN-4245: Generalize config file handling in container- executor.
YARN-4252: Log container-executor invocation details when exit code is non- zero.
YARN-4253: Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor.
YARN-4255: container-executor does not clean up docker operation command files.
YARN-4258: Add support for controlling capabilities for docker containers.
YARN-4262: Allow whitelisted users to run privileged docker containers.
YARN-4267: Add additional logging to container launch implementations in container-executor.
YARN-4270: Limit application resource reservation on nodes for non- node/rack specific requests.
YARN-4285: Display resource usage as percentage of queue and cluster in the RM UI.
YARN-4287: Capacity Scheduler: Rack Locality improvement.
YARN-4288: Fixed RMProxy to retry on IOException from local host.
YARN-4290: Add -showDetails option to YARN Nodes CLI to print all nodes reports information.
YARN-4304: AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics.
YARN-4309: Add container launch related debug information to container logs when a container fails.
YARN-4313: Race condition in MiniMRYarnCluster when getting history server address.
YARN-4315: NaN in Queue percentage for cluster apps page.
YARN-4325: Nodemanager log handlers fail to send finished/failed events in some cases.
YARN-4329: [YARN-5437] Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler.
YARN-4345: yarn rmadmin -updateNodeResource doesn't work.
YARN-4347: Resource manager fails with Null pointer exception.
YARN-4349: YARN_APPLICATION call to ATS does not have YARN_APPLICATION_CALLER_CONTEXT.
YARN-4365: FileSystemNodeLabelStore should check for root dir existence on startup.
YARN-4384: updateNodeResource CLI should not accept negative values for resource.
YARN-4390: Do surgical preemption based on reserved container in CapacityScheduler.
YARN-4392: ApplicationCreatedEvent event time resets after RM restart/failover.
YARN-4393: TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently.
YARN-4403: (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period.
YARN-4405: Support node label store in non-appendable file system.
YARN-4411: RMAppAttemptImpl#createApplicationAttemptReport throws IllegalArgumentException.
YARN-4414: Nodemanager connection errors are retried at multiple levels.
YARN-4414: Nodemanager connection errors are retried at multiple levels.
YARN-4416: Deadlock due to synchronised get Methods in AbstractCSQueue.
YARN-4417: Make RM and Timeline-server REST APIs more consistent.
YARN-4418: AM Resource Limit per partition can be updated to ResourceUsage as well.
YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.
YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.
YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.
YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.
YARN-4439: Clarify NMContainerStatus#toString method.
YARN-4454: NM to nodelabel mapping going wrong after RM restart.
YARN-4465: SchedulerUtils#validateRequest for Label check should happen only when nodelabel enabled.
YARN-4477: FairScheduler: Handle condition which can result in an infinite loop in attemptScheduling.
YARN-4479: Change CS LeafQueue pendingOrderingPolicy to hornor recovered apps.
YARN-4484: Available Resource calculation for a queue is not correct when used with labels.
YARN-4496: Improve HA ResourceManager Failover detection on the client.
YARN-4497: RM might fail to restart when recovering apps whose attempts are missing.
YARN-4502: Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda).
YARN-4522: Queue acl can be checked at app submission.
YARN-4535: Fix checkstyle error in CapacityScheduler.java.
YARN-4546: YARN-4546. ResourceManager crash due to scheduling opportunity overflow.
YARN-4546: YARN-4546. ResourceManager crash due to scheduling opportunity overflow.
YARN-4553: Add cgroups support for docker containers.
YARN-4556: TestFifoScheduler.testResourceOverCommit fails.
YARN-4557: Fix improper Queues sorting in PartitionedQueueComparator when accessible-node-labels=*.
YARN-4565:.
YARN-4571: Make app id/name available to the yarn authorizer provider for better auditing.
YARN-4573: Fix test failure in TestRMAppTransitions#testAppRunningKill and testAppKilledKilled.
YARN-4577: Enable aux services to have their own custom classpath/jar file.
YARN-4578: Directories that are mounted in docker containers need to be more restrictive/container-specific.
YARN-4582: Label-related invalid resource request exception should be able to properly handled by application.
YARN-4584: RM startup failure when AM attempts greater than max-attempts.
YARN-4591: YARN Web UIs should provide a robots.txt. Junping Du.
YARN-4595: Add support for configurable read-only mounts when launching Docker containers.
YARN-4596: SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities.
YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.
YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.
YARN-4607: Pagination support for AppAttempt page TotalOutstandingResource Requests table.
YARN-4610: Reservations continue looking for one app causes other apps to starve.
YARN-4617: LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy.
YARN-4623: TestSystemMetricsPublisher#testPublishAppAttemptMetricsForUnmanagedAM fails with NPE on branch-2.7.
YARN-4624: NPE in PartitionQueueCapacitiesInfo while accessing Scheduler UI.
YARN-4625: Make ApplicationSubmissionContext and ApplicationSubmissionContextInfo more consistent.
YARN-4633: TestRMRestart.testRMRestartAfterPreemption fails intermittently.
YARN-4634: Scheduler UI/Metrics need to consider cases like non-queue label mappings.
YARN-4643: Container recovery is broken with delegating container runtime.
YARN-4667: RM Admin CLI for refreshNodesResources throws NPE when nothing is configured.
YARN-4680: Fix TimerTasks leak in ATS V1.5 Writer.
YARN-4696: TimelineClient to add flush operation for deterministic writes (including testing) and Changes to EntityGroupFSTimelineStore for testability.
YARN-4697: NM aggregation thread pool is not bound by limits.
YARN-4699: Scheduler UI and REST o/p is not in sync when -replaceLabelsOnNode is used to change label of a node.
YARN-4709: NMWebServices produces incorrect JSON for containers.
YARN-4709: NMWebServices produces incorrect JSON for containers.
YARN-4723: NodesListManager$UnknownNodeId ClassCastException.
YARN-4737: Add CSRF filter support in YARN.
YARN-4737: Add CSRF filter support in YARN.
YARN-4744: Too many signal to container failure in case of LCE.
YARN-4746: yarn web services should convert parse failures of appId, appAttemptId and containerId to 400.
YARN-4748: ApplicationHistoryManagerOnTimelineStore should not swallow exceptions on generateApplicationReport.
YARN-4759: Fix signal handling for docker containers.
YARN-4762: Fixed CgroupHandler's creation and usage to avoid NodeManagers crashing when LinuxContainerExecutor is enabled.
YARN-4764: Application submission fails when submitted queue is not available in scheduler xml.
YARN-4767: Network issues can cause persistent RM UI outage.
YARN-4769: Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI.
YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.
YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.
YARN-4811: Generate histograms in ContainerMetrics for actual container resource usage.
YARN-4814: ATS 1.5 timelineclient impl call flush after every event write.
YARN-4815: ATS 1.5 timelineclinet impl try to create attempt directory for every event call.
YARN-4817: TimelineClient ATSv1.5 logging is very noisy.
YARN-4820: ResourceManager web redirects in HA mode drops query parameters.
YARN-4822: Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates.
YARN-4832: NM side resource value should get updated if change applied in RM side.
YARN-4833: For Queue AccessControlException client retries multiple times on both RM.
YARN-4842: Fixed "yarn logs" command to guess (and thus not require) the appOwner argument when viewing another user's logs.
YARN-4844: Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource.
YARN-4846: Fix random failures for TestCapacitySchedulerPreemption #testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers.
YARN-4851: Metric improvements for ATS v1.5 storage components. Li Lu via junping_du.
YARN-4863: AHS Security login should be in serviceInit() instead of serviceStart().
YARN-4865: Track Reserved resources in ResourceUsage and QueueCapacities.
YARN-4890: Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate.
YARN-4893: Fix intermittent failures in TestRMAdminService.
YARN-4905: Improved "yarn logs" command-line to optionally show log metadata also.
YARN-4906: Capture container start/finish time in container metrics.
YARN-4909: Fix intermittent failures of TestRMWebServices And TestRMWithCSRFFilter.
YARN-4913: Yarn logs should take a -out option to write to a directory.
YARN-4916: TestNMProxy.tesNMProxyRPCRetry fails.
YARN-4920: ATS/NM should support a link to dowload/get the logs in text format.
YARN-4927: TestRMHA#testTransitionedToActiveRefreshFail fails with FairScheduler.
YARN-4928: Some yarn.server.timeline.* tests fail on Windows attempting to use a test root path containing a colon.
YARN-4934: Reserved Resource for QueueMetrics needs to be handled correctly in few cases.
YARN-4955: Add retry for SocketTimeoutException in TimelineClient.
YARN-4956: findbug issue on LevelDBCacheTimelineStore.
YARN-4966: Improve yarn logs to fetch container logs without specifying nodeId.
YARN-4983: JVM and UGI metrics disappear after RM transitioned to standby mode.
YARN-4984: LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak.
YARN-4987: Read cache concurrency issue between read and evict in EntityGroupFS timeline store.
YARN-4990: Re-direction of a particular log file within in a container in NM UI does not redirect properly.
YARN-5002: getApplicationReport call may raise NPE for removed queues.
YARN-5023: TestAMRestart#testShouldNotCountFailureToMaxAttemptRetry random failure.
YARN-5029: RM needs to send update event with YarnApplicationState as Running to ATS/AHS.
YARN-5048: DelegationTokenRenewer#skipTokenRenewal may throw NPE.
YARN-5069: TestFifoScheduler.testResourceOverCommit race condition.
YARN-5076: YARN web interfaces lack XFS (Cross-Frame Script) protection.
YARN-5080: Cannot obtain logs using YARN CLI -am for either KILLED or RUNNING AM.
YARN-5083: YARN CLI for AM logs does not give any error message if entered invalid am value.
YARN-5088: Improve "yarn log" command-line to read the last K bytes for the log files.
YARN-5089: Improve "yarn log" command-line "logFiles" option to support regex.
YARN-5098: Fixed ResourceManager's DelegationTokenRenewer to replace expiring system-tokens if RM stops and only restarts after a long time.
YARN-5100: The YarnApplicationState is always running in ATS even application is finished.
YARN-5103: With NM recovery enabled, restarting NM multiple times results in AM restart.
YARN-5112: Excessive log warnings for directory permission issue on NM recovery.
YARN-5115: Avoid setting CONTENT-DISPOSITION header in the container-logs web-service.
YARN-5121: fix some container-executor portability issues.
YARN-5122: "yarn logs" for running containers should print an explicit footer saying that the log may be incomplete.
YARN-5131: Distributed shell AM fails when extra container arrives during finishing. Wangda Tan via junping_du.
YARN-5141: Get Container logs for the Running application from Yarn Logs CommandLine.
YARN-5190: Registering/unregistering container metrics triggered by ContainerEvent and ContainersMonitorEvent are conflict which cause uncaught exception in ContainerMonitorImpl.
YARN-5191: Renamed the newly added “download=true” option for getting logs via NMWebServices and AHSWebServices to be a better "format" option.
YARN-5199: Close LogReader in in AHSWebServices#getStreamingOutput and FileInputStream in NMWebServices#getLogs.
YARN-5200: Enhanced "yarn logs" to be able to get a list of containers whose logs are aggregated via a "show_container_log_info" option.
YARN-5206: RegistrySecurity includes id:pass in exception text if considered invalid.
YARN-5208: Run TestAMRMClient TestNMClient TestYarnClient TestClientRMTokens TestAMAuthorization tests with hadoop.security.token.service.use_ip enabled.
YARN-5213: Fix a bug in LogCLIHelpers which cause TestLogsCLI#testFetchApplictionLogs fails intermittently.
YARN-5214: Fixed locking in DirectoryCollection to avoid hanging NMs when various code-paths hit slow disks.
YARN-5223: Container line in yarn logs output for a live application should include the hostname for the container.
YARN-5224: Added new web-services /containers/{containerid}/logs & /containers/{containerid}/logs/{filename} and using them in "yarn logs" CLI to get logs of finished containers of a running application.
YARN-5226: Remove AHS enable check from LogsCLI#fetchAMContainerLogs.
YARN-5227: Yarn logs command: no need to specify applicationId when specifying containerId.
YARN-5233: Support for specifying a path for ATS plugin jars.
YARN-5237: Fix missing log files issue in rolling log aggregation.
YARN-5246: NMWebAppFilter web redirects drop query parameters.
YARN-5251: Yarn CLI to obtain App logs for last 'n' bytes fails.
YARN-5266: Wrong exit code while trying to get app logs using regex via CLI.
YARN-5270: Solve miscellaneous issues caused by YARN-4844.
YARN-5294: Pass remote ip address down to YarnAuthorizationProvider.
YARN-5296: NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl.
YARN-5297: Avoid printing a stack trace when recovering an app after the RM restarts.
YARN-5298: Mount usercache and NM filecache directories into Docker container.
YARN-5299: Log Docker run command when container fails.
YARN-5314: Fixed a ConcurrentModificationException in ATS v1.5 EntityGroupFSTimelineStore.
YARN-5333: Some recovered apps are put into default queue when RM HA.
YARN-5337: Fix Dshell OOM issues.
YARN-5339: Passing file to -out for YARN log CLI doesnt give warning or error code.
YARN-5340: Fixed a race condition in RollingLevelDBTimelineStore that caused loss of Timeline events.
YARN-5342: Improve non-exclusive node partition resource allocation in Capacity Scheduler.
YARN-5353: ResourceManager can leak delegation tokens when they are shared across apps. (Jason Lowe via Varun Saxena).
YARN-5363: For AM containers, or for containers of running-apps, "yarn logs" incorrectly only (tries to) shows syslog file-type by default.
YARN-5368: Fixed a memory leak in Timeline server.
YARN-5432: Lock already held by another process while LevelDB cache store creation for dag.
YARN-5440: Use AHSClient in YarnClient when TimelineServer is running.
YARN-5456: container-executor support for FreeBSD, NetBSD, and others if conf path is absolute. .
YARN-5470: Differentiate exactly match with regex in yarn log CLI.
YARN-5476: Not existed application reported as ACCEPTED state by YarnClientImpl.
YARN-5523: Yarn running container log fetching causes OutOfMemoryError.
YARN-5524: Yarn live log aggregation does not throw if command line arg is wrong.
YARN-5539: TimelineClient failed to retry on java.net.SocketTimeoutException: Read timed out.
YARN-5549: AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately.
YARN-5551: : Ignore file backed pages from memory computation when smaps is enabled.
YARN-5659: getPathFromYarnURL should use standard methods.
YARN-5677: RM should transition to standby when connection is lost for an extended period.
YARN-5704: Provide config knobs to control enabling/disabling new/work in progress features in container-executor.
YARN-5711: Propogate exceptions back to client when using hedging RM failover provider. Create a new task attempt with failed map task.
YARN-5718: TimelineClient (and other places in YARN) shouldn't over-write HDFS client retry settings which could cause unexpected behavior.
YARN-5731: Preemption calculation is not accurate when reserved containers are present in queue.
YARN-5773: RM recovery too slow due to LeafQueue#activateApplications.
YARN-5864: Capacity Scheduler queue priority.
YARN-5873: RM crashes with NPE if generic application history is enabled.
YARN-5910: Support for multi-cluster delegation tokens.
YARN-5915: ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write.
YARN-5923: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled.
YARN-5923: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled.
YARN-5988: RM unable to start in secure setup.
YARN-5999: AMRMClientAsync will stop if any exceptions thrown on allocate call.
YARN-6011: Add a new web service to list the files on a container in AHSWebService.
YARN-6029: CapacityScheduler deadlock when ParentQueue#getQueueUserAclInfo is called by one thread and LeafQueue#assignContainers is releasing excessive reserved container by another thread.
YARN-6031: Application recovery has failed when node label feature is turned off during RM recovery.
YARN-6054: TimelineServer fails to start when some LevelDb state files are missing.
YARN-6068: Log aggregation get failed when NM restart even with recovery. .
YARN-6081: LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container.
YARN-6082: Invalid REST api response for getApps since queueUsagePercentage is coming as INF.
YARN-6099: Improve webservice to list aggregated log files.
YARN-6100: Improve YARN webservice to output aggregated container logs.
YARN-6104: RegistrySecurity overrides zookeeper sasl system properties.
YARN-6108: Improve AHS webservice to accept NM address as a parameter to get container logs.
YARN-6113: Re-direct NM Web Service to get container logs for finished applications.
YARN-6123: Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed.
YARN-6126: Obtaining app logs for Running application fails with json parse error.
YARN-6137: Yarn client implicitly invoke ATS client which accesses HDFS.
YARN-6145: Improve log message on fail over.
YARN-6174: Log files pattern should be same for both running and finished container.
YARN-6177: Yarn client should exit with an informative error message if an incompatible Jersey library is used at client.
YARN-6189: ApplicationNotFoundException due to RM failed over should throw more user friendly exception.
YARN-6228: EntityGroupFSTimelineStore should allow configurable cache stores.
YARN-6270: WebUtils.getRMWebAppURLWithScheme() needs to honor RM HA setting.
YARN-6271: yarn rmadin -getGroups returns information from standby RM.
YARN-6294: ATS client should better handle Socket closed case.
YARN-6304: Skip rm.transitionToActive call to RM if RM is already active.
YARN-6313: yarn logs cli should provide logs for a completed container even when application is still running.
YARN-6314: Potential infinite redirection on YARN log redirection web service.
YARN-6332: Make RegistrySecurity use short user names for ZK ACLs.
YARN-6339: Improve performance for createAndGetApplicationReport.
YARN-6367: YARN logs CLI needs alway check containerLogsInfo/containerLogInfo before parse the JSON object from NMWebService.
YARN-6726: Fix issues with docker commands executed by container-executor.
YARN-6890: Not display killApp button on UI if UI is unsecured but cluster is secured.
YARN-7090: testRMRestartAfterNodeLabelDisabled[1] UT Fails.

HDP 2.6.1 provided Hadoop Common 2.7.3 and the following Apache patches:

HADOOP-12173: NetworkTopology::add calls toString always.
HDFS-11499: Decommissioning stuck because of failing recovery.
HDFS-11545: Propagate DataNode's slow disks info to the NameNode via Heartbeat.
HDFS-11547: Add logs for slow BlockReceiver while writing data to disk.
HDFS-11551: Handle SlowDiskReport from DataNode at the NameNode. .
HDFS-11560: Expose slow disks via NameNode JMX.
HDFS-11570: Unit test for NameNodeStatusMXBean.
HDFS-11603: Improve slow mirror/disk warnings in BlockReceiver.
HDFS-11722: Change Datanode file IO profiling sampling to percentage.
HDFS-11842: TestDataNodeOutlierDetectionViaMetrics UT fails intermittently.
MAPREDUCE-6702: Fixed failures in TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv.
YARN-5368: Fixed a memory leak in Timeline server.

HDP 2.6.0 provided Hadoop Common 2.7.3 the following Apache patches:

HADOOP-1023: Add more logging in WASB page blob read function.
HADOOP-1026: Logging bug in WASB causing out of index exception.
HADOOP-1540: breaks backward compatibilty.
HADOOP-1540: Support file exclusion list in distcp.
HADOOP-3733: "s3x:" URLs break when Secret Key contains a slash, even if encoded.
HADOOP-5323: Trash documentation should describe its directory structure and configurations.
HADOOP-5470: RunJar.unJar() should write the last modified time found in the jar entry to the uncompressed file.
HADOOP-7139: Allow appending to existing SequenceFiles
HADOOP-7256: Resource leak during failure scenario of closing of resources.
HADOOP-7713: dfs -count -q should label output column
HADOOP-7713: dfs -count -q should label output column .
HADOOP-7817: RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos()
HADOOP-7930: Kerberos relogin interval in UserGroupInformation should be configurable
HADOOP-8151: Error handling in snappy decompressor throws invalid exceptions.
HADOOP-8436: NPE In getLocalPathForWrite
HADOOP-8437: getLocalPathForWrite should throw IOException for invalid paths.
HADOOP-8751: NPE in Token.toString() when Token is constructed using null identifier.
HADOOP-8818: Use equals instead == in MD5MD5CRC32FileChecksum and TFileDumper.
HADOOP-8934: Shell command ls should include sort options
HADOOP-9121: InodeTree.java has redundant check for vName while throwing exception.
HADOOP-9242: Duplicate surefire plugin config in hadoop-common.
HADOOP-9477: Add posixGroups support for LDAP groups mapping service.
HADOOP-10035: Cleanup TestFilterFileSystem.
HADOOP-10048: LocalDirAllocator should avoid holding locks while accessing the filesystem.
HADOOP-10300: Allowed deferred sending of call responses.
HADOOP-10365: BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block.
HADOOP-10406: TestIPC.testIpcWithReaderQueuing may fail.
HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.
HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.
HADOOP-1062:added hadoop-azure-datalake and azure-data-lake-store-sdk to all the wasb locations.
HADOOP-10775: Shell operations to fail with meaningful errors on windows if winutils.exe not found.
HADOOP-10823: TestReloadingX509TrustManager is flaky.
HADOOP-10965: Print fully qualified path in CommandWithDestination error messages.
HADOOP-10971: Add -C flag to make `hadoop fs -ls` print filenames only.
HADOOP-11031: Design Document for Credential Provider API.
HADOOP-11098: [JDK8] Max Non Heap Memory default changed between JDK7 and 8.
HADOOP-11212: NetUtils.wrapException to handle SocketException explicitly.
HADOOP-11218: Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.
HADOOP-11252: RPC client does not time out by default
HADOOP-11262: Enable YARN to use S3A.
HADOOP-11361: Fix a race condition in MetricsSourceAdapter.updateJmxCache.
HADOOP-11404: Clarify the "expected client Kerberos principal is null" authorization message.
HADOOP-11491: HarFs incorrectly declared as requiring an authority.
HADOOP-11581: Multithreaded correctness Warnings #org.apache.hadoop.fs.shell.Ls
HADOOP-11613: Remove commons-httpclient dependency from hadoop-azure.
HADOOP-11628: SPNEGO auth does not work with CNAMEs in JDK8.
HADOOP-11685: StorageException complaining "no lease ID" during HBase distributed log splitting.
HADOOP-11687: Ignore x-* and response headers when copying an Amazon S3 object.
HADOOP-11692: Improve authentication failure WARN message to avoid user confusion.
HADOOP-11713: ViewFileSystem should support snapshot methods.
HADOOP-11717: Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth
HADOOP-11772: RPC Invoker relies on static ClientCache which has synchronized(this) blocks.
HADOOP-11780: Prevent IPC reader thread death.
HADOOP-11785: Reduce the number of listStatus operation in distcp buildListing
HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm
HADOOP-11812: Implement listLocatedStatus for ViewFileSystem to speed up split calculation
HADOOP-11827: Speed-up distcp buildListing() using threadpool
HADOOP-11859: PseudoAuthenticationHandler fails with httpcomponents v4.4.
HADOOP-11864: JWTRedirectAuthenticationHandler breaks java8 javadocs.
HADOOP-11876: Refactor code to make it more readable, minor maybePrintStats bug
HADOOP-11901: BytesWritable fails to support 2G chunks due to integer overflow.
HADOOP-11918: Listing an empty s3a root directory throws FileNotFound.
HADOOP-11924: Tolerate JDK-8047340-related exceptions in Shell#isSetSidAvailable preventing class init.
HADOOP-11932: MetricsSinkAdapter may hang when being stopped.
HADOOP-11960: Enable Azure-Storage Client Side logging.
HADOOP-12001: Fixed LdapGroupsMapping to include configurable Posix UID and GID attributes during the search.
HADOOP-12006: Remove unimplemented option for from document in branch-2.7.
HADOOP-12009: Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus. backport to 2.8.
HADOOP-12017: Hadoop archives command should use configurable replication factor when closing
HADOOP-12043: Display warning if defaultFs is not set when running fs commands.
HADOOP-12045: Enable LocalFileSystem#setTimes to change atime.
HADOOP-12049: Control http authentication cookie persistence via configuration.
HADOOP-12052:IPC client downgrades all exception types to IOE, breaks callers trying to use them.
HADOOP-12054: RPC client should not retry for InvalidToken exceptions.
HADOOP-12056: Use DirectoryStream in DiskChecker#checkDirs to detect errors when listing a directory.
HADOOP-12058: Fix dead links to DistCp and Hadoop Archives pages.
HADOOP-12059: S3Credentials should support use of CredentialProvider.
HADOOP-12061: Incorrect command in single cluster setup document.
HADOOP-12073: Azure FileSystem PageBlobInputStream does not return -1 on EOF.
HADOOP-12074: in Shell.java#runCommand() rethrow InterruptedException as InterruptedIOException
HADOOP-12076: Incomplete Cache Mechanism in CredentialProvider API.
HADOOP-12089: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.
HADOOP-12095: org.apache.hadoop.fs.shell.TestCount fails.
HADOOP-12097: Allow port range to be specified while starting webapp.
HADOOP-12100: ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermission.
HADOOP-12103: Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing.
HADOOP-12107: long running apps may have a huge number of StatisticsData instances under FileSystem
HADOOP-12119: hadoop fs -expunge does not work for federated namespace
HADOOP-12158: Improve error message in TestCryptoStreamsWithOpensslAesCtrCryptoCodec when OpenSSL is not installed.
HADOOP-12159: Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces
HADOOP-12161: Add getStoragePolicy API to the FileSystem interface
HADOOP-12169:ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse.
HADOOP-12185: NetworkTopology is not efficient adding/getting/removing nodes.
HADOOP-12186: ActiveStandbyElector shouldn't call monitorLockNodeAsync multiple times
HADOOP-12189: Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.
HADOOP-12191: Bzip2Factory is not thread safe.
HADOOP-12213: Interrupted exception can occur when Client#stop is called.
HADOOP-12229:Fix inconsistent subsection titles in filesystem.md.
HADOOP-12230: hadoop-project declares duplicate, conflicting curator dependencies.
HADOOP-12232: Upgrade Tomcat dependency to 6.0.44.
HADOOP-12239: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.
HADOOP-12242: Add in-page TOC to filesystem specification pages.
HADOOP-12252: LocalDirAllocator should not throw NPE with empty string configuration.
HADOOP-12258: Need translate java.nio.file.NoSuchFileException to FileNotFoundException to avoid regression.
HADOOP-12259: Utility to Dynamic port allocation
HADOOP-12269: Update aws-sdk dependency to 1.10.6
HADOOP-12280: Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded
HADOOP-12284: UserGroupInformation doAs can throw misleading exception
HADOOP-12291: Add support for nested groups in LdapGroupsMapping.
HADOOP-12292: Make use of DeleteObjects optional.
HADOOP-12296: when setnetgrent returns 0 in linux, exception should be thrown.
HADOOP-12304: Applications using FileContext fail with the default file system configured to be wasb/s3/etc.
HADOOP-12317: Applications fail on NM restart on some linux distro because NM container recovery declares AM container as LOST
HADOOP-12324: Better exception reporting in SaslPlainServer.
HADOOP-12325: RPC Metrics : Add the ability track and log slow RPCs.
HADOOP-12334: Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries.
HADOOP-12345: Pad hostname correctly in CredentialsSys.java.
HADOOP-12346: Increase some default timeouts / retries for S3a connector.
HADOOP-12348: MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.
HADOOP-12350: WASB Logging: Improve WASB Logging around deletes, reads and writes.
HADOOP-12358: Add -safely flag to rm to prompt when deleting many files.
HADOOP-12359: hadoop fs -getmerge doc is wrong.
HADOOP-12366: expose calculated paths.
HADOOP-12374: Description of hdfs expunge command is confusing.
HADOOP-12386: RetryPolicies.RETRY_FOREVER should be able to specify a retry interval.
HADOOP-12404: Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class.
HADOOP-12406: Fixed AbstractMapWritable.readFields to use the thread's ClassLoader to load class instead of System ClassLoader.
HADOOP-12407: Test failing: hadoop.ipc.TestSaslRPC.
HADOOP-12413: AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.
HADOOP-12415: Add io.netty dependency to hadoop-nfs and to hadoop-hdfs, needed to build Bigtop successfuly, see BIGTOP-2049.
HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.
HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.
HADOOP-12423: Handle failure of registering shutdownhook by ShutdownHookManager in static block
HADOOP-12426: Add Entry point for Kerberos health check.
HADOOP-12437: Allow SecurityUtil to lookup alternate hostnames.
HADOOP-12438: TestLocalFileSystem tests can fail on Windows after HDFS-8767 fix for handling pipe.
HADOOP-12440: TestRPC#testRPCServerShutdown did not produce the desired thread states before shutting down.
HADOOP-12441: Fixed kill-command behavior to work correctly across OSes by using bash shell built-in.
HADOOP-12442: Display help if the command option to 'hdfs dfs' is not valid
HADOOP-12444: Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel.
HADOOP-12449: TestDNS and TestNetUtils failing if no network.
HADOOP-12453: Support decoding KMS Delegation Token with its own Identifier.
HADOOP-12457: [JDK8] Fix a failure of compiling common by javadoc.
HADOOP-12463: Fix TestShell.testGetSignalKillCommand failure on windows.
HADOOP-12464: Interrupted client may try to fail-over and retry.
HADOOP-12465: Incorrect javadoc in WritableUtils.java.
HADOOP-12469: distcp should not ignore the ignoreFailures option.
HADOOP-12472: Make GenericTestUtils.assertExceptionContains robust.
HADOOP-12478: Shell.getWinUtilsPath() has been renamed Shell.getWinutilsPath().
HADOOP-12481: JWTRedirectAuthenticationHandler doesn't Retain Original Query String.
HADOOP-12482: Race condition in JMX cache update.
HADOOP-12483: Maintain wrapped SASL ordering for postponed IPC responses.
HADOOP-12484: Single File Rename Throws Incorrectly In Potential Race Condition Scenarios.
HADOOP-12508: delete fails with exception when lease is held on blob.
HADOOP-12520: Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests.
HADOOP-12526: there are duplicate dependency definitions in pom's
HADOOP-12533: Introduce FileNotFoundException in WASB for read and seek API.
HADOOP-12535: Run FileSystem contract tests with hadoop-azure.
HADOOP-12537: S3A to support Amazon STS temporary credentials.
HADOOP-12540: TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown.
HADOOP-12542: TestDNS fails on Windows after HADOOP-12437.
HADOOP-12545: Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp.
HADOOP-12548: Read s3a creds from a Credential Provider.
HADOOP-12551: Introduce FileNotFoundException for WASB FileSystem API
HADOOP-12555: WASB to read credentials from a credential provider.
HADOOP-12559: KMS connection failures should trigger TGT renewal.
HADOOP-12564: Upgrade JUnit3 TestCase to JUnit 4 in org.apache.hadoop.io package.
HADOOP-12565: Replace DSA with RSA for SSH key type in SingleCluster.md.
HADOOP-12568: Update core-default.xml to describe posixGroups support.
HADOOP-12570: HDFS Secure Mode Documentation updates.
HADOOP-12573: TestRPC.testClientBackOff failing.
HADOOP-12577: Bump up commons-collections version to 3.2.2 to address a security flaw.
HADOOP-12588: Fix intermittent test failure of TestGangliaMetrics.
HADOOP-12589: Fix intermittent test failure of TestCopyPreserveFlag
HADOOP-12590: TestCompressorDecompressor failing without stack traces
HADOOP-12602: TestMetricsSystemImpl#testQSize occasionally fails.
HADOOP-12603: TestSymlinkLocalFSFileContext#testSetTimesSymlinkToDir occasionally fail.
HADOOP-12604: Exception may be swallowed in KMSClientProvider.
HADOOP-12605: Fix intermittent failure of TestIPC.testIpcWithReaderQueuingi.
HADOOP-12608: Fix exception message in WASB when connecting with anonymous credential.
HADOOP-12609: Fix intermittent failure of TestDecayRpcScheduler.
HADOOP-12613: TestFind.processArguments occasionally fails.
HADOOP-12622: Improve the loggings in RetryPolicies and RetryInvocationHandler.
HADOOP-12634: Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy.
HADOOP-12635: Adding Append API support for WASB.
HADOOP-12636: Prevent ServiceLoader failure init for unused FileSystems.
HADOOP-12653: Use SO_REUSEADDR to avoid getting "Address already in use" when using kerberos and attempting to bind to any port on the local IP address.
HADOOP-12659: Incorrect usage of config parameters in token manager of KMS.
HADOOP-12663: Remove Hard-Coded Values From FileSystem.java.
HADOOP-12666: Support Microsoft Azure Data Lake - as a file system in Hadoop.
HADOOP-12667:s3a to support createNonRecursive API.
HADOOP-12668: Support excluding weak Ciphers in HttpServer2 through ssl- server.conf.
HADOOP-12672: RPC timeout should not override IPC ping interval
HADOOP-12675: Fix description about retention period in usage of expunge command.
HADOOP-12678: Handle empty rename pending metadata file during atomic rename in redo path.
HADOOP-12682: Fix TestKMS#testKMSRestart* failure.
HADOOP-12688: Fix deadlinks in Compatibility.md.
HADOOP-12689: S3 filesystem operations stopped working correctly
HADOOP-12691: Add CSRF Filter for REST APIs to Hadoop Common.
HADOOP-12691: Move files to correct location.
HADOOP-12696: Add tests for S3Filesystem Contract
HADOOP-12699: TestKMS#testKMSProvider intermittently fails during 'test rollover draining'.
HADOOP-12700: Remove unused import in TestCompressorDecompressor.java.
HADOOP-12706: TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally.
HADOOP-12711: Remove dependency on commons-httpclient for ServletUtil
HADOOP-12715: TestValueQueue#testgetAtMostPolicyALL fails intermittently.
HADOOP-12716: KerberosAuthenticator#doSpnegoSequence use incorrect class to determine isKeyTab in JDK8.
HADOOP-12717: NPE when trying to rename a directory in Windows Azure Storage FileSystem.
HADOOP-12723:S3A: Add ability to plug in any AWSCredentialsProvider.
HADOOP-12731: Remove useless boxing/unboxing code.
HADOOP-12736: TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out.
HADOOP-12738: Create unit test to automatically compare Common related classes and core-default.xml.
HADOOP-12751: While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple.
HADOOP-12752: Improve diagnostics/use of envvar/sysprop credential propagation.
HADOOP-12758: Extend CSRF Filter with UserAgent Checks.
HADOOP-12761: incremental maven build is not really incremental
HADOOP-12765: HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections.
HADOOP-12767: Update apache httpclient version to 4.5.2; httpcore to 4.4.4.
HADOOP-12772: NetworkTopologyWithNodeGroup.getNodeGroup() can loop infinitely for invalid 'loc' values.
HADOOP-12773: HBase classes fail to load with client/job classloader enabled
HADOOP-12774: s3a should use UGI.getCurrentUser.getShortname() for username.
HADOOP-12776: Remove getaclstatus call for non-acl commands in getfacl.
HADOOP-12780: During WASB atomic rename handle crash when one directory has been renamed but not file under it.
HADOOP-12782: Faster LDAP group name resolution with ActiveDirectory.
HADOOP-12786: "hadoop key" command usage is not documented.
HADOOP-12787: KMS SPNEGO sequence does not work with WEBHDFS.
HADOOP-12788: OpensslAesCtrCryptoCodec should log which random number generator is used.
HADOOP-12789: log classpath of ApplicationClassLoader at INFO level.
HADOOP-12792: TestUserGroupInformation#testGetServerSideGroups fails in chroot.
HADOOP-12793: Write a new group mapping service guide
HADOOP-12794: Support additional compression levels for GzipCodec.
HADOOP-12795: KMS does not log detailed stack trace for unexpected errors.
HADOOP-12800: Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs.
HADOOP-12801: Suppress obsolete S3FileSystem tests.
HADOOP-12804: Read Proxy Password from Credential Providers in S3 FileSystem.
HADOOP-12805: Annotate CanUnbuffer with @InterfaceAudience.Public
HADOOP-12807: S3AFileSystem should read AWS credentials from environment variables.
HADOOP-12810: FileSystem#listLocatedStatus causes unnecessary RPC calls
HADOOP-12817: Enable TLS v1.1 and 1.2
HADOOP-12825: Log slow name resolutions.
HADOOP-12828: Print user when services are started.
HADOOP-12829: StatisticsDataReferenceCleaner swallows interrupt exceptions
HADOOP-12831: LocalFS/FSOutputSummer NPEs in constructor if bytes per checksum set to 0
HADOOP-12844:Recover when S3A fails on IOException in read().
HADOOP-12846: Credential Provider Recursive Dependencies.
HADOOP-12847: hadoop daemonlog should support https and SPNEGO for Kerberized cluster.
HADOOP-12849: TestSymlinkLocalFSFileSystem fails intermittently.
HADOOP-12851: S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders.
HADOOP-12853: Change WASB documentation regarding page blob support.
HADOOP-12870: Fix typo admininistration in CommandsManual.md.
HADOOP-12871: Fix dead link to NativeLibraries.html in CommandsManual.md.
HADOOP-12872: Fix formatting in ServiceLevelAuth.md.
HADOOP-12875: [Azure Data Lake] Support for contract test and unit test cases.
HADOOP-12878: KMS SPNEGO sequence does not work with WEBHDFS.
HADOOP-12886: Exclude weak ciphers in SSLFactory through ssl-server.xml.
HADOOP-12891: S3AFileSystem should configure Multipart Copy threshold and chunk size.
HADOOP-12893: Update LICENSE.txt and NOTICE.txt.
HADOOP-12893: Update LICENSE.txt and NOTICE.txt.
HADOOP-12893: Update LICENSE.txt and NOTICE.txt".
HADOOP-12895: SSLFactory#createSSLSocketFactory exception message is wrong.
HADOOP-12901: Add warning log when KMSClientProvider cannot create a connection to the KMS server.
HADOOP-12902: JavaDocs for SignerSecretProvider are out-of-date in AuthenticationFilter.
HADOOP-12903: IPC Server should allow suppressing exception logging by type,.
HADOOP-12903: IPC Server should allow suppressing exception logging by type, not log 'server too busy' messages.
HADOOP-12906: AuthenticatedURL should convert a 404/Not Found into an FileNotFoundException.
HADOOP-12916: Allow RPC scheduler/callqueue backoff using response times.
HADOOP-12926: lz4.c does not detect 64-bit mode properly.
HADOOP-12929: JWTRedirectAuthenticationHandler must accommodate null expiration time.
HADOOP-12943: Add -w -r options in dfs -test command.
HADOOP-12947: Update documentation Hadoop Groups Mapping to add static group mapping, negative cache.
HADOOP-12950: ShutdownHookManager should have a timeout for each of the Registered shutdown hook.
HADOOP-12954: Add a way to change hadoop.security.token.service.use_ip
HADOOP-12958: PhantomReference for filesystem statistics can trigger OOM.
HADOOP-12962: KMS key names are incorrectly encoded when creating key.
HADOOP-12963:Allow using path style addressing for accessing the s3 endpoint.
HADOOP-12964: Http server vulnerable to clickjacking.
HADOOP-12973: Make DU pluggable.
HADOOP-12975: Add jitter to CachingGetSpaceUsed's thread
HADOOP-12977:s3a to handle delete("/", true) robustly.
HADOOP-12982:Document missing S3A and S3 properties.
HADOOP-12984: Add GenericTestUtils.getTestDir method and use it for emporary directory in tests.
HADOOP-12985: Support MetricsSource interface for DecayRpcScheduler Metrics.
HADOOP-12989: Some tests in org.apache.hadoop.fs.shell.find occasionally time out.
HADOOP-12993: Change ShutdownHookManger complete shutdown log from INFO to DEBUG.
HADOOP-12994: Specify PositionedReadable, add contract tests, fix problems.
HADOOP-13008: Add XFS Filter for UIs to Hadoop Common.
HADOOP-13011: Clearly Document the Password Details for Keystore-based Credential Providers
HADOOP-13018: Make Kdiag check whether hadoop.token.files points to existent and valid files.
HADOOP-13024: Distcp with -delete feature on raw data not implemented.
HADOOP-13026: Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator.
HADOOP-13028:add low level counter metrics for S3A; use in read performance tests.
HADOOP-13030: Handle special characters in passwords in KMS startup script.
HADOOP-13037: Refactor Azure Data Lake Store as an independent FileSystem.
HADOOP-13039: Add documentation for configuration property ipc.maximum.data.length
HADOOP-13042: Restore lost leveldbjni LICENSE and NOTICE changes.
HADOOP-13047:S3a Forward seek in stream length to be configurable.
HADOOP-13052: ChecksumFileSystem mishandles crc file permissions.
HADOOP-13058:S3A FS fails during init against a read-only FS if multipart purge.
HADOOP-13065: Add a new interface for retrieving FS and FC Statistics
HADOOP-13072: WindowsGetSpaceUsed constructor should be public
HADOOP-13079: Add -q option to Ls to print ? instead of non-printable characters.
HADOOP-13084: Fix ASF License warnings in branch-2.7.
HADOOP-13098: Dynamic LogLevel setting page should accept case-insensitive log level string.
HADOOP-13103: Group resolution from LDAP may fail on javax.naming.ServiceUnavailableException.
HADOOP-13105: Support timeouts in LDAP queries in LdapGroupsMapping.
HADOOP-13113:Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel.
HADOOP-13116:Jets3tNativeS3FileSystemContractTest does not run.
HADOOP-13122:Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.
HADOOP-13130: s3a failures can surface as RTEs, not IOEs.
HADOOP-13131: Add tests to verify that S3A supports SSE-S3 encryption.
HADOOP-13138: Unable to append to a SequenceFile with Compression.NONE.
HADOOP-13139:Branch-2: S3a to use thread pool that blocks clients.
HADOOP-13140: FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes.
HADOOP-13145:In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.
HADOOP-13146: Refactor RetryInvocationHandler.
HADOOP-13155: Implement TokenRenewer to renew and cancel delegation tokens in KMS.
HADOOP-13158:S3AFileSystem#toString might throw NullPointerException due to null cannedACL.
HADOOP-13159: Fix potential NPE in Metrics2 source for DecayRpcScheduler.
HADOOP-13162: Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs.
HADOOP-13163:Reuse pre-computed filestatus in Distcp-CopyMapper
HADOOP-13164:Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories.
HADOOP-13168: Support Future.get with timeout in ipc async calls
HADOOP-13169: Randomize file list in SimpleCopyListing.
HADOOP-13171: Add StorageStatistics to S3A; instrument some more operations.
HADOOP-13179: GenericOptionsParser is not thread-safe because commons-cli OptionBuilder is not thread-safe.
HADOOP-13183: S3A proxy tests fail after httpclient/httpcore upgrade.
HADOOP-13188:S3A file-create should throw error rather than overwrite directories.
HADOOP-13192: org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly.
HADOOP-13197: Add non-decayed call metrics for DecayRpcScheduler.
HADOOP-13202: Avoid possible overflow in org.apache.hadoop.util.bloom.BloomFilter#getNBytes.
HADOOP-13203: S3a: Consider reducing the number of connection aborts by setting correct length in s3 request.
HADOOP-13207: Specify FileSystem listStatus, listFiles and RemoteIterator.
HADOOP-13208: S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories.
HADOOP-13212:Provide an option to set the socket buffers in S3AFileSystem
HADOOP-13213: Small Documentation bug with AuthenticatedURL in hadoop-auth.
HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials.
HADOOP-13239:Deprecate s3:// in branch-2.
HADOOP-13241: document s3a better.
HADOOP-13242: Authenticate to Azure Data Lake using client ID and keys.
HADOOP-13244: o.a.h.ipc.Server#Server should honor handlerCount when queueSizePerHandler is specified in consturctor.
HADOOP-13249: RetryInvocationHandler need wrap InterruptedException in IOException when call Thread.sleep.
HADOOP-13251: Authenticate with Kerberos credentials when renewing KMS delegation token.
HADOOP-13252: Tune S3A provider plugin mechanism.
HADOOP-13254: Create framework for configurable disk checkers
HADOOP-13255: KMSClientProvider should check and renew tgt when doing delegation token operations.
HADOOP-13257: Improve Azure Data Lake contract tests.
HADOOP-13263: Reload cached groups in background after expiry.
HADOOP-13270: BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks.
HADOOP-13280: FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps
HADOOP-13283: Support reset operation for new global storage statistics and per FS storage stats.
HADOOP-13285: DecayRpcScheduler MXBean should only report decayed CallVolumeSummary.
HADOOP-13287: TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +.
HADOOP-13288: Guard null stats key in FileSystemStorageStatistics
HADOOP-13290: Appropriate use of generics in FairCallQueue.
HADOOP-13291: Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented.
HADOOP-13299: JMXJsonServlet is vulnerable to TRACE.
HADOOP-13305: Define common statistics names across schemes.
HADOOP-13309: Document S3A known limitations in file ownership and permission model.
HADOOP-13315: FileContext#umask is not initialized properly.
HADOOP-13316: Enforce Kerberos authentication for required ops in DelegationTokenAuthenticator.
HADOOP-13320: Fix arguments check in documentation for WordCount v2.0.
HADOOP-13323: Downgrade stack trace on FS load from Warn to debug.
HADOOP-13324: s3a tests don't authenticate with S3 frankfurt
HADOOP-13336:S3A to support per-bucket configuration.
HADOOP-13350: Additional fix to LICENSE and NOTICE.
HADOOP-13351: TestDFSClientSocketSize buffer size tests are flaky
HADOOP-13352: Make X-FRAME-OPTIONS configurable in HttpServer2.
HADOOP-13353: LdapGroupsMapping getPassward shouldn't return null when IOException throws.
HADOOP-13354: Update WASB driver to use the latest version
HADOOP-13368: DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation.
HADOOP-13381: KMS clients should use KMS Delegation Tokens from current UGI.
HADOOP-13382: Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, HADOOP-openstack.
HADOOP-13387:users always get told off for using S3 —even when not using it.
HADOOP-13389:TestS3ATemporaryCredentials.testSTS error when using IAM credentials.
HADOOP-13392: [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured.
HADOOP-13393: Omit unsupported fs.defaultFS setting in ADLS documentation.
HADOOP-13403: AzureNativeFileSystem rename/delete performance improvements.
HADOOP-13405:doc for fs.s3a.acl.default indicates incorrect values.
HADOOP-13406:S3AFileSystem: Consider reusing filestatus in delete() and mkdirs().
HADOOP-13422: ZKDelegationTokenSecretManager JaasConfig does not work well with other ZK users in process.
HADOOP-13433:Race in UGI.reloginFromKeytab.
HADOOP-13434: Add quoting to Shell class.
HADOOP-13443: KMS should check the type of underlying keyprovider of KeyProviderExtension before falling back to default.
HADOOP-13446: Support running isolated unit tests separate from AWS integration tests.
HADOOP-13447: Refactor S3AFileSystem to support introduction of separate metadata repository and tests.
HADOOP-13458: LoadBalancingKMSClientProvider#doOp should log IOException stacktrace.
HADOOP-13459: hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time.
HADOOP-13461: NPE in KeyProvider.rollNewVersion.
HADOOP-13466: Add an AutoCloseableLock class.
HADOOP-13467: Shell#getSignalKillCommand should use the bash builtin on Linux.
HADOOP-13470: GenericTestUtils$LogCapturer is flaky.
HADOOP-13494: ReconfigurableBase can log sensitive information.
HADOOP-13502: Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests.
HADOOP-13512: ReloadingX509TrustManager should keep reloading in case of exception.
HADOOP-13513: Java 1.7 support for org.apache.hadoop.fs.azure testcases.
HADOOP-13535: Add jetty6 acceptor startup issue workaround to branch-2.
HADOOP-13540:improve section on troubleshooting s3a auth problems.
HADOOP-13541:explicitly declare the Joda time version S3A depends on.
HADOOP-13546: Override equals and hashCode to avoid connection leakage
HADOOP-13552: RetryInvocationHandler logs all remote exceptions.
HADOOP-13552: RetryInvocationHandler logs all remote exceptions.
HADOOP-13558: UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket.
HADOOP-13560: S3ABlockOutputStream to support huge
HADOOP-13565: KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request.
HADOOP-13590: Retry until TGT expires even if the UGI renewal thread encountered exception.
HADOOP-13599: s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown.
HADOOP-13614: Purge some superfluous/obsolete S3 FS tests that are slowing test runs down.
HADOOP-13638: KMS should set UGI's Configuration object properly.
HADOOP-13674: S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.
HADOOP-13675: Bug in return value for delete() calls in WASB.
HADOOP-13680: fs.s3a.readahead.range to use getLongBytes.
HADOOP-13686: Adding additional unit test for Trash
HADOOP-13692: hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.
HADOOP-13707: If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed.
HADOOP-13710: Supress CachingGetSpaceUsed from logging interrupted exception stacktrace.
HADOOP-13716: Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup.
HADOOP-13727: S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider.
HADOOP-13735:ITestS3AFileContextStatistics.testStatistics() failing.
HADOOP-13737: Cleanup DiskChecker interface.
HADOOP-13738: DiskChecker should perform some disk IO.
HADOOP-13742: Expose NumOpenConnectionsPerUser as a metric. Brahma Reddy Battula.
HADOOP-13749: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.
HADOOP-13782: Make MutableRates metrics thread-local write, aggregate-on-read.
HADOOP-13792: Stackoverflow for schemeless defaultFS with trailing slash.
HADOOP-13801:regression: ITestS3AMiniYarnCluster failing.
HADOOP-13804: MutableStat mean loses accuracy if add(long, long) is used.
HADOOP-13822: Use GlobalStorageStatistics.INSTANCE.reset() at FileSystem#clearStatistics(). Contribute by Brahma Reddy Battula .
HADOOP-13823: s3a rename: fail if dest file exists.
HADOOP-13824: FsShell can suppress the real error if no error message is present.
HADOOP-13826: S3A Deadlock in multipart copy due to thread pool limits.
HADOOP-13830:Intermittent failure of ITestS3NContractRootDir.testRecursiveRootListing "Can not create a Path from an empty string".
HADOOP-13831: Correct check for error code to detect Azure Storage Throttling and provide retries.
HADOOP-13838: KMSTokenRenewer should close providers
HADOOP-13847: KMSWebApp should close KeyProviderCryptoExtension.
HADOOP-13855: Fix a couple of the s3a statistic names to be consistent with the rest.
HADOOP-13857: S3AUtils.translateException to map
HADOOP-13863: Azure: Add a new SAS key mode for WASB.
HADOOP-13871: ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful.
HADOOP-13890: Maintain HTTP/host as SPNEGO SPN support and fix KerberosName parsing.
HADOOP-13930: Azure: Add Authorization support to WASB.
HADOOP-13945: Azure: Add Kerberos and Delegation token support to WASB client.
HADOOP-13956: Read ADLS credentials from Credential Provider.
HADOOP-13962: Update ADLS SDK to 2.1.4.
HADOOP-13988: KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser.
HADOOP-14017: User friendly name for ADLS user and group.
HADOOP-14019: Fix some typos in the s3a docs.
HADOOP-14028: S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.
HADOOP-14029: Fix KMSClientProvider for non-secure proxyuser use case.
HADOOP-14048: REDO operation of WASB#AtomicRename should create placeholder blob for destination folder.
HADOOP-14049: Honour AclBit flag associated to file/folder permission for Azure datalake account.
HADOOP-14055: SwiftRestClient includes pass length in exception if auth fails.
HADOOP-14081:S3A: Consider avoiding array copy in S3ABlockOutputStream
HADOOP-14091: AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari.
HADOOP-14100: Upgrade Jsch jar to latest version to fix vulnerability in old versions.
HADOOP-14138:Remove S3A ref from META-INF service discovery, rely on existing core-default entry.
HADOOP-14150: Implement getHomeDirectory() method in NativeAzureFileSystem.
HADOOP-14214: DomainSocketWatcher::add()/delete() should not self interrupt while looping await().
HDFS-2043: TestHFlush failing intermittently.
HDFS-2580: NameNode#main(...) can make use of GenericOptionsParser.
HDFS-27: HDFS CLI with --config set to default config complains log file not found error.
HDFS-3677: dfs.namenode.edits.dir.required is missing from hdfs- default.xml.
HDFS-3918: EditLogTailer shouldn't log WARN when other node is in standby mode.
HDFS-4015: Safemode should count and report orphaned blocks.
HDFS-4176: EditLogTailer should call rollEdits with a timeout.
HDFS-4185: Add a metric for number of active leases
HDFS-4366: Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks.
HDFS-4396: Add START_MSG/SHUTDOWN_MSG for ZKFC
HDFS-4937: ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom.
HDFS-4946: Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable.
HDFS-5640: Add snapshot methods to FileContext.
HDFS-6054: MiniQJMHACluster should not use static port to avoid binding failure in unit test.
HDFS-6101: TestReplaceDatanodeOnFailure fails occasionally.
HDFS-6184: Capture NN's thread dump when it fails over.
HDFS-6407: Add sorting and pagination in the datanode tab of the NN Web UI.
HDFS-6481: DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs.
HDFS-6533: TestBPOfferService#testBasicFunctionalitytest fails intermittently.
HDFS-6581: Support for writing to single replica in RAM. Applied HDFS-6581.merge.15.patch to Champlain. This is a consolidated patch containing all feature changes and was applied to trunk.
HDFS-6666: Abort NameNode and DataNode startup if security is enabled but block access token is not enabled.
HDFS-6763: Initialize file system-wide quota once on transitioning to active
HDFS-6832: Fix the usage of 'hdfs namenode' command.
HDFS-6860: BlockStateChange logs are too noisy.
HDFS-6945: BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed.
HDFS-7163: WebHdfsFileSystem should retry reads according to the configured retry policy.
HDFS-7166: SbNN Web UI shows #Under replicated blocks and #pending deletion blocks.
HDFS-7284: Add more debug info to BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas.
HDFS-7314: When the DFSClient lease cannot be renewed, abort open-for- write files rather than the entire DFSClient.
HDFS-7390: Provide JMX metrics per storage type.
HDFS-7452: skip StandbyException log for getCorruptFiles()
HDFS-7483: Display information per tier on the Namenode UI.
HDFS-7597: DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping.
HDFS-7608: hdfs dfsclient newConnectedPeer has no write timeout
HDFS-7609: Avoid retry cache collision when Standby NameNode loading edits.
HDFS-7645: Rolling upgrade is restoring blocks from trash multiple times
HDFS-7701: Support reporting per storage type quota and usage with hadoop/hdfs shell.
HDFS-7725: Incorrect "nodes in service" metrics caused all writes to fail.
HDFS-7758: Retire FsDatasetSpi#getVolumes() and use FsDatasetSpi#getVolumeRefs() instead
HDFS-7890: Improve information on Top users for metrics in RollingWindowsManager and lower log level.
HDFS-7923: The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages
HDFS-7928: Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy.
HDFS-7933: fsck should also report decommissioning replicas.
HDFS-7964: Add support for async edit logging.
HDFS-7978: Add LOG.isDebugEnabled() guard for some LOG.debug(..).
HDFS-7990: IBR delete ack should not be delayed.
HDFS-8008: Support client-side back off when the datanodes are congested.
HDFS-8008: Support client-side back off when the datanodes are congested.
HDFS-8009: Signal congestion on the DataNode.
HDFS-8009: Signal congestion on the DataNode.
HDFS-8046: Allow better control of getContentSummary
HDFS-8055: NullPointerException when topology script is missing.
HDFS-8071:(Contributed by Tsz Wo Nicholas Sze).
HDFS-8099: Change "DFSInputStream has been closed already" message to debug log level.
HDFS-8101: DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS classes at runtime.
HDFS-8113: Add check for null BlockCollection pointers in BlockInfoContiguous structures.
HDFS-8131: Implement a space balanced block placement policy.
HDFS-8144: Split TestLazyPersistFiles into multiple tests.
HDFS-8152: Refactoring of lazy persist storage cases.
HDFS-8155: Support OAuth2 in WebHDFS.
HDFS-8180: AbstractFileSystem Implementation for WebHdfs.
HDFS-8205: CommandFormat#parse() should not parse option as value of option.
HDFS-8209: Support different number of datanode directories in MiniDFSCluster.
HDFS-8211: DataNode UUID is always null in the JMX counter.
HDFS-8219: setStoragePolicy with folder behavior is different after cluster restart.
HDFS-8224: Schedule a block for scanning if its metadata file is corrupt.
HDFS-8229: LAZY_PERSIST file gets deleted after NameNode restart.
HDFS-8232: Missing datanode counters when using Metrics2 sink interface.
HDFS-8247: TestDiskspaceQuotaUpdate#testAppendOverTypeQuota is failing.
HDFS-8276: LazyPersistFileScrubber should be disabled if scrubber interval configured zero.
HDFS-8278: When computing max-size-to-move in Balancer, count only the storage with remaining >= default block size.
HDFS-8309: Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows.
HDFS-8310: Fix TestCLI.testAll "help: help for find" on Windows.
HDFS-8311: DataStreamer.transfer() should timeout the socket InputStream.
HDFS-8337: Accessing httpfs via webhdfs doesn't work from a jar with kerberos.
HDFS-8345: Storage policy APIs must be exposed via the FileSystem interface.
HDFS-8384: Allow NN to startup if there are files having a lease but are not under construction.
HDFS-8431: hdfs crypto class not found in Windows.
HDFS-8432: Introduce a minimum compatible layout version to allow downgrade in more rolling upgrade use cases.
HDFS-8435: Support CreateFlag in WebHDFS.
HDFS-8492: DN should notify NN when client requests a missing block
HDFS-8539: Hdfs doesnt have class 'debug' in windows.
HDFS-8542: WebHDFS getHomeDirectory behavior does not match specification.
HDFS-8546: Use try with resources in DataStorage and Storage.
HDFS-8548: Minicluster throws NPE on shutdown.
HDFS-8549: Abort the balancer if an upgrade is in progress.
HDFS-8554: TestDatanodeLayoutUpgrade fails on Windows.
HDFS-8573: Move creation of restartMeta file logic from BlockReceiver to ReplicaInPipeline.
HDFS-8578: On upgrade, Datanode should process all storage/data dirs in parallel.
HDFS-8581: ContentSummary on / skips further counts on yielding lock
HDFS-8593: Calculation of effective layout version mishandles comparison to current layout version in storage.
HDFS-8615: Correct HTTP method in WebHDFS document.
HDFS-8633: Fix setting of dfs.datanode.readahead.bytes in hdfs-default.xml to match DFSConfigKeys.
HDFS-8642: Make TestFileTruncate more reliable.
HDFS-8647: Abstract BlockManager's rack policy into BlockPlacementPolicy.
HDFS-8656: Preserve compatibility of ClientProtocol#rollingUpgrade after finalization.
HDFS-8659: Block scanner INFO message is spamming logs.
HDFS-8674: Improve performance of postponed block scans.
HDFS-8676: Delayed rolling upgrade finalization can cause heartbeat expiration.
HDFS-8696: Make the lower and higher watermark in the DN Netty server configurable.
HDFS-8709: Clarify automatic sync in FSEditLog#logEdit.
HDFS-8722: Optimize datanode writes for small writes and flushes.
HDFS-8729: Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately which occasionally failed.
HDFS-8767: RawLocalFileSystem.listStatus() returns null for UNIX pipefile.
HDFS-8772: Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails.
HDFS-8778: TestBlockReportRateLimiting#testLeaseExpiration can deadlock.
HDFS-8785: TestDistributedFileSystem is failing in trunk.
HDFS-8792: BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory.
HDFS-8797: WebHdfsFileSystem creates too many connections for pread.
HDFS-8806: Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared
HDFS-8809: HDFS fsck reports under construction blocks as CORRUPT.
HDFS-8815: DFS getStoragePolicy implementation using single RPC call
HDFS-8816: Improve visualization for the Datanode tab in the NN UI.
HDFS-8818: Changes the global moveExecutor to per datanode executors and changes MAX_SIZE_TO_MOVE to be configurable.
HDFS-8824: Do not use small blocks for balancing the cluster.
HDFS-8826: In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes.
HDFS-8828: Utilize Snapshot diff report to build diff copy list in distcp.
HDFS-8829: Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning.
HDFS-8831: Trash Support for deletion in HDFS encryption zone.
HDFS-8844: TestHDFSCLI does not cleanup the test directory
HDFS-8845: DiskChecker should not traverse the entire tree
HDFS-8846: Add a unit test for INotify functionality across a layout version upgrade
HDFS-8850: VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks.
HDFS-8852: HDFS architecture documentation of version 2.x is outdated about append write support.
HDFS-8855: Webhdfs client leaks active NameNode connections.
HDFS-8856: Make LeaseManager#countPath O(1).
HDFS-8859: Improve DataNode ReplicaMap memory footprint to save about 45%.
HDFS-8860: Remove unused Replica copyOnWrite code
HDFS-8863: The remaining space check in BlockPlacementPolicyDefault is flawed.
HDFS-8867: Enable optimized block reports.
HDFS-8879: Quota by storage type usage incorrectly initialized upon namenode restart.
HDFS-8880: NameNode metrics logging.
HDFS-8883: NameNode Metrics : Add FSNameSystem lock Queue Length.
HDFS-8884: Fail-fast check in BlockPlacementPolicyDefault#chooseTarget.
HDFS-8885: ByteRangeInputStream used in webhdfs does not override available().
HDFS-8887: Expose storage type and storage ID in BlockLocation.
HDFS-8891: HDFS concat should keep srcs order.
HDFS-8896: DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager.
HDFS-8898: Create API and command-line argument to get quota and quota usage without detailed content summary.
HDFS-8911: NameNode Metric : Add Editlog counters as a JMX metric.
HDFS-8914: Document HA support in the HDFS HdfsDesign.md.
HDFS-8923: Add -source flag to balancer usage message.
HDFS-8930: Block report lease may leak if the 2nd full block report comes when NN is still in safemode
HDFS-8932: NPE thrown in NameNode when try to get TotalSyncCount metric before editLogStream initialization.
HDFS-8939: Test(S)WebHdfsFileContextMainOperations failing on branch-2.
HDFS-8946: Improve choosing datanode storage for block placement.
HDFS-8950: NameNode refresh doesn't remove DataNodes that are no longer in the allowed list.
HDFS-8964: When validating the edit log, do not read at or beyond the file offset that is being written
HDFS-8965: Harden edit log reading code against out of memory errors
HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.
HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.
HDFS-8983: NameNode support for protected directories.
HDFS-8983: NameNode support for protected directories.
HDFS-8995: Flaw in registration bookeeping can make DN die on reconnect.
HDFS-9008: Balancer#Parameters class could use a builder pattern.
HDFS-9008:was not backported correctly.
HDFS-9009: Send metrics logs to NullAppender by default.
HDFS-9019: Adding informative message to sticky bit permission denied exception.
HDFS-9033: dfsadmin -metasave prints "NaN" for cache used%.
HDFS-9038: DFS reserved space is erroneously counted towards non-DFS used.
HDFS-9042: Update document for the Storage policy name
HDFS-9043: Doc updation for commands in HDFS Federation
HDFS-9047: Retire libwebhdfs.
HDFS-9048: DistCp documentation is out-of-dated
HDFS-9063: Correctly handle snapshot path for getContentSummary.
HDFS-9072: Fix random failures in TestJMXGet.
HDFS-9082: Change the log level in WebHdfsFileSystem.initialize() from INFO to DEBUG.
HDFS-9083: Replication violates block placement policy.
HDFS-9106: Transfer failure during pipeline recovery causes permanent write failures.
HDFS-9107: Prevent NNs unrecoverable death spiral after full GC
HDFS-9112: Improve error message for Haadmin when multiple name service IDs are configured.
HDFS-9128: TestWebHdfsFileContextMainOperations and TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows.
HDFS-9130: Use GenericTestUtils#setLogLevel to the logging level.
HFDS-9137:should be backported correctly.
HDFS-9142: Separating Configuration object for namenode(s) in MiniDFSCluster.
HDFS-9145: Tracking methods that hold FSNamesytemLock for too long.
HDFS-9148: Incorrect assert message in TestWriteToReplica#testWriteToTemporary
HDFS-9175: Change scope of 'AccessTokenProvider.getAccessToken()' and 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to public.
HDFS-9178: Slow datanode I/O can cause a wrong node to be marked bad.
HDFS-9184: Logging HDFS operation's caller context into audit logs.
HDFS-9198: Coalesce IBR processing in the NN.
HDFS-9205: Do not schedule corrupt blocks for replication.
HDFS-9210: Fix some misuse of %n in VolumeScanner#printStats.
HDFS-9220: Reading small file
HDFS-9221: HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array.
HDFS-9226: Unit Tests failing due to mockito version issue.
HDFS-9239: DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness.
HDFS-9251: Refactor TestWriteToReplica and TestFsDatasetImpl to avoid explicitly creating Files in the tests code.
HDFS-9259: Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario.
HDFS-9264: Minor cleanup of operations on FsVolumeList#volumes.
HDFS-9273: ACLs on root directory may be lost after NN restart.
HDFS-9276: Failed to Update HDFS Delegation Token for long running application in HA mode.
HDFS-9289: Make DataStreamer#block thread safe and verify genStamp in commitBlock.
HDFS-9290: DFSClient#callAppend() is not backward compatible for slightly older NameNodes.
HDFS-9294: DFSClient deadlock when close file and failed to renew lease.
HDFS-9305: Delayed heartbeat processing causes storm of subsequent heartbeats.
HDFS-9311: Support optional offload of NameNode HA service health checks to a separate RPC server.
HDFS-9313: Possible NullPointerException in BlockManager if no excess replica can be chosen.
HDFS-9314: Improve BlockPlacementPolicyDefault's picking of excess replicas.
HDFS-9317: Document fsck -blockId and -storagepolicy options in branch-2.7.
HDFS-9343: Empty caller context considered invalid.
HDFS-9347: Invariant assumption in TestQuorumJournalManager.shutdown() is wrong
HDFS-9350: Avoid creating temprorary strings in Block.toString() and getBlockName()
HDFS-9354: Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows.
HDFS-9358: TestNodeCount#testNodeCount timed out.
HDFS-9362: TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
HDFS-9364: Unnecessary DNS resolution attempts when creating NameNodeProxies.
HDFS-9383: TestByteArrayManager#testByteArrayManager fails.
HDFS-9384: TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server.
HDFS-9395: Make HDFS audit logging consistant.
HDFS-9397: Fix typo for readChecksum() LOG.warn in BlockSender.java.
HDFS-9402: Switch DataNode.LOG to use slf4j.
HDFS-9406: FSImage may get corrupted after deleting snapshot.
HDFS-9412: getBlocks occupies FSLock and takes too long to complete.
HDFS-9413: getContentSummary() on standby should throw StandbyException.
HDFS-9415: Document dfs.cluster.administrators and dfs.permissions.superusergroup.
HDFS-9426: Rollingupgrade finalization is not backward compatible.
HDFS-9428: Fix intermittent failure of TestDNFencing.testQueueingWithAppend.
HDFS-9431: DistributedFileSystem#concat fails if the target path is relative.
HDFS-9434: Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messags.
HDFS-9445: Datanode may deadlock while handling a bad volume.
HDFS-9466: TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky
HDFS-9467: Fix data race accessing writeLockHeldTimeStamp in FSNamesystem.
HDFS-9470: Encryption zone on root not loaded from fsimage after NN restart. Xiao Chen via wang.
HDFS-9476: TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail
HDFS-9478: Reason for failing ipc.FairCallQueue contruction should be thrown.
HDFS-9484: NNThroughputBenchmark$BlockReportStats should not send empty block reports.
HDFS-9493: Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk.
HDFS-9500: Fix software version counts for DataNodes during rolling upgrade.
HDFS-9505: HDFS Architecture documentation needs to be refreshed.
HDFS-9516: Truncate file fails with data dirs on multiple disks.
HDFS-9517: Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes
HDFS-9521: TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk.
HDFS-9525: hadoop utilities need to support provided delegation tokens.
HDFS-9530: ReservedSpace is not cleared for abandoned Blocks
HDFS-9533: seen_txid in the shared edits directory is modified during bootstrapping.
HDFS-9534: Add CLI command to clear storage policy from a path.
HDFS-9549: TestCacheDirectives#testExceedsCapacity is flaky
HDFS-9555: LazyPersistFileScrubber should still sleep if there are errors in the clear progress.
HDFS-9557: Reduce object allocation in PB conversion.
HDFS-9566: Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method
HDFS-9569: Log the name of the fsimage being loaded for better supportability.
HDFS-9572: Prevent DataNode log spam if a client connects on the data transfer port but sends no data.
HDFS-9574: Reduce client failures during datanode restart.
HDFS-9584: NPE in distcp when ssl configuration file does not exist in class path.
HDFS-9589: Block files which have been hardlinked should be duplicated before the DataNode appends to the them
HDFS-9600: Do not check replication if the block is under construction.
HDFS-9605: Add links to failed volumes to explorer.html in HDFS Web UI.
HDFS-9608: Disk IO imbalance in HDFS with heterogeneous storages.
HDFS-9612: DistCp worker threads are not terminated after jobs are done.
HDFS-9618: Fix mismatch between log level and guard in BlockManager#computeRecoveryWorkForBlocks
HDFS-9619: SimulatedFSDataset sometimes can not find blockpool for the correct namenode
HDFS-9623: Update example configuration of block state change log in log4j.properties.
HDFS-9624: DataNode start slowly due to the initial DU command operations.
HDFS-9625: set replication for empty file failed when set storage policy
HDFS-9626: TestBlockReplacement#testBlockReplacement fails occasionally.
HDFS-9629: Update the footer of Web UI to show year 2016
HDFS-9630: DistCp minor refactoring and clean up.
HDFS-9634: webhdfs client side exceptions don't provide enough details.
HDFS-9638: to branch-2.7.
HDFS-9644: Update encryption documentation to reflect nested EZs.
HDFS-9655: NN should start JVM pause monitor before loading fsimage.
HDFS-9661: Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage moveBlockAcrossStorage and createRbw.
HDFS-9669: TcpPeerServer should respect ipc.server.listen.queue.size
HDFS-9670: DistCp throws NPE when source is root.
HDFS-9672: o.a.h.hdfs.TestLeaseRecovery2 fails intermittently
HDFS-9688: Test the effect of nested encryption zones in HDFS downgrade.
HDFS-9696: Garbage snapshot records linger forever.
HDFS-9701: DN may deadlock when hot-swapping under load.
HDFS-9706: Log more details in debug logs in BlockReceiver's constructor.
HDFS-9710: DN can be configured to send block receipt IBRs in batches.
HDFS-9711: Integrate CSRF prevention filter in WebHDFS.
HDFS-9713: DataXceiver#copyBlock should return if block is pinned.
HDFS-9715: Check storage ID uniqueness on datanode startup
HDFS-9721: Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
HDFS-9724: Degraded performance in WebHDFS listing as it does not reuse ObjectMapper.
HDFS-9726: Refactor IBR code to a new class.
HDFS-9730: Storage ID update does not happen when there is a layout change.
HDFS-9739: DatanodeStorage.isValidStorageId() is broken.
HDFS-9740: Use a reasonable limit in DFSTestUtil.waitForMetric().
HDFS-9743: Fix TestLazyPersistFiles#testFallbackToDiskFull in branch-2.7.
HDFS-9748: Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice.
HDFS-9752: Permanent write failures may happen to slow writers during datanode rolling upgrades
HDFS-9760: WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler
HDFS-9764: DistCp doesn't print value for several arguments including -numListstatusThreads.
HDFS-9765: TestBlockScanner#testVolumeIteratorWithCaching fails intermittently.
HDFS-9766: TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently.
HDFS-9768: Reuse ObjectMapper instance in HDFS to improve the performance.
HDFS-9772: TestBlockReplacement#testThrottler doesn't work as expected.
HDFS-9777: Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh).
HDFS-9779: . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value.
HDFS-9781: FsDatasetImpl#getBlockReports can occasionally throw NullPointerException.
HDFS-9784: Example usage is not correct in Transparent Encryption document.
HDFS-9790: HDFS Balancer should exit with a proper message if upgrade is not finalized
HDFS-9797: Log Standby exceptions thrown by RequestHedgingProxyProvider at DEBUG Level
HDFS-9799: Reimplement getCurrentTrashDir to remove incompatibility.
HDFS-9812: Streamer threads leak if failure happens when closing DFSOutputStream.
HDFS-9839: Reduce verbosity of processReport logging.
HDFS-9842: dfs.datanode.balance.bandwidthPerSec should accept friendly size units.
HDFS-9844: Correct path creation in getTrashRoot to handle root dir.
HDFS-9851: NameNode throws NPE when setPermission is called on a path that does not exist.
HDFS-9854: Log cipher suite negotiation more verbosely.
HDFS-9864: Correct reference for RENEWDELEGATIONTOKEN and CANCELDELEGATIONTOKEN in webhdfs doc.
HDFS-9865: TestBlockReplacement fails intermittently in trunk
HDFS-9871: "Bytes Being Moved" -ve(-1 B) when cluster was already balanced.
HDFS-9874: Long living DataXceiver threads cause volume shutdown to block.
HDFS-9880: TestDatanodeRegistration fails occasionally.
HDFS-9881: DistributedFileSystem#getTrashRoot returns incorrect path for encryption zones.
HDFS-9882: Add heartbeatsTotal in Datanode metrics.
HDFS-9902: Support different values of dfs.datanode.du.reserved per storage type.
HDFS-9904: testCheckpointCancellationDuringUpload occasionally fails.
HDFS-9905: WebHdfsFileSystem#runWithRetry should display original stack trace on error.
HDFS-9906: Remove spammy log spew when a datanode is restarted.
HDFS-9917: IBR accumulate more objects when SNN was down for sometime.
HDFS-9926: MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils.
HDFS-9941: Do not log StandbyException on NN, other minor logging fixes.
HDFS-9947: Block#toString should not output information from derived classes
HDFS-9949: Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared
HDFS-9958: BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages.
HDFS-10178: Permanent write failures can happen if pipeline recoveries occur for the first packet.
HDFS-10182: Hedged read might overwrite user's buf.
HDFS-10186: DirectoryScanner: Improve logs by adding full path of both actual and expected block directories.
HDFS-10189: PacketResponder#toString should include the downstreams for PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE
HDFS-10199: Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing.
HDFS-10216: Distcp -diff throws exception when handling relative path.
HDFS-10217: show 'blockScheduled' tooltip in datanodes table.
HDFS-10223: peerFromSocketAndKey performs SASL exchange before setting connection timeouts.
HDFS-10225: DataNode hot swap drives should disallow storage type changes.
HDFS-10228: TestHDFSCLI fails.
HDFS-10235: Last contact for Live Nodes should be relative time.
HDFS-10239: Fsshell mv fails if port usage doesn't match in src and destination paths.
HDFS-10245: Fix the findbugs warnings in branch-2.7.
HDFS-10253: Fix TestRefreshCallQueue failure
HDFS-10261: TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports.
HDFS-10264: Logging improvements in FSImageFormatProtobuf.Saver.
HDFS-10270: TestJMXGet:testNameNode() fails.
HDFS-10271: Extra bytes are getting released from reservedSpace for append
HDFS-10275: TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly.
HDFS-10277: PositionedReadable test testReadFullyZeroByteFile failing in HDFS.
HDFS-10279: Improve validation of the configured number of tolerated failed volumes.
HDFS-10281: TestPendingCorruptDnMessages fails intermittently.
HDFS-10291: TestShortCircuitLocalRead failing
HDFS-10301: BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order .
HDFS-10301: Remove FBR tracking state to fix false zombie storage detection for interleaving block reports.
HDFS-10309: Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga).
HDFS-10312: Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.
HDFS-10313: Distcp need to enforce the order of snapshot names passed to -diff.
HDFS-10319: Balancer should not try to pair storages with different types.
HDFS-10320: Rack failures may result in NN terminate.
HDFS-10324: Trash directory in an encryption zone should be pre-created with correct permissions.
HDFS-10329: Bad initialisation of StringBuffer in RequestHedgingProxyProvider.
HDFS-10330: Add Corrupt Blocks Information in Metasave output.
HDFS-10335:Mover$Processor#chooseTarget() always chooses the first matching target storage group.
HDFS-10341: Add a metric to expose the timeout number of pending replication blocks
HDFS-10342: BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt.
HDFS-10343: BlockManager#createLocatedBlocks may return blocks on failed storages.
HDFS-10344: DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash.
HDFS-10347: Namenode report bad block method doesn't log the bad block or datanode.
HDFS-10360: DataNode may format directory and lose blocks if current/VERSION is missing.
HDFS-10367: TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception.
HDFS-10372: Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume.
HDFS-10377: CacheReplicationMonitor shutdown log message should use INFO level
HDFS-10390: Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
HDFS-10396: Using -diff option with DistCp may get "Comparison method violates its general contract" exception.
HDFS-10397: Distcp should ignore -delete option if -diff option is provided instead of exiting.
HDFS-10410: RedundantEditLogInputStream.LOG is set to wrong class.
HDFS-10415: TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called
HDFS-10417: Improve error message from checkBlockLocalPathAccess.
HDFS-10423: Increase default value of httpfs maxHttpHeaderSize.
HDFS-10424: DatanodeLifelineProtocol not able to use under security cluster
HDFS-10431: Refactor and speedup TestAsyncDFSRename
HDFS-10437: ReconfigurationProtocol not covered by HDFSPolicyProvider.
HDFS-10438: When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.
HDFS-10440: Improve DataNode web UI
HDFS-10448: CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1
HDFS-10455: Logging the username when deny the setOwner operation.
HDFS-10457: DataNode should not auto-format block pool directory if VERSION is missing.
HDFS-10458: getFileEncryptionInfo should return quickly for non-encrypted cluster.
HDFS-10468: HDFS read ends up ignoring an interrupt.
HDFS-10469: Add number of active xceivers to datanode metrics.
HDFS-10471: DFSAdmin#SetQuotaCommand's help msg is not correct.
HDFS-10474: hftp copy fails when file name with Chinese+special char in branch-2
HDFS-10481: HTTPFS server should correctly impersonate as end user to open file.
HDFS-10485: Fix findbugs warning in FSEditLog.java
HDFS-10488: Update WebHDFS documentation regarding CREATE and MKDIR default permissions.
HDFS-10493: Add links to datanode web UI in namenode datanodes page.
HDFS-10499: TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails Intermittently.
HDFS-10508: DFSInputStream should set thread's interrupt status after catching InterruptException from sleep.
HDFS-10512: VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks.
HDFS-10525: Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap
HDFS-10544: Balancer doesn't work with IPFailoverProxyProvider.
HDFS-10556: DistCpOptions should be validated automatically.
HDFS-10569: A bug causes OutOfIndex error in BlockListAsLongs.
HDFS-10579: HDFS web interfaces lack configs for X-FRAME-OPTIONS protection.
HDFS-10589: Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params.
HDFS-10609: Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications.
HDFS-10617: PendingReconstructionBlocks.size() should be synchronized.
HDFS-10623: Remove unused import of httpclient.HttpConnection from TestWebHdfsTokens.
HDFS-10625: VolumeScanner to report why a block is found bad.
HDFS-10627: Volume Scanner marks a block as "suspect" even if the exception is network-related.
HDFS-10642: TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently.
HDFS-10643: [Unit Test] Namenode should use loginUser(hdfs) to generateEncryptedKey.
HDFS-10643: HDFS namenode should always use service user
HDFS-10653: Optimize conversion from path string to components.
HDFS-10655: Fix path related byte array conversion bugs.
HDFS-10660: Expose storage policy apis via HDFSAdmin interface.
HDFS-10684: WebHDFS DataNode calls fail without parameter createparent.
HDFS-10688: BPServiceActor may run into a tight loop for sending block report when hitting IOException.
HDFS-10703: HA NameNode Web UI should show last checkpoint time.
HDFS-10710: In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui.
HDFS-10713: Throttle FsNameSystem lock warnings.
HDFS-10715: NPE when applying AvailableSpaceBlockPlacementPolicy.
HDFS-10716: In Balancer, the target task should be removed when its size < 0.
HDFS-10735:Distcp using webhdfs on secure HA clusters fails with StandbyException .
HDFS-10752: Several log refactoring/improvement suggestion in HDFS.
HDFS-10757: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.
HDFS-10763: Open files can leak permanently due to inconsistent lease update.
HDFS-10763: Open files can leak permanently due to inconsistent lease update TestTrashWithSecureEncryptionZones diverged from HDFS-10906.
HDFS-10764: Fix INodeFile#getBlocks to not return null.
HDFS-10793: Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184. HDFS-9700. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets for DataTransferProtocol.
HDFS-10798: Make the threshold of reporting FSNamesystem lock contention configurable.
HDFS-10817: Add Logging for Long-held NN Read Locks.
HDFS-10822: Log DataNodes in the write pipeline. John Zhuge via Lei Xu.
HDFS-10913: Introduce fault injectors to simulate slow mirrors.
HDFS-10917: Collect peer performance statistics on DataNode.
HDFS-10922: Adding additional unit tests for Trash
HDFS-10930: Refactor: Wrap Datanode IO related operations.
HDFS-10934: TestDFSShell#testStat fails intermittently.
HDFS-10956: Remove rename/delete performance penalty when not using snapshots.
HDFS-10959: Update DataNode to use DatasetVolumeChecker.
HDFS-10962: TestRequestHedgingProxyProvider is flaky.
HDFS-10963: Reduce log level when network topology cannot find enough datanodes.
HDFS-10965: Add unit test for HDFS command 'dfsadmin -printTopology'.
HDFS-10966: Enhance Dispatcher logic on deciding when to give up a source DataNode.
HDFS-10984: Expose nntop output as metrics.
HDFS-10987: Make Decommission less expensive when lot of blocks present.
HDFS-11003: Expose XmitsInProgress through DataNodeMXBean.
HDFS-11008: Change unit test for testing parsing "-source" parameter in Balancer CLI.
HDFS-11011: Add unit tests for HDFS command 'dfsadmin -set/clrSpaceQuota'.
HDFS-11015: Enforce timeout in balancer.
HDFS-11018: Incorrect check and message in FsDatasetImpl#invalidate.
HDFS-11031: Add additional unit test for DataNode startup behavior when volumes fail.
HDFS-11047: Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode.
HDFS-11054: Suppress verbose log message in BlockPlacementPolicyDefault.
HDFS-11056: Addendum fix for self-healing branch.
HDFS-11056: Concurrent append and read operations lead to checksum error.
HDFS-11069: Tighten the authorization of datanode RPC.
HDFS-11076: Add unit test for extended Acls.
HDFS-11083: Add unit test for DFSAdmin -report command.
HDFS-11087: NamenodeFsck should check if the output writer is still writable.
HDFS-11112: Journal Nodes should refuse to format non-empty directories.
HDFS-11114: Support for running async disk checks in DataNode.
HDFS-11119: Support for parallel checking of StorageLocations on DataNode startup.
HDFS-11122: TestDFSAdmin#testReportCommand fails due to timed out.
HDFS-11128: CreateEditsLog throws NullPointerException.
HDFS-11148: Update DataNode to use StorageLocationChecker at startup.
HDFS-11149: Fix build break.
HDFS-11149: Support for parallel checking of FsVolumes.
HDFS-11160: VolumeScanner reports write-in-progress replicas as corrupt incorrectly.
HDFS-11180: Intermittent deadlock in NameNode when failover happens.
HDFS-11182: Addendum fix for Fenton to fix a unit test.
HDFS-11182: Update DataNode to use DatasetVolumeChecker.
HDFS-11194: Maintain aggregated peer performance metrics on NameNode.
HDFS-11195: Return error when appending files by webhdfs rest api fails.
HDFS-11197: Listing encryption zones fails when deleting a EZ that is on a snapshotted directory.
HDFS-11209: SNN can't checkpoint when rolling upgrade is not finalized.
HDFS-11224: Lifeline message should be ignored for dead nodes
HDFS-11229: HDFS-11056 failed to close meta file.
HDFS-11240: Remove snapshot version of SDK dependency from Azure Data Lake Store File System.
HDFS-11251: ConcurrentModificationException during DataNode#refreshVolumes .
HDFS-11253: FileInputStream leak on failure path in BlockSender.
HDFS-11258: File mtime change could not save to editlog.
HDFS-11263: ClassCastException when we use Bzipcodec for Fsimage compression.
HDFS-11267: Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage.
HDFS-11274: Datanode should only check the failed volume upon IO errors.
HDFS-11279: Cleanup unused DataNode#checkDiskErrorAsync().
HDFS-11280: Allow WebHDFS to reuse HTTP connections to NN.
HDFS-11282: Document the missing metrics of DataNode Volume IO operations.
HDFS-11292: log lastWrittenTxId etc info in logSyncAll.
HDFS-11299: Support multiple Datanode File IO hooks.
HDFS-11307: The rpc to portmap service for NFS has hardcoded timeout.
HDFS-11337:
HDFS-11339: Support File IO sampling for Datanode IO profiling hooks.
HDFS-11369: Change exception message in StorageLocationChecker.
HDFS-11379: DFSInputStream may infinite loop requesting block locations.
HDFS-11387: Socket reuse address option is not honored in PrivilegedNfsGatewayStarter.
HDFS-11391: Numeric usernames do no work with WebHDFS FS write access.
HDFS-11395: RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode.
HDFS-11403: Zookeper ACLs on NN HA enabled clusters should be handled consistently.
HDFS-11461: DataNode Disk Outlier Detection.
HDFS-11462: Fix occasional BindException in TestNameNodeMetricsLogger.
HDFS-11476: Fix NPE in FsDatasetImpl#checkAndUpdate
HDFS-11477: Simplify file IO profiling configuration.
HDFS-11479: Socket re-use address option should be used in SimpleUdpServer.
HDFS-11508: Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket is in TIME_WAIT state.
HDFS-11517: Expose slow disks via DataNode JMX.
HDFS-11533: reuseAddress option should be used for child channels in Portmap and SimpleTcpServer.
MAPREDUCE-5485: Allow repeating job commit by extending OutputCommitter API
MAPREDUCE-6197: Cache MapOutputLocations in ShuffleHandler.
MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.
MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.
MAPREDUCE-6273: HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state.
MAPREDUCE-6302:
MAPREDUCE-6338: MR AppMaster does not honor ephemeral port range.
MAPREDUCE-6350: JobHistory doesn't support fully-functional search.
MAPREDUCE-6404: Allow AM to specify a port range for starting its webapp.
MAPREDUCE-6436: JobHistory cache issue.
MAPREDUCE-6436: JobHistory cache issue.
MAPREDUCE-6460: TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails.
MAPREDUCE-6485: Create a new task attempt with failed map task priority if in-progress attempts are unassigned.
MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.
MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.
MAPREDUCE-6514: Job hangs as ask is not updated after ramping down of all reducers.
MAPREDUCE-6543: Migrate MR Client test cases part 2.
MAPREDUCE-6549: multibyte delimiters with LineRecordReader cause duplicate records
MAPREDUCE-6566: Add retry support to mapreduce CLI tool.
MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
MAPREDUCE-6579: TestNetworkedJob.testNetworkedJob failed due to YARN-3946.
MAPREDUCE-6593: TestJobHistoryEventHandler.testTimelineEventHandling fails because of NPE.
MAPREDUCE-6616: Fail to create jobhistory file if there are some multibyte characters in the job name.
MAPREDUCE-6618: YarnClientProtocolProvider leaking the YarnClient thread.(xgong).
MAPREDUCE-6621: Memory Leak in JobClient#submitJobInternal().
MAPREDUCE-6625: TestCLI#testGetJob fails occasionally
MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.
MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.
MAPREDUCE-6639: Process hangs in LocatedFileStatusFetcher if FileSystem.get throws.
MAPREDUCE-6672: TestTeraSort fails on Windows.
MAPREDUCE-6684: High contention on scanning of user directory under immediate_done in Job History Server.
MAPREDUCE-6689: MapReduce job can infinitely increase number of reducer resource requests.
MAPREDUCE-6693: ArrayIndexOutOfBoundsException occurs when the length of the job name is equal to mapreduce.jobhistory.jobname.limit.
MAPREDUCE-6701: application master log can not be available when clicking jobhistory's am logs link.
MAPREDUCE-6711: After enabling preemption on cluster, jobs failed without making any attempts made after being preempted.
MAPREDUCE-6724: Single shuffle to memory must not exceed Integer#MAX_VALUE.
MAPREDUCE-6762: ControlledJob#toString failed with NPE when job status is not successfully updated.
MAPREDUCE-6789: Fix TestAMWebApp failure.
MAPREDUCE-6792: Allow user's full principal name as owner of MapReduce staging directory in JobSubmissionFiles#JobStagingDir().
MAPREDUCE-6797: Job history server scans can become blocked on a single, slow entry.
MAPREDUCE-6852: Job#updateStatus() failed with NPE due to race condition.
YARN-857: Localization failures should be available in container diagnostics.
YARN-998: Keep NM resource updated through dynamic resource config for RM/NM restart.
YARN-1376: NM need to notify the log aggregation status to RM through heartbeat.
YARN-1402: Update related Web UI and CLI with exposing client API to check log aggregation status.
YARN-1462: AHS API and other AHS changes to handle tags for completed MR jobs.
YARN-1556: NPE getting application report with a null appId.
YARN-1880: Cleanup TestApplicationClientProtocolOnHA
YARN-1993: Cross-site scripting vulnerability in TextView.java.
YARN-2019: Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore.
YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.
YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.
YARN-2194: Fix bug causing CGroups functionality to fail on RHEL7.
YARN-2392: Add more diags about app retry limits on AM failures.
YARN-2495: Allow admin specify labels from each NM
YARN-2498: Respect labels in preemption policy of capacity scheduler for inter-queue preemption.
YARN-2571:RM to support YARN registry.
YARN-2619: Added NodeManager support for disk io isolation through cgroups.
YARN-2666: TestFairScheduler.testContinuousScheduling fails Intermittently.
YARN-2696: Queue sorting in CapacityScheduler should consider node label.
YARN-2740: Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled.
YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.
YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.
YARN-2868: FairScheduler: Metric for latency to allocate first container for an application.
YARN-2901: Add errors and warning metrics page to RM, NM web UI.
YARN-3021: YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp.
YARN-3026: Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp.
YARN-3102: Decommisioned Nodes not listed in Web UI.
YARN-3102: Decommisioned Nodes not listed in Web UI.
YARN-3110: Few issues in ApplicationHistory web ui.
YARN-3136: Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers.
YARN-3148: Allow CORS related headers to passthrough in WebAppProxyServlet.
YARN-3215: Respect labels in CapacityScheduler when computing headroom.
YARN-3216: Max-AM-Resource-Percentage should respect node labels.
YARN-3225: New parameter of CLI for decommissioning node gracefully in RMAdmin CLI.
YARN-3243: CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits.
YARN-3248: Display count of nodes blacklisted by apps in the web UI.
YARN-3266: RMContext#inactiveNodes should have NodeId as map key.
YARN-3269: Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path.
YARN-3293: Track and display capacity scheduler health metrics in web UI.
YARN-3294: Allow dumping of Capacity Scheduler debug logs via web UI for a fixed time period.
YARN-3305: Normalize AM resource request on app submission.
YARN-3318: Create Initial OrderingPolicy Framework and FifoOrderingPolicy.
YARN-3319: Implement a FairOrderingPolicy.
YARN-3326: Support RESTful API for getLabelsToNodes.
YARN-3343: Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate timeout.
YARN-3345: Add non-exclusive node label API.
YARN-3347: Improve YARN log command to get AMContainer logs as well as running containers logs.
YARN-3348: Add a 'yarn top' tool to help understand cluster usage.
YARN-3354: Add node label expression in ContainerTokenIdentifier to support RM recovery.
YARN-3356: Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label.
YARN-3357: Move TestFifoScheduler to FIFO package.
YARN-3360: Add JMX metrics to TimelineDataManager .
YARN-3361: CapacityScheduler side changes to support non-exclusive node labels.
YARN-3362: Add node label usage in RM CapacityScheduler web UI.
YARN-3365: Enhanced NodeManager to support using the 'tc' tool via container-executor for outbound network traffic control.
YARN-3366: Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers.
YARN-3383: AdminService should use "warn" instead of "info" to log exception when operation fails.
YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.
YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.
YARN-3394: Enrich WebApplication proxy documentation.
YARN-3397: yarn rmadmin should skip -failover.
YARN-3404: Display queue name on application page.
YARN-3406: Display count of running containers in the RM's Web UI.
YARN-3410: YARN admin should be able to remove individual application records from RMStateStore.
YARN-3413: Changed Nodelabel attributes
YARN-3425: NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed.
YARN-3433: Jersey tests failing with Port in Use -again.
YARN-3435: AM container to be allocated Appattempt AM container shown as null.
YARN-3443: Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM.
YARN-3448: Added a rolling time-to-live LevelDB timeline store implementation.
YARN-3451: Display attempt start time and elapsed time on the web UI.
YARN-3453: Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing.
YARN-3459: Fix failiure of TestLog4jWarningErrorMetricsAppender.
YARN-3463: Integrate OrderingPolicy Framework with CapacityScheduler.
YARN-3467: Expose allocatedMB, allocatedVCores, and runningContainers metrics on running Applications in RM Web UI.
YARN-3480: Remove attempts that are beyond max-attempt limit from state store.
YARN-3494: Expose AM resource limit and usage in CS QueueMetrics.
YARN-3503: Expose disk utilization percentage and bad local and log dir counts in NM metrics.
YARN-3505: Node's Log Aggregation Report with SUCCEED should not cached in RMApps.
YARN-3511: Add errors and warnings page to ATS.
YARN-3517: RM web ui for dumping scheduler logs should be for admins only
YARN-3521: Support return structured NodeLabel objects in REST API.
YARN-3530: ATS throws exception on trying to filter results without otherinfo.
YARN-3541: Add version info on timeline service / generic history web UI and REST API.
YARN-3543: ApplicationReport should be able to tell whether the Application is AM managed or not.
YARN-3552: RM Web UI shows -1 running containers for completed apps
YARN-3565: NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String.
YARN-3579: CommonNodeLabelsManager should support NodeLabel instead of string label name when getting node-to-label/label-to-label mappings.
YARN-3580: [JDK8] TestClientRMService.testGetLabelsToNodes fails.
YARN-3581: Deprecate -directlyAccessNodeLabelStore in RMAdminCLI.
YARN-3583: Support of NodeLabel object instead of plain String in YarnClient side.
YARN-3589: RM and AH web UI display DOCTYPE wrongly.
YARN-3591: Resource localization on a bad disk causes subsequent containers failure.
YARN-3593: Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page.
YARN-3600: AM container link is broken
YARN-3604: removeApplication in ZKRMStateStore should also disable watch.
YARN-3632: Ordering policy should be allowed to reorder an application when demand changes.
YARN-3635: Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework.
YARN-3654: ContainerLogsPage web UI should not have meta-refresh.
YARN-3695: ServerProxy
YARN-3695: ServerProxy
YARN-3700: Made generic history service load a number of latest applications according to the parameter or the configuration.
YARN-3707: RM Web UI queue filter doesn't work.
YARN-3717: Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API.
YARN-3727: For better error recovery, check if the directory exists before using it for localization.
YARN-3740: Fixed the typo in the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS.
YARN-3751: Fixed AppInfo to check if used resources are null.
YARN-3766: Fixed the apps table column error of generic history web UI.
YARN-3769: Consider user limit when calculating total pending resource for preemption policy in Capacity Scheduler.
YARN-3785: Support for Resource as an argument during submitApp call in MockRM test class.
YARN-3846: RM Web UI queue filter is not working for sub queue.
YARN-3849:(Too much of preemption activity causing continuos killing of containers across queues) to 2.3-maint.
YARN-3873: PendingApplications in LeafQueue should also use OrderingPolicy.
YARN-3885: ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level.
YARN-3894: RM startup should fail for wrong CS xml NodeLabel capacity configuration.
YARN-3896: RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously.
YARN-3920: FairScheduler container reservation on a node should be configurable to limit it to large containers
YARN-3932: SchedulerApplicationAttempt#getResourceUsageReport and UserInfo should based on total-used-resources.
YARN-3941: Proportional Preemption policy should try to avoid sending duplicate PREEMPT_CONTAINER event to scheduler.
YARN-3946: Update exact reason as to why a submitted app is in ACCEPTED state to app's diagnostic message.
YARN-3948: Display Application Priority in RM Web UI.(Sunil G via rohithsharmaks).
YARN-3961: Expose pending, running and reserved containers of a queue in REST api and yarn top
YARN-3966: Fix excessive loggings in CapacityScheduler.
YARN-3971: Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery.
YARN-3978: Configurably turn off the saving of container info in Generic AHS
YARN-3982: container-executor parsing of container-executor.cfg broken in trunk and branch-2.
YARN-3983: Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic.
YARN-3986: getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface.
YARN-3987: am container complete msg ack to NM once RM receive it.
YARN-4000: RM crashes with NPE if leaf queue becomes parent queue during restart.
YARN-4023: Publish Application Priority to TimelineServer.
YARN-4026: Refactored ContainerAllocator to accept a list of priorites rather than a single priority.
YARN-4082: Container shouldn't be killed when node's label updated.
YARN-4087: Followup fixes after YARN-2019 regarding RM behavior when state-store error occurs.
YARN-4092: Fixed UI redirection to print useful messages when both RMs are in standby mode.
YARN-4101: RM should print alert messages if ZooKeeper and Resourcemanager gets connection issue.
YARN-4108: CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request.
YARN-4113: RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER.
YARN-4140: RM container allocation delayed incase of app submitted to Nodelabel partition.
YARN-4149: yarn logs -am should provide an option to fetch all the log files.
YARN-4155: TestLogAggregationService.testLogAggregationServiceWithInterval failing.
YARN-4162: CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API.
YARN-4215: RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request.
YARN-4225: Add preemption status to yarn queue -status for capacity scheduler.
YARN-4233: YARN Timeline Service plugin: ATS v1.5.
YARN-4245: Generalize config file handling in container-executor.
YARN-4270: Limit application resource reservation on nodes for non-node/rack specific requests
YARN-4285: Display resource usage as percentage of queue and cluster in the RM UI
YARN-4287: Capacity Scheduler: Rack Locality improvement
YARN-4288: Fixed RMProxy to retry on IOException from local host.
YARN-4290: Add -showDetails option to YARN Nodes CLI to print all nodes reports information.
YARN-4304: AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics.
YARN-4309: Add container launch related debug information to container logs when a container fails.
YARN-4313: Race condition in MiniMRYarnCluster when getting history server address.
YARN-4315: NaN in Queue percentage for cluster apps page.
YARN-4329: [YARN-5437] Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler
YARN-4345: yarn rmadmin -updateNodeResource doesn't work
YARN-4347: Resource manager fails with Null pointer exception.
YARN-4349:YARN_APPLICATION call to ATS does not have YARN_APPLICATION_CALLER_CONTEXT.
YARN-4365: FileSystemNodeLabelStore should check for root dir existence on startup.
YARN-4384: updateNodeResource CLI should not accept negative values for resource.
YARN-4390: Do surgical preemption based on reserved container in CapacityScheduler.
YARN-4392: ApplicationCreatedEvent event time resets after RM restart/failover.
YARN-4393: TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently.
YARN-4403:
YARN-4405: Support node label store in non-appendable file system.
YARN-4414: Nodemanager connection errors are retried at multiple levels.
YARN-4414: Nodemanager connection errors are retried at multiple levels.
YARN-4416: Deadlock due to synchronised get Methods in AbstractCSQueue.
YARN-4417: Make RM and Timeline-server REST APIs more consistent.
YARN-4418: AM Resource Limit per partition can be updated to ResourceUsage as well.
YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.
YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.
YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.
YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.
YARN-4439: Clarify NMContainerStatus#toString method.
YARN-4454: NM to nodelabel mapping going wrong after RM restart.
YARN-4477: FairScheduler: Handle condition which can result in an infinite loop in attemptScheduling.
YARN-4479: Change CS LeafQueue pendingOrderingPolicy to hornor recovered apps.
YARN-4497: RM might fail to restart when recovering apps whose attempts are missing.
YARN-4502: Fix two AM containers get allocated when AM restart.
YARN-4522: Queue acl can be checked at app submission.
YARN-4535: Fix checkstyle error in CapacityScheduler.java
YARN-4546: YARN-4546. ResourceManager crash due to scheduling opportunity overflow.
YARN-4546: YARN-4546. ResourceManager crash due to scheduling opportunity overflow.
YARN-4556: TestFifoScheduler.testResourceOverCommit fails.
YARN-4557: Fix improper Queues sorting in PartitionedQueueComparator when accessible-node-labels=*.
YARN-4565:
YARN-4573: Fix test failure in TestRMAppTransitions#testAppRunningKill and testAppKilledKilled.
YARN-4584: RM startup failure when AM attempts greater than max-attempts.
YARN-4591: YARN Web UIs should provide a robots.txt. Junping Du.
YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.
YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.
YARN-4607: Pagination support for AppAttempt page TotalOutstandingResource Requests table.
YARN-4610: Reservations continue looking for one app causes other apps to starve.
YARN-4617: LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy.
YARN-4623: TestSystemMetricsPublisher#testPublishAppAttemptMetricsForUnmanagedAM fails with NPE on branch-2.7.
YARN-4625: Make ApplicationSubmissionContext and ApplicationSubmissionContextInfo more consistent.
YARN-4633: TestRMRestart.testRMRestartAfterPreemption fails intermittently.
YARN-4634: Scheduler UI/Metrics need to consider cases like non-queue label mappings.
YARN-4667: RM Admin CLI for refreshNodesResources throws NPE when nothing is configured.
YARN-4680: Fix TimerTasks leak in ATS V1.5 Writer.
YARN-4696: TimelineClient to add flush operation for deterministic writes
YARN-4697: NM aggregation thread pool is not bound by limits
YARN-4699: Scheduler UI and REST o/p is not in sync when -replaceLabelsOnNode is used to change label of a node.
YARN-4709: NMWebServices produces incorrect JSON for containers.
YARN-4709: NMWebServices produces incorrect JSON for containers.
YARN-4723: NodesListManager$UnknownNodeId ClassCastException.
YARN-4737: Add CSRF filter support in YARN
YARN-4737: Add CSRF filter support in YARN
YARN-4746: yarn web services should convert parse failures of appId, appAttemptId and containerId to 400.
YARN-4767: Network issues can cause persistent RM UI outage.
YARN-4769: Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI.
YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.
YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.
YARN-4811: Generate histograms in ContainerMetrics for actual container resource usage.
YARN-4814: ATS 1.5 timelineclient impl call flush after every event write.
YARN-4815: ATS 1.5 timelineclinet impl try to create attempt directory for every event call.
YARN-4817: TimelineClient ATSv1.5 logging is very noisy.
YARN-4822: Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates.
YARN-4832: NM side resource value should get updated if change applied in RM side.
YARN-4842: Fixed "yarn logs" command to guess
YARN-4844: Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource.
YARN-4846: Fix random failures for TestCapacitySchedulerPreemption #testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers.
YARN-4865: Track Reserved resources in ResourceUsage and QueueCapacities.
YARN-4890: Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate.
YARN-4893: Fix intermittent failures in TestRMAdminService.
YARN-4905: Improved "yarn logs" command-line to optionally show log metadata also.
YARN-4913: Yarn logs should take a -out option to write to a directory.
YARN-4916: TestNMProxy.tesNMProxyRPCRetry fails.
YARN-4920: ATS/NM should support a link to dowload/get the logs in text format.
YARN-4934: Reserved Resource for QueueMetrics needs to be handled correctly in few cases.
YARN-4955: Add retry for SocketTimeoutException in TimelineClient.
YARN-4966: Improve yarn logs to fetch container logs without specifying nodeId.
YARN-4984: LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak.
YARN-4990: Re-direction of a particular log file within in a container in NM UI does not redirect properly.
YARN-5023: TestAMRestart#testShouldNotCountFailureToMaxAttemptRetry random failure.
YARN-5048: DelegationTokenRenewer#skipTokenRenewal may throw NPE.
YARN-5069: TestFifoScheduler.testResourceOverCommit race condition.
YARN-5076: YARN web interfaces lack XFS
YARN-5080: Cannot obtain logs using YARN CLI -am for either KILLED or RUNNING AM.
YARN-5083: YARN CLI for AM logs does not give any error message if entered invalid am value.
YARN-5088: Improve "yarn log" command-line to read the last K bytes for the log files.
YARN-5089: Improve "yarn log" command-line "logFiles" option to support regex.
YARN-5112: Excessive log warnings for directory permission issue on NM recovery.
YARN-5115: Avoid setting CONTENT-DISPOSITION header in the container-logs web-service.
YARN-5121: fix some container-executor portability issues.
YARN-5122: "yarn logs" for running containers should print an explicit footer saying that the log may be incomplete.
YARN-5131: Distributed shell AM fails when extra container arrives during finishing. Wangda Tan via junping_du.
YARN-5141: Get Container logs for the Running application from Yarn Logs CommandLine.
YARN-5190: Registering/unregistering container metrics triggered by ContainerEvent and ContainersMonitorEvent are conflict which cause uncaught exception in ContainerMonitorImpl.
YARN-5191: Renamed the newly added “download=true” option for getting logs via NMWebServices and AHSWebServices to be a better "format" option.
YARN-5199: Close LogReader in in AHSWebServices#getStreamingOutput and FileInputStream in NMWebServices#getLogs.
YARN-5200: Enhanced "yarn logs" to be able to get a list of containers whose logs are aggregated via a "show_container_log_info" option.
YARN-5206: RegistrySecurity includes id:pass in exception text if considered invalid.
YARN-5208: Run TestAMRMClient TestNMClient TestYarnClient TestClientRMTokens TestAMAuthorization tests with hadoop.security.token.service.use_ip enabled.
YARN-5213: Fix a bug in LogCLIHelpers which cause TestLogsCLI#testFetchApplictionLogs fails intermittently.
YARN-5214: Fixed locking in DirectoryCollection to avoid hanging NMs when various code-paths hit slow disks.
YARN-5223: Container line in yarn logs output for a live application should include the hostname for the container.
YARN-5224: Added new web-services /containers/{containerid}/logs & /containers/{containerid}/logs/{filename} and using them in "yarn logs" CLI to get logs of finished containers of a running application.
YARN-5226: Remove AHS enable check from LogsCLI#fetchAMContainerLogs.
YARN-5227: Yarn logs command: no need to specify applicationId when specifying containerId.
YARN-5251: Yarn CLI to obtain App logs for last 'n' bytes fails.
YARN-5266: Wrong exit code while trying to get app logs using regex via CLI.
YARN-5270: Solve miscellaneous issues caused by YARN-4844.
YARN-5294: Pass remote ip address down to YarnAuthorizationProvider.
YARN-5296: NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl.
YARN-5297: Avoid printing a stack trace when recovering an app after the RM restarts.
YARN-5337: Fix Dshell OOM issues.
YARN-5339: Passing file to -out for YARN log CLI doesnt give warning or error code.
YARN-5353: ResourceManager can leak delegation tokens when they are shared across apps.
YARN-5363: For AM containers, or for containers of running-apps, "yarn logs" incorrectly only
YARN-5456: container-executor support for FreeBSD, NetBSD, and others if conf path is absolute.
YARN-5470: Differentiate exactly match with regex in yarn log CLI.
YARN-5476: Not existed application reported as ACCEPTED state by YarnClientImpl.
YARN-5523: Yarn running container log fetching causes OutOfMemoryError
YARN-5524: Yarn live log aggregation does not throw if command line arg is wrong.
YARN-5539: TimelineClient failed to retry on java.net.SocketTimeoutException: Read timed out.
YARN-5549: AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately.
YARN-5551: Ignore file backed pages from memory computation when smaps is enabled.
YARN-5659: getPathFromYarnURL should use standard methods.
YARN-5677: RM should transition to standby when connection is lost for an extended period.
YARN-5704: Provide config knobs to control enabling/disabling new/work in progress features in container-executor.
YARN-5711: Propogate exceptions back to client when using hedging RM failover provider.
YARN-5718: TimelineClient
YARN-5773: RM recovery too slow due to LeafQueue#activateApplications.
YARN-5864: Capacity Scheduler queue priority.
YARN-5873: RM crashes with NPE if generic application history is enabled.
YARN-5915: ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write.
YARN-5923: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled.
YARN-5999: AMRMClientAsync will stop if any exceptions thrown on allocate call.
YARN-6011: Add a new web service to list the files on a container in AHSWebService.
YARN-6029: CapacityScheduler deadlock when ParentQueue#getQueueUserAclInfo is called by one thread and LeafQueue#assignContainers is releasing excessive reserved container by another thread.
YARN-6068: Log aggregation get failed when NM restart even with recovery.
YARN-6081:LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container.
YARN-6099: Improve webservice to list aggregated log files.
YARN-6100: Improve YARN webservice to output aggregated container logs.
YARN-6108: Improve AHS webservice to accept NM address as a parameter to get container logs.
YARN-6113: Re-direct NM Web Service to get container logs for finished applications.
YARN-6123: Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed.
YARN-6126: Obtaining app logs for Running application fails with json parse error.
YARN-6137: Yarn client implicitly invoke ATS client which accesses HDFS.
YARN-6145: Improve log message on fail over.
YARN-6174: Log files pattern should be same for both running and finished container.
YARN-6189: ApplicationNotFoundException due to RM failed over should throw more user friendly exception.
YARN-6228: EntityGroupFSTimelineStore should allow configurable cache stores.
YARN-6270: WebUtils.getRMWebAppURLWithScheme() needs to honor RM HA setting.
YARN-6313: yarn logs cli should provide logs for a completed container even when application is still running.
YARN-6314: Potential infinite redirection on YARN log redirection web service.
YARN-6332: Make RegistrySecurity use short user names for ZK ACLs.
YARN-6367: YARN logs CLI needs alway check containerLogsInfo/containerLogInfo before parse the JSON object from NMWebService.