HADOOP-1540: breaks backward compatibilty.

HADOOP-1540: Support file exclusion list in distcp.

HADOOP-3733: "s3x:" URLs break when Secret Key contains a slash, even if encoded.

HADOOP-5323: Trash documentation should describe its directory structure and configurations.

HADOOP-5470: RunJar.unJar() should write the last modified time found in the jar entry to the uncompressed file.

HADOOP-5732: Add SFTPFileSystem.

HADOOP-7139: Allow appending to existing SequenceFiles.

HADOOP-7256: Resource leak during failure scenario of closing of resources.

HADOOP-7713: dfs -count -q should label output column.

HADOOP-7713: dfs -count -q should label output column.

HADOOP-7817: RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos() .

HADOOP-7930: Kerberos relogin interval in UserGroupInformation should be configurable.

HADOOP-8151: Error handling in snappy decompressor throws invalid exceptions.

HADOOP-8436: NPE In getLocalPathForWrite ( path, conf ) when the required context item is not configured. .

HADOOP-8437: getLocalPathForWrite should throw IOException for invalid paths.

HADOOP-8751: NPE in Token.toString() when Token is constructed using null identifier.

HADOOP-8818: Use equals instead == in MD5MD5CRC32FileChecksum and TFileDumper.

HADOOP-8934: Shell command ls should include sort options.

HADOOP-9121: InodeTree.java has redundant check for vName while throwing exception.

HADOOP-9242: Duplicate surefire plugin config in hadoop-common.

HADOOP-9477: Add posixGroups support for LDAP groups mapping service.

HADOOP-10035: Cleanup TestFilterFileSystem.

HADOOP-10048: LocalDirAllocator should avoid holding locks while accessing the filesystem.

HADOOP-1023: Add more logging in WASB page blob read function.

HADOOP-1026: Logging bug in WASB causing out of index exception.

HADOOP-10300: Allowed deferred sending of call responses.

HADOOP-10365: BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block.

HADOOP-10406: TestIPC.testIpcWithReaderQueuing may fail.

HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.

HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.

HADOOP-1062: added hadoop-azure-datalake and azure-data-lake-store-sdk to all the wasb locations.

HADOOP-1062: added hadoop-azure-datalake and azure-data-lake-store-sdk to all the wasb locations.

HADOOP-10774: Update KerberosTestUtils for hadoop-auth tests when using IBM Java.

HADOOP-10775: Shell operations to fail with meaningful errors on windows if winutils.exe not found.

HADOOP-10823: TestReloadingX509TrustManager is flaky.

HADOOP-10829: Iteration on CredentialProviderFactory.serviceLoader is thread-unsafe.

HADOOP-10965: Print fully qualified path in CommandWithDestination error messages.

HADOOP-10971: Add -C flag to make `hadoop fs -ls` print filenames only.

HADOOP-11031: Design Document for Credential Provider API.

HADOOP-11098: [JDK8] Max Non Heap Memory default changed between JDK7 and 8.

HADOOP-11180: Change log message "token.Token: Cannot find class for token kind kms-dt" to debug.

HADOOP-11212: NetUtils.wrapException to handle SocketException explicitly.

HADOOP-11218: Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.

HADOOP-11252: RPC client does not time out by default.

HADOOP-11262: Enable YARN to use S3A.

HADOOP-11361: Fix a race condition in MetricsSourceAdapter.updateJmxCache.

HADOOP-11404: Clarify the "expected client Kerberos principal is null" authorization message.

HADOOP-11418: Property "io.compression.codec.lzo.class" does not work with other value besides default.

HADOOP-11491: HarFs incorrectly declared as requiring an authority.

HADOOP-11581: Multithreaded correctness Warnings #org.apache.hadoop.fs.shell.Ls.

HADOOP-11613: Remove commons-httpclient dependency from hadoop-azure.

HADOOP-11628: SPNEGO auth does not work with CNAMEs in JDK8. (Daryn Sharp via stevel).

HADOOP-11677: Add cookie flags for logs and static contexts.

HADOOP-11685: StorageException complaining "no lease ID" during HBase distributed log splitting.

HADOOP-11687: Ignore x-* and response headers when copying an Amazon S3 object.

HADOOP-11692: Improve authentication failure WARN message to avoid user confusion.

HADOOP-11713: ViewFileSystem should support snapshot methods.

HADOOP-11717: Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth.

HADOOP-11772: RPC Invoker relies on static ClientCache which has synchronized(this) blocks.

HADOOP-11780: Prevent IPC reader thread death.

HADOOP-11785: Reduce the number of listStatus operation in distcp buildListing.

HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm.

HADOOP-11812: Implement listLocatedStatus for ViewFileSystem to speed up split calculation.

HADOOP-11819: HttpServerFunctionalTest#prepareTestWebapp should create web app directory if it does not exist.

HADOOP-11827: Speed-up distcp buildListing() using threadpool.

HADOOP-11859: PseudoAuthenticationHandler fails with httpcomponents v4.4.

HADOOP-11864: JWTRedirectAuthenticationHandler breaks java8 javadocs.

HADOOP-11876: Refactor code to make it more readable, minor maybePrintStats bug.

HADOOP-11901: BytesWritable fails to support 2G chunks due to integer overflow.

HADOOP-11918: Listing an empty s3a root directory throws FileNotFound.

HADOOP-11924: Tolerate JDK-8047340-related exceptions in Shell#isSetSidAvailable preventing class init.

HADOOP-11932: MetricsSinkAdapter may hang when being stopped.

HADOOP-11960: Enable Azure-Storage Client Side logging.

HADOOP-12001: Fixed LdapGroupsMapping to include configurable Posix UID and GID attributes during the search.

HADOOP-12006: Remove unimplemented option for from document in branch-2.7.

HADOOP-12009: Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus. backport to 2.8.

HADOOP-12017: Hadoop archives command should use configurable replication factor when closing.

HADOOP-12043: Display warning if defaultFs is not set when running fs commands.

HADOOP-12045: Enable LocalFileSystem#setTimes to change atime.

HADOOP-12049: Control http authentication cookie persistence via configuration.

HADOOP-12052: IPC client downgrades all exception types to IOE, breaks callers trying to use them.

HADOOP-12054: RPC client should not retry for InvalidToken exceptions.

HADOOP-12056: Use DirectoryStream in DiskChecker#checkDirs to detect errors when listing a directory.

HADOOP-12058: Fix dead links to DistCp and Hadoop Archives pages.

HADOOP-12059: S3Credentials should support use of CredentialProvider.

HADOOP-12061: Incorrect command in single cluster setup document.

HADOOP-12073: Azure FileSystem PageBlobInputStream does not return -1 on EOF.

HADOOP-12074: in Shell.java#runCommand() rethrow InterruptedException as InterruptedIOException.

HADOOP-12076: Incomplete Cache Mechanism in CredentialProvider API.

HADOOP-12087: [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel).'.

HADOOP-12089: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.

HADOOP-12095: org.apache.hadoop.fs.shell.TestCount fails.

HADOOP-12097: Allow port range to be specified while starting webapp.

HADOOP-12100: ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermission.

HADOOP-12103: Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing.

HADOOP-12107: long running apps may have a huge number of StatisticsData instances under FileSystem.

HADOOP-12119: hadoop fs -expunge does not work for federated namespace.

HADOOP-12158: Improve error message in TestCryptoStreamsWithOpensslAesCtrCryptoCodec when OpenSSL is not installed.

HADOOP-12159: Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces.

HADOOP-12161: Add getStoragePolicy API to the FileSystem interface.

HADOOP-12169: ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse.

HADOOP-12173: NetworkTopology::add calls toString always.

HADOOP-12185: NetworkTopology is not efficient adding/getting/removing nodes.

HADOOP-12186: ActiveStandbyElector shouldn't call monitorLockNodeAsync multiple times.

HADOOP-12189: Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.

HADOOP-12191: Bzip2Factory is not thread safe.

HADOOP-12213: Interrupted exception can occur when Client#stop is called.

HADOOP-12229: Fix inconsistent subsection titles in filesystem.md.

HADOOP-12230: hadoop-project declares duplicate, conflicting curator dependencies.

HADOOP-12232: Upgrade Tomcat dependency to 6.0.44.

HADOOP-12239: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.

HADOOP-12242: Add in-page TOC to filesystem specification pages.

HADOOP-12252: LocalDirAllocator should not throw NPE with empty string configuration.

HADOOP-12258: Need translate java.nio.file.NoSuchFileException to FileNotFoundException to avoid regression.

HADOOP-12259: Utility to Dynamic port allocation.

HADOOP-12269: Update aws-sdk dependency to 1.10.6.

HADOOP-12280: Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded.

HADOOP-12284: UserGroupInformation doAs can throw misleading exception.

HADOOP-12291: Add support for nested groups in LdapGroupsMapping.

HADOOP-12292: Make use of DeleteObjects optional.

HADOOP-12296: when setnetgrent returns 0 in linux, exception should be thrown.

HADOOP-12304: Applications using FileContext fail with the default file system configured to be wasb/s3/etc.

HADOOP-12317: Applications fail on NM restart on some linux distro because NM container recovery declares AM container as LOST.

HADOOP-12324: Better exception reporting in SaslPlainServer. (Mike Yoder via stevel).

HADOOP-12325: RPC Metrics : Add the ability track and log slow RPCs.

HADOOP-12334: Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries.

HADOOP-12345: Pad hostname correctly in CredentialsSys.java.

HADOOP-12346: Increase some default timeouts / retries for S3a connector.

HADOOP-12348: MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.

HADOOP-12350: WASB Logging: Improve WASB Logging around deletes, reads and writes.

HADOOP-12358: Add -safely flag to rm to prompt when deleting many files.

HADOOP-12359: hadoop fs -getmerge doc is wrong.

HADOOP-12366: expose calculated paths.

HADOOP-12374: Description of hdfs expunge command is confusing.

HADOOP-12386: RetryPolicies.RETRY_FOREVER should be able to specify a retry interval.

HADOOP-12404: Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class.

HADOOP-12406: Fixed AbstractMapWritable.readFields to use the thread's ClassLoader to load class instead of System ClassLoader.

HADOOP-12407: Test failing: hadoop.ipc.TestSaslRPC.

HADOOP-12413: AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.

HADOOP-12415: Add io.netty dependency to hadoop-nfs and to hadoop-hdfs, needed to build Bigtop successfuly, see BIGTOP-2049.

HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.

HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.".

HADOOP-12418: TestRPC.testRPCInterruptedSimple fails intermittently.

HADOOP-12423: Handle failure of registering shutdownhook by ShutdownHookManager in static block.

HADOOP-12426: Add Entry point for Kerberos health check.

HADOOP-12437: Allow SecurityUtil to lookup alternate hostnames.

HADOOP-12438: TestLocalFileSystem tests can fail on Windows after HDFS-8767 fix for handling pipe.

HADOOP-12440: TestRPC#testRPCServerShutdown did not produce the desired thread states before shutting down.

HADOOP-12441: Fixed kill-command behavior to work correctly across OSes by using bash shell built-in.

HADOOP-12442: Display help if the command option to 'hdfs dfs' is not valid.

HADOOP-12444: Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel.

HADOOP-12449: TestDNS and TestNetUtils failing if no network.

HADOOP-12453: Support decoding KMS Delegation Token with its own Identifier.

HADOOP-12457: [JDK8] Fix a failure of compiling common by javadoc.

HADOOP-12463: Fix TestShell.testGetSignalKillCommand failure on windows.

HADOOP-12464: Interrupted client may try to fail-over and retry.

HADOOP-12465: Incorrect javadoc in WritableUtils.java.

HADOOP-12469: distcp should not ignore the ignoreFailures option.

HADOOP-12472: Make GenericTestUtils.assertExceptionContains robust.

HADOOP-12478: Shell.getWinUtilsPath() has been renamed Shell.getWinutilsPath().

HADOOP-12481: JWTRedirectAuthenticationHandler doesn't Retain Original Query String.

HADOOP-12482: Race condition in JMX cache update.

HADOOP-12483: Maintain wrapped SASL ordering for postponed IPC responses.

HADOOP-12484: Single File Rename Throws Incorrectly In Potential Race Condition Scenarios.

HADOOP-12508: delete fails with exception when lease is held on blob.

HADOOP-12520: Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests.

HADOOP-12526: there are duplicate dependency definitions in pom's.

HADOOP-12533: Introduce FileNotFoundException in WASB for read and seek API.

HADOOP-12535: Run FileSystem contract tests with hadoop-azure.

HADOOP-12537: S3A to support Amazon STS temporary credentials.

HADOOP-12540: TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown.

HADOOP-12542: TestDNS fails on Windows after HADOOP-12437.

HADOOP-12545: Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp.

HADOOP-12548: Read s3a creds from a Credential Provider.

HADOOP-12551: Introduce FileNotFoundException for WASB FileSystem API.

HADOOP-12555: WASB to read credentials from a credential provider.

HADOOP-12559: KMS connection failures should trigger TGT renewal.

HADOOP-12564: Upgrade JUnit3 TestCase to JUnit 4 in org.apache.hadoop.io package.

HADOOP-12565: Replace DSA with RSA for SSH key type in SingleCluster.md.

HADOOP-12568: Update core-default.xml to describe posixGroups support.

HADOOP-12570: HDFS Secure Mode Documentation updates.

HADOOP-12573: TestRPC.testClientBackOff failing.

HADOOP-12577: Bump up commons-collections version to 3.2.2 to address a security flaw.

HADOOP-12588: Fix intermittent test failure of TestGangliaMetrics.

HADOOP-12589: Fix intermittent test failure of TestCopyPreserveFlag.

HADOOP-12590: TestCompressorDecompressor failing without stack traces.

HADOOP-12602: TestMetricsSystemImpl#testQSize occasionally fails.

HADOOP-12603: TestSymlinkLocalFSFileContext#testSetTimesSymlinkToDir occasionally fail.

HADOOP-12604: Exception may be swallowed in KMSClientProvider.

HADOOP-12605: Fix intermittent failure of TestIPC.testIpcWithReaderQueuingi.

HADOOP-12608: Fix exception message in WASB when connecting with anonymous credential.

HADOOP-12609: Fix intermittent failure of TestDecayRpcScheduler.

HADOOP-12613: TestFind.processArguments occasionally fails.

HADOOP-12617: SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal.

HADOOP-12622: Improve the loggings in RetryPolicies and RetryInvocationHandler.

HADOOP-12634: Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy.

HADOOP-12635: Adding Append API support for WASB.

HADOOP-12636: Prevent ServiceLoader failure init for unused FileSystems.

HADOOP-12653: Use SO_REUSEADDR to avoid getting "Address already in use" when using kerberos and attempting to bind to any port on the local IP address.

HADOOP-12659: Incorrect usage of config parameters in token manager of KMS.

HADOOP-12663: Remove Hard-Coded Values From FileSystem.java.

HADOOP-12666: Support Microsoft Azure Data Lake - as a file system in Hadoop.

HADOOP-12667: s3a to support createNonRecursive API.

HADOOP-12668: Support excluding weak Ciphers in HttpServer2 through ssl- server.conf.

HADOOP-12672: RPC timeout should not override IPC ping interval.

HADOOP-12675: Fix description about retention period in usage of expunge command.

HADOOP-12678: Handle empty rename pending metadata file during atomic rename in redo path.

HADOOP-12682: Fix TestKMS#testKMSRestart* failure.

HADOOP-12688: Fix deadlinks in Compatibility.md.

HADOOP-12689: S3 filesystem operations stopped working correctly.

HADOOP-12691: Add CSRF Filter for REST APIs to Hadoop Common.

HADOOP-12691: Move files to correct location.

HADOOP-12696: Add tests for S3Filesystem Contract.

HADOOP-12699: TestKMS#testKMSProvider intermittently fails during 'test rollover draining'.

HADOOP-12700: Remove unused import in TestCompressorDecompressor.java.

HADOOP-12706: TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally.

HADOOP-12711: Remove dependency on commons-httpclient for ServletUtil.

HADOOP-12715: TestValueQueue#testgetAtMostPolicyALL fails intermittently.

HADOOP-12716: KerberosAuthenticator#doSpnegoSequence use incorrect class to determine isKeyTab in JDK8.

HADOOP-12717: NPE when trying to rename a directory in Windows Azure Storage FileSystem.

HADOOP-12723: S3A: Add ability to plug in any AWSCredentialsProvider.

HADOOP-12731: Remove useless boxing/unboxing code.

HADOOP-12736: TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out.

HADOOP-12738: Create unit test to automatically compare Common related classes and core-default.xml.

HADOOP-12751: While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple. (Bolke de Bruin via stevel).

HADOOP-12752: Improve diagnostics/use of envvar/sysprop credential propagation.

HADOOP-12758: Extend CSRF Filter with UserAgent Checks.

HADOOP-12761: incremental maven build is not really incremental.

HADOOP-12765: HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections.

HADOOP-12767: Update apache httpclient version to 4.5.2; httpcore to 4.4.4.

HADOOP-12772: NetworkTopologyWithNodeGroup.getNodeGroup() can loop infinitely for invalid 'loc' values.

HADOOP-12773: HBase classes fail to load with client/job classloader enabled.

HADOOP-12774: s3a should use UGI.getCurrentUser.getShortname() for username.

HADOOP-12776: Remove getaclstatus call for non-acl commands in getfacl.

HADOOP-12780: During WASB atomic rename handle crash when one directory has been renamed but not file under it.

HADOOP-12782: Faster LDAP group name resolution with ActiveDirectory.

HADOOP-12786: "hadoop key" command usage is not documented.

HADOOP-12787: KMS SPNEGO sequence does not work with WEBHDFS.

HADOOP-12788: OpensslAesCtrCryptoCodec should log which random number generator is used.

HADOOP-12789: log classpath of ApplicationClassLoader at INFO level.

HADOOP-12792: TestUserGroupInformation#testGetServerSideGroups fails in chroot.

HADOOP-12793: Write a new group mapping service guide.

HADOOP-12794: Support additional compression levels for GzipCodec.

HADOOP-12795: KMS does not log detailed stack trace for unexpected errors.

HADOOP-12800: Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs.

HADOOP-12801: Suppress obsolete S3FileSystem tests.

HADOOP-12804: Read Proxy Password from Credential Providers in S3 FileSystem.

HADOOP-12805: Annotate CanUnbuffer with @InterfaceAudience.Public.

HADOOP-12807: S3AFileSystem should read AWS credentials from environment variables.

HADOOP-12810: FileSystem#listLocatedStatus causes unnecessary RPC calls.

HADOOP-12817: Enable TLS v1.1 and 1.2.

HADOOP-12825: Log slow name resolutions.

HADOOP-12828: Print user when services are started.

HADOOP-12829: StatisticsDataReferenceCleaner swallows interrupt exceptions.

HADOOP-12831: LocalFS/FSOutputSummer NPEs in constructor if bytes per checksum set to 0.

HADOOP-12844: Recover when S3A fails on IOException in read().

HADOOP-12846: Credential Provider Recursive Dependencies.

HADOOP-12847: hadoop daemonlog should support https and SPNEGO for Kerberized cluster.

HADOOP-12849: TestSymlinkLocalFSFileSystem fails intermittently.

HADOOP-12851: S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders.

HADOOP-12853: Change WASB documentation regarding page blob support.

HADOOP-12870: Fix typo admininistration in CommandsManual.md.

HADOOP-12871: Fix dead link to NativeLibraries.html in CommandsManual.md.

HADOOP-12872: Fix formatting in ServiceLevelAuth.md.

HADOOP-12875: [Azure Data Lake] Support for contract test and unit test cases.

HADOOP-12878: KMS SPNEGO sequence does not work with WEBHDFS.

HADOOP-12886: Exclude weak ciphers in SSLFactory through ssl-server.xml.

HADOOP-12891: S3AFileSystem should configure Multipart Copy threshold and chunk size.

HADOOP-12893: Update LICENSE.txt and NOTICE.txt.

HADOOP-12893: Update LICENSE.txt and NOTICE.txt.

HADOOP-12893: Update LICENSE.txt and NOTICE.txt".

HADOOP-12895: SSLFactory#createSSLSocketFactory exception message is wrong.

HADOOP-12901: Add warning log when KMSClientProvider cannot create a connection to the KMS server.

HADOOP-12902: JavaDocs for SignerSecretProvider are out-of-date in AuthenticationFilter.

HADOOP-12903: IPC Server should allow suppressing exception logging by type,.

HADOOP-12903: IPC Server should allow suppressing exception logging by type, not log 'server too busy' messages.

HADOOP-12906: AuthenticatedURL should convert a 404/Not Found into an FileNotFoundException.

HADOOP-12909: Change ipc.Client to support asynchronous calls.

HADOOP-12916: Allow RPC scheduler/callqueue backoff using response times.

HADOOP-12926: lz4.c does not detect 64-bit mode properly.

HADOOP-12929: JWTRedirectAuthenticationHandler must accommodate null expiration time.

HADOOP-12943: Add -w -r options in dfs -test command.

HADOOP-12947: Update documentation Hadoop Groups Mapping to add static group mapping, negative cache.

HADOOP-12950: ShutdownHookManager should have a timeout for each of the Registered shutdown hook.

HADOOP-12954: Add a way to change hadoop.security.token.service.use_ip.

HADOOP-12957: Limit the number of outstanding async calls.

HADOOP-12958: PhantomReference for filesystem statistics can trigger OOM.

HADOOP-12962: KMS key names are incorrectly encoded when creating key.

HADOOP-12963: Allow using path style addressing for accessing the s3 endpoint.

HADOOP-12964: Http server vulnerable to clickjacking.

HADOOP-12973: Make DU pluggable.

HADOOP-12975: Add jitter to CachingGetSpaceUsed's thread.

HADOOP-12977: s3a to handle delete("/", true) robustly.

HADOOP-12982: Document missing S3A and S3 properties.

HADOOP-12984: Add GenericTestUtils.getTestDir method and use it for emporary directory in tests.

HADOOP-12985: Support MetricsSource interface for DecayRpcScheduler Metrics.

HADOOP-12989: Some tests in org.apache.hadoop.fs.shell.find occasionally time out.

HADOOP-12993: Change ShutdownHookManger complete shutdown log from INFO to DEBUG.

HADOOP-12994: Specify PositionedReadable, add contract tests, fix problems.

HADOOP-13008: Add XFS Filter for UIs to Hadoop Common.

HADOOP-13011: Clearly Document the Password Details for Keystore-based Credential Providers.

HADOOP-13018: Make Kdiag check whether hadoop.token.files points to existent and valid files.

HADOOP-13024: Distcp with -delete feature on raw data not implemented.

HADOOP-13026: Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator.

HADOOP-13028: add low level counter metrics for S3A; use in read performance tests.

HADOOP-13030: Handle special characters in passwords in KMS startup script.

HADOOP-13034: Log message about input options in distcp lacks some items.

HADOOP-13037: Refactor Azure Data Lake Store as an independent FileSystem.

HADOOP-13039: Add documentation for configuration property ipc.maximum.data.length.

HADOOP-13042: Restore lost leveldbjni LICENSE and NOTICE changes.

HADOOP-13047: S3a Forward seek in stream length to be configurable.

HADOOP-13052: ChecksumFileSystem mishandles crc file permissions.

HADOOP-13058: S3A FS fails during init against a read-only FS if multipart purge.

HADOOP-13065: Add a new interface for retrieving FS and FC Statistics.

HADOOP-13072: WindowsGetSpaceUsed constructor should be public.

HADOOP-13079: Add -q option to Ls to print ? instead of non-printable characters.

HADOOP-13084: Fix ASF License warnings in branch-2.7.

HADOOP-13098: Dynamic LogLevel setting page should accept case-insensitive log level string.

HADOOP-13103: Group resolution from LDAP may fail on javax.naming.ServiceUnavailableException.

HADOOP-13105: Support timeouts in LDAP queries in LdapGroupsMapping.

HADOOP-13113: Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel.

HADOOP-13116: Jets3tNativeS3FileSystemContractTest does not run.

HADOOP-13122: Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.

HADOOP-13130: s3a failures can surface as RTEs, not IOEs.

HADOOP-13131: Add tests to verify that S3A supports SSE-S3 encryption.

HADOOP-13138: Unable to append to a SequenceFile with Compression.NONE.

HADOOP-13139:: S3a to use thread pool that blocks clients.

HADOOP-13140: FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes.

HADOOP-13145: In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.

HADOOP-13146: Refactor RetryInvocationHandler.

HADOOP-13155: Implement TokenRenewer to renew and cancel delegation tokens in KMS.

HADOOP-13158: S3AFileSystem#toString might throw NullPointerException due to null cannedACL.

HADOOP-13159: Fix potential NPE in Metrics2 source for DecayRpcScheduler.

HADOOP-13162: Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs.

HADOOP-13163: Reuse pre-computed filestatus in Distcp-CopyMapper.

HADOOP-13164: Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories.

HADOOP-13168: Support Future.get with timeout in ipc async calls.

HADOOP-13169: Randomize file list in SimpleCopyListing.

HADOOP-13171: Add StorageStatistics to S3A; instrument some more operations.

HADOOP-13179: GenericOptionsParser is not thread-safe because commons-cli OptionBuilder is not thread-safe.

HADOOP-13183: S3A proxy tests fail after httpclient/httpcore upgrade.

HADOOP-13188: S3A file-create should throw error rather than overwrite directories.

HADOOP-13192: org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly.

HADOOP-13197: Add non-decayed call metrics for DecayRpcScheduler.

HADOOP-13202: Avoid possible overflow in org.apache.hadoop.util.bloom.BloomFilter#getNBytes.

HADOOP-13203: S3a: Consider reducing the number of connection aborts by setting correct length in s3 request.

HADOOP-13204: Add support for SSE-KMS and SSE-C in s3a filesystem.

HADOOP-13207: Specify FileSystem listStatus, listFiles and RemoteIterator.

HADOOP-13208: S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories.

HADOOP-13212: Provide an option to set the socket buffers in S3AFileSystem.

HADOOP-13213: Small Documentation bug with AuthenticatedURL in hadoop-auth.

HADOOP-13237: : s3a initialization against public bucket fails if caller lacks any credentials.

HADOOP-13239: Deprecate s3:// in branch-2.

HADOOP-13241: document s3a better.

HADOOP-13242: Authenticate to Azure Data Lake using client ID and keys.

HADOOP-13244: o.a.h.ipc.Server#Server should honor handlerCount when queueSizePerHandler is specified in consturctor.

HADOOP-13249: RetryInvocationHandler need wrap InterruptedException in IOException when call Thread.sleep.

HADOOP-13251: Authenticate with Kerberos credentials when renewing KMS delegation token.

HADOOP-13252: Tune S3A provider plugin mechanism.

HADOOP-13254: Create framework for configurable disk checkers.

HADOOP-13255: KMSClientProvider should check and renew tgt when doing delegation token operations.

HADOOP-13257: Improve Azure Data Lake contract tests.

HADOOP-13263: Reload cached groups in background after expiry.

HADOOP-13270: BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks.

HADOOP-13280: FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps.

HADOOP-13283: Support reset operation for new global storage statistics and per FS storage stats.

HADOOP-13285: DecayRpcScheduler MXBean should only report decayed CallVolumeSummary.

HADOOP-13287: TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +.

HADOOP-13288: Guard null stats key in FileSystemStorageStatistics.

HADOOP-13290: Appropriate use of generics in FairCallQueue.

HADOOP-13291: Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented.

HADOOP-13299: JMXJsonServlet is vulnerable to TRACE.

HADOOP-13305: Define common statistics names across schemes.

HADOOP-13309: Document S3A known limitations in file ownership and permission model.

HADOOP-13315: FileContext#umask is not initialized properly.

HADOOP-13316: Enforce Kerberos authentication for required ops in DelegationTokenAuthenticator.

HADOOP-13320: Fix arguments check in documentation for WordCount v2.0.

HADOOP-13323: Downgrade stack trace on FS load from Warn to debug.

HADOOP-13324: s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints).

HADOOP-13336: S3A to support per-bucket configuration.

HADOOP-13350: Additional fix to LICENSE and NOTICE.

HADOOP-13351: TestDFSClientSocketSize buffer size tests are flaky.

HADOOP-13352: Make X-FRAME-OPTIONS configurable in HttpServer2.

HADOOP-13353: LdapGroupsMapping getPassward shouldn't return null when IOException throws.

HADOOP-13354: Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients.

HADOOP-13368: DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation.

HADOOP-13381: KMS clients should use KMS Delegation Tokens from current UGI.

HADOOP-13382: Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Ported to Hortonworks HDP-2.5 by Matt Foley.

HADOOP-13387: users always get told off for using S3 —even when not using it.

HADOOP-13389: TestS3ATemporaryCredentials.testSTS error when using IAM credentials.

HADOOP-13392: [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured.

HADOOP-13393: Omit unsupported fs.defaultFS setting in ADLS documentation.

HADOOP-13403: AzureNativeFileSystem rename/delete performance improvements.

HADOOP-13405: doc for fs.s3a.acl.default indicates incorrect values.

HADOOP-13406: S3AFileSystem: Consider reusing filestatus in delete() and mkdirs().

HADOOP-13422: ZKDelegationTokenSecretManager JaasConfig does not work well with other ZK users in process.

HADOOP-13433: Race in UGI.reloginFromKeytab.

HADOOP-13434: Add quoting to Shell class.

HADOOP-13443: KMS should check the type of underlying keyprovider of KeyProviderExtension before falling back to default.

HADOOP-13446: Support running isolated unit tests separate from AWS integration tests.

HADOOP-13447: Refactor S3AFileSystem to support introduction of separate metadata repository and tests.

HADOOP-13458: LoadBalancingKMSClientProvider#doOp should log IOException stacktrace.

HADOOP-13459: hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time.

HADOOP-13461: NPE in KeyProvider.rollNewVersion.

HADOOP-13466: Add an AutoCloseableLock class.

HADOOP-13467: Shell#getSignalKillCommand should use the bash builtin on Linux.

HADOOP-13470: GenericTestUtils$LogCapturer is flaky.

HADOOP-13494: ReconfigurableBase can log sensitive information.

HADOOP-13502: Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests.

HADOOP-13512: ReloadingX509TrustManager should keep reloading in case of exception.

HADOOP-13513: Java 1.7 support for org.apache.hadoop.fs.azure testcases.

HADOOP-13535: Add jetty6 acceptor startup issue workaround to branch-2.

HADOOP-13540: improve section on troubleshooting s3a auth problems.

HADOOP-13541: explicitly declare the Joda time version S3A depends on.

HADOOP-13546: Override equals and hashCode to avoid connection leakage.

HADOOP-13552: RetryInvocationHandler logs all remote exceptions.

HADOOP-13552: RetryInvocationHandler logs all remote exceptions.

HADOOP-13558: UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket.

HADOOP-13560: S3ABlockOutputStream to support huge (many GB) file writes.

HADOOP-13565: KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request.

HADOOP-13590: Retry until TGT expires even if the UGI renewal thread encountered exception.

HADOOP-13599: s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown.

HADOOP-13614: Purge some superfluous/obsolete S3 FS tests that are slowing test runs down.

HADOOP-13638: KMS should set UGI's Configuration object properly.

HADOOP-13674: S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.

HADOOP-13675: Bug in return value for delete() calls in WASB.

HADOOP-13680: fs.s3a.readahead.range to use getLongBytes.

HADOOP-13686: Adding additional unit test for Trash (I).

HADOOP-13692: hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.

HADOOP-13707: If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed.

HADOOP-13710: Supress CachingGetSpaceUsed from logging interrupted exception stacktrace.

HADOOP-13716: Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup.

HADOOP-13727: S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider.

HADOOP-13735: ITestS3AFileContextStatistics.testStatistics() failing.

HADOOP-13737: Cleanup DiskChecker interface.

HADOOP-13738: DiskChecker should perform some disk IO.

HADOOP-13742: Expose NumOpenConnectionsPerUser as a metric. Brahma Reddy Battula.

HADOOP-13749: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.

HADOOP-13782: Make MutableRates metrics thread-local write, aggregate-on- read.

HADOOP-13792: Stackoverflow for schemeless defaultFS with trailing slash.

HADOOP-13801: regression: ITestS3AMiniYarnCluster failing.

HADOOP-13804: MutableStat mean loses accuracy if add(long, long) is used.

HADOOP-13822: Use GlobalStorageStatistics.INSTANCE.reset() at FileSystem#clearStatistics(). Contribute by Brahma Reddy Battula.

HADOOP-13823: s3a rename: fail if dest file exists.

HADOOP-13824: FsShell can suppress the real error if no error message is present.

HADOOP-13826: S3A Deadlock in multipart copy due to thread pool limits.

HADOOP-13830: Intermittent failure of ITestS3NContractRootDir.testRecursiveRootListing "Can not create a Path from an empty string". .

HADOOP-13831: Correct check for error code to detect Azure Storage Throttling and provide retries.

HADOOP-13838: KMSTokenRenewer should close providers.

HADOOP-13847: KMSWebApp should close KeyProviderCryptoExtension.

HADOOP-13855: Fix a couple of the s3a statistic names to be consistent with the rest.

HADOOP-13857: S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs.

HADOOP-13863: Azure: Add a new SAS key mode for WASB.

HADOOP-13871: ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful.

HADOOP-13890: Maintain HTTP/host as SPNEGO SPN support and fix KerberosName parsing.

HADOOP-13928: TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime error.

HADOOP-13929: ADLS connector should not check in contract-test- options.xml.

HADOOP-13930: Azure: Add Authorization support to WASB.

HADOOP-13945: Azure: Add Kerberos and Delegation token support to WASB client.

HADOOP-13956: Read ADLS credentials from Credential Provider.

HADOOP-13962: Update ADLS SDK to 2.1.4.

HADOOP-13988: KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser.

HADOOP-14017: User friendly name for ADLS user and group.

HADOOP-14019: Fix some typos in the s3a docs.

HADOOP-14028: S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.

HADOOP-14029: Fix KMSClientProvider for non-secure proxyuser use case.

HADOOP-14038: Rename ADLS credential properties.

HADOOP-14048: REDO operation of WASB#AtomicRename should create placeholder blob for destination folder.

HADOOP-14049: Honour AclBit flag associated to file/folder permission for Azure datalake account.

HADOOP-14055: SwiftRestClient includes pass length in exception if auth fails.

HADOOP-14081: S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock).

HADOOP-14091: AbstractFileSystem implementaion for 'wasbs' scheme.

HADOOP-14100: Upgrade Jsch jar to latest version to fix vulnerability in old versions.

HADOOP-14102: Relax error message assertion in S3A test ITestS3AEncryptionSSEC.

HADOOP-14113: Review ADL Docs.

HADOOP-14123: Remove misplaced ADL service provider config file for FileSystem.

HADOOP-14138: Remove S3A ref from META-INF service discovery, rely on existing core-default entry.

HADOOP-14150: Implement getHomeDirectory() method in NativeAzureFileSystem.

HADOOP-14153: ADL module has messed doc structure.

HADOOP-14170: FileSystemContractBaseTest is not cleaning up test directory clearly.

HADOOP-14173: Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE.

HADOOP-14174: Set default ADLS access token provider type to ClientCredential.

HADOOP-14195: CredentialProviderFactory$getProviders is not thread-safe.

HADOOP-14196: Azure Data Lake doc is missing required config entry.

HADOOP-14197: Fix ADLS doc for credential provider.

HADOOP-14205: No FileSystem for scheme: adl.

HADOOP-14214: DomainSocketWatcher::add()/delete() should not self interrupt while looping await().

HADOOP-14230: TestAdlFileSystemContractLive fails to clean up.

HADOOP-14241: Add ADLS sensitive config keys to default list.

HADOOP-14274: Azure: Simplify Ranger-WASB policy model.

HADOOP-14321: explicitly exclude s3a root dir ITests from parallel runs.

HADOOP-14349: Rename ADLS CONTRACT_ENABLE_KEY.

HADOOP-14351: Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache.

HADOOP-14361: Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently.

HADOOP-14416: Path starting with 'wasb///' not resolved correctly while authorizing with WASB-Ranger.

HADOOP-14442: Owner support for ranger-wasb integration.

HADOOP-14443: Azure: Support retry and client side failover for authorization, SASKey and delegation token generation.3.

HADOOP-14465: LdapGroupsMapping - support user and group search base.

HADOOP-14472: Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently.

HADOOP-14478: Optimize NativeAzureFsInputStream for positional reads.

HADOOP-14500: Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails.

HADOOP-14512: WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename.

HADOOP-14535: wasb: implement high-performance random access and seek of block blobs.

HADOOP-14536: Update azure-storage sdk to version 5.3.0.

HADOOP-14546: Azure: Concurrent I/O does not work when secure.mode is enabled.

HADOOP-14547: [WASB] the configured retry policy is not used for all storage operations.'.

HADOOP-14581: Restrict setOwner to list of user when security is enabled in wasb.

HADOOP-14640: Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation.

HADOOP-14642: wasb: add support for caching Authorization and SASKeys.

HADOOP-14658: branch-2 compilation is broken in hadoop-azure.

HADOOP-14680: Azure: IndexOutOfBoundsException in BlockBlobInputStream.'.

HADOOP-14722: Azure: BlockBlobInputStream position incorrect after seek.

HDFS-1477: Support reconfiguring dfs.heartbeat.interval and dfs.namenode.heartbeat.recheck-interval without NN restart.

HDFS-2043: TestHFlush failing intermittently.

HDFS-2580: NameNode#main(...) can make use of GenericOptionsParser.

HDFS-27: HDFS CLI with --config set to default config complains log file not found error.

HDFS-3677: dfs.namenode.edits.dir.required is missing from hdfs- default.xml.

HDFS-3918: EditLogTailer shouldn't log WARN when other node is in standby mode.

HDFS-4015: Safemode should count and report orphaned blocks.

HDFS-4176: EditLogTailer should call rollEdits with a timeout.

HDFS-4185: Add a metric for number of active leases.

HDFS-4366: Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks.

HDFS-4396: Add START_MSG/SHUTDOWN_MSG for ZKFC.

HDFS-4681: TestBlocksWithNotEnoughRacks#testCorruptBlockRereplicatedAcrossRacks fails using IBM java.

HDFS-4937: ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom.

HDFS-4946: Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable.

HDFS-5640: Add snapshot methods to FileContext.

HDFS-6054: MiniQJMHACluster should not use static port to avoid binding failure in unit test.

HDFS-6101: TestReplaceDatanodeOnFailure fails occasionally.

HDFS-6184: Capture NN's thread dump when it fails over.

HDFS-6407: Add sorting and pagination in the datanode tab of the NN Web UI.

HDFS-6481: DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs.

HDFS-6533: TestBPOfferService#testBasicFunctionalitytest fails intermittently.

HDFS-6581: Support for writing to single replica in RAM. Applied HDFS-6581.merge.15.patch to Champlain. This is a consolidated patch containing all feature changes and was applied to trunk.

HDFS-6666: Abort NameNode and DataNode startup if security is enabled but block access token is not enabled.

HDFS-6763: Initialize file system-wide quota once on transitioning to active.

HDFS-6832: Fix the usage of 'hdfs namenode' command.

HDFS-6860: BlockStateChange logs are too noisy.

HDFS-6874: Add GETFILEBLOCKLOCATIONS operation to HttpFS.

HDFS-6945: BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed.

HDFS-7163: WebHdfsFileSystem should retry reads according to the configured retry policy.

HDFS-7166: SbNN Web UI shows #Under replicated blocks and #pending deletion blocks.

HDFS-7284: Add more debug info to BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas.

HDFS-7314: When the DFSClient lease cannot be renewed, abort open-for- write files rather than the entire DFSClient.

HDFS-7390: Provide JMX metrics per storage type.

HDFS-7452: skip StandbyException log for getCorruptFiles() .

HDFS-7483: Display information per tier on the Namenode UI.

HDFS-7597: DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping.

HDFS-7608: hdfs dfsclient newConnectedPeer has no write timeout.

HDFS-7609: Avoid retry cache collision when Standby NameNode loading edits.

HDFS-7645: Fix CHANGES.txt.

HDFS-7645: Rolling upgrade is restoring blocks from trash multiple times.

HDFS-7701: Support reporting per storage type quota and usage with hadoop/hdfs shell.

HDFS-7725: Incorrect "nodes in service" metrics caused all writes to fail.

HDFS-7758: Retire FsDatasetSpi#getVolumes() and use FsDatasetSpi#getVolumeRefs() instead.

HDFS-7833: DataNode reconfiguration does not recalculate valid volumes required, based on configured failed volumes tolerated.

HDFS-7858: Improve HA Namenode Failover detection on the client.

HDFS-7890: Improve information on Top users for metrics in RollingWindowsManager and lower log level.

HDFS-7923: The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages.

HDFS-7928: Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy.

HDFS-7933: fsck should also report decommissioning replicas.

HDFS-7964: Add support for async edit logging.

HDFS-7978: Add LOG.isDebugEnabled() guard for some LOG.debug(..).

HDFS-7990: IBR delete ack should not be delayed.

HDFS-8008: Support client-side back off when the datanodes are congested.

HDFS-8008: Support client-side back off when the datanodes are congested.

HDFS-8009: Signal congestion on the DataNode.

HDFS-8009: Signal congestion on the DataNode.

HDFS-8046: Allow better control of getContentSummary.

HDFS-8055: NullPointerException when topology script is missing.

HDFS-8071:.

HDFS-8099: Change "DFSInputStream has been closed already" message to debug log level.

HDFS-8101: DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS classes at runtime.

HDFS-8113: Add check for null BlockCollection pointers in BlockInfoContiguous structures.

HDFS-8131: Implement a space balanced block placement policy.

HDFS-8144: Split TestLazyPersistFiles into multiple tests.

HDFS-8152: Refactoring of lazy persist storage cases.

HDFS-8155: Support OAuth2 in WebHDFS.

HDFS-8180: AbstractFileSystem Implementation for WebHdfs.

HDFS-8205: CommandFormat#parse() should not parse option as value of option.

HDFS-8209: Support different number of datanode directories in MiniDFSCluster.

HDFS-8211: DataNode UUID is always null in the JMX counter.

HDFS-8219: setStoragePolicy with folder behavior is different after cluster restart.

HDFS-8224: Schedule a block for scanning if its metadata file is corrupt.

HDFS-8229: LAZY_PERSIST file gets deleted after NameNode restart.

HDFS-8232: Missing datanode counters when using Metrics2 sink interface.

HDFS-8247: TestDiskspaceQuotaUpdate#testAppendOverTypeQuota is failing.

HDFS-8276: LazyPersistFileScrubber should be disabled if scrubber interval configured zero.

HDFS-8278: When computing max-size-to-move in Balancer, count only the storage with remaining >= default block size. (szetszwo).

HDFS-8307: Spurious DNS Queries from hdfs shell.

HDFS-8309: Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows.

HDFS-8310: Fix TestCLI.testAll "help: help for find" on Windows.

HDFS-8311: DataStreamer.transfer() should timeout the socket InputStream.

HDFS-8312: Added permission check for moving file to Trash.

HDFS-8337: Accessing httpfs via webhdfs doesn't work from a jar with kerberos.

HDFS-8345: Storage policy APIs must be exposed via the FileSystem interface.

HDFS-8384: Allow NN to startup if there are files having a lease but are not under construction.

HDFS-8431: hdfs crypto class not found in Windows.

HDFS-8432: Introduce a minimum compatible layout version to allow downgrade in more rolling upgrade use cases.

HDFS-8435: Support CreateFlag in WebHDFS.

HDFS-8492: DN should notify NN when client requests a missing block.

HDFS-8539: Hdfs doesnt have class 'debug' in windows.

HDFS-8542: WebHDFS getHomeDirectory behavior does not match specification.

HDFS-8546: Use try with resources in DataStorage and Storage.

HDFS-8548: Minicluster throws NPE on shutdown.

HDFS-8549: Abort the balancer if an upgrade is in progress.

HDFS-8554: TestDatanodeLayoutUpgrade fails on Windows.

HDFS-8573: Move creation of restartMeta file logic from BlockReceiver to ReplicaInPipeline.

HDFS-8578: On upgrade, Datanode should process all storage/data dirs in parallel. .

HDFS-8581: ContentSummary on / skips further counts on yielding lock.

HDFS-8582: Support getting a list of reconfigurable config properties and do not generate spurious reconfig warnings.

HDFS-8593: Calculation of effective layout version mishandles comparison to current layout version in storage.

HDFS-8615: Correct HTTP method in WebHDFS document.

HDFS-8633: Fix setting of dfs.datanode.readahead.bytes in hdfs-default.xml to match DFSConfigKeys.

HDFS-8642: Make TestFileTruncate more reliable.

HDFS-8647: Abstract BlockManager's rack policy into BlockPlacementPolicy.

HDFS-8656: Preserve compatibility of ClientProtocol#rollingUpgrade after finalization.

HDFS-8659: Block scanner INFO message is spamming logs.

HDFS-8674: Improve performance of postponed block scans.

HDFS-8676: Delayed rolling upgrade finalization can cause heartbeat expiration.

HDFS-8696: Make the lower and higher watermark in the DN Netty server configurable.

HDFS-8709: Clarify automatic sync in FSEditLog#logEdit.

HDFS-8711: setSpaceQuota command should print the available storage type when input storage type is wrong.

HDFS-8722: Optimize datanode writes for small writes and flushes.

HDFS-8729: Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately which occasionally failed.

HDFS-8767: RawLocalFileSystem.listStatus() returns null for UNIX pipefile.

HDFS-8772: Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails. (Walter Su via wang).

HDFS-8778: TestBlockReportRateLimiting#testLeaseExpiration can deadlock.

HDFS-8785: TestDistributedFileSystem is failing in trunk.

HDFS-8792: BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory.

HDFS-8797: WebHdfsFileSystem creates too many connections for pread.

HDFS-8806: Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared.

HDFS-8809: HDFS fsck reports under construction blocks as CORRUPT.

HDFS-8815: DFS getStoragePolicy implementation using single RPC call.

HDFS-8816: Improve visualization for the Datanode tab in the NN UI.

HDFS-8818: Changes the global moveExecutor to per datanode executors and changes MAX_SIZE_TO_MOVE to be configurable. (szetszwo).

HDFS-8824: Do not use small blocks for balancing the cluster. (szetszwo).

HDFS-8826: In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo).

HDFS-8828: Utilize Snapshot diff report to build diff copy list in distcp.

HDFS-8829: Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning.

HDFS-8831: Trash Support for deletion in HDFS encryption zone.

HDFS-8844: TestHDFSCLI does not cleanup the test directory.

HDFS-8845: DiskChecker should not traverse the entire tree.

HDFS-8846: Add a unit test for INotify functionality across a layout version upgrade.

HDFS-8850: VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks.

HDFS-8852: HDFS architecture documentation of version 2.x is outdated about append write support.

HDFS-8855: Webhdfs client leaks active NameNode connections.

HDFS-8856: Make LeaseManager#countPath O(1).

HDFS-8859: Improve DataNode ReplicaMap memory footprint to save about 45%.

HDFS-8860: Remove unused Replica copyOnWrite code.

HDFS-8863: The remaining space check in BlockPlacementPolicyDefault is flawed.

HDFS-8867: Enable optimized block reports.

HDFS-8879: Quota by storage type usage incorrectly initialized upon namenode restart.

HDFS-8880: NameNode metrics logging.

HDFS-8883: NameNode Metrics : Add FSNameSystem lock Queue Length.

HDFS-8884: Fail-fast check in BlockPlacementPolicyDefault#chooseTarget.

HDFS-8885: ByteRangeInputStream used in webhdfs does not override available().

HDFS-8887: Expose storage type and storage ID in BlockLocation.

HDFS-8891: HDFS concat should keep srcs order.

HDFS-8896: DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager.

HDFS-8898: Create API and command-line argument to get quota and quota usage without detailed content summary.

HDFS-8911: NameNode Metric : Add Editlog counters as a JMX metric.

HDFS-8914: Document HA support in the HDFS HdfsDesign.md.

HDFS-8923: Add -source flag to balancer usage message. .

HDFS-8930: Block report lease may leak if the 2nd full block report comes when NN is still in safemode.

HDFS-8932: NPE thrown in NameNode when try to get TotalSyncCount metric before editLogStream initialization.

HDFS-8939: Test(S)WebHdfsFileContextMainOperations failing on branch-2.

HDFS-8946: Improve choosing datanode storage for block placement.

HDFS-8950: NameNode refresh doesn't remove DataNodes that are no longer in the allowed list.

HDFS-8964: When validating the edit log, do not read at or beyond the file offset that is being written.

HDFS-8965: Harden edit log reading code against out of memory errors.

HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.

HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.

HDFS-8983: NameNode support for protected directories.

HDFS-8983: NameNode support for protected directories.

HDFS-8995: Flaw in registration bookeeping can make DN die on reconnect.

HDFS-9008: Balancer#Parameters class could use a builder pattern.

HDFS-9008: was not backported correctly.

HDFS-9009: Send metrics logs to NullAppender by default.

HDFS-9019: Adding informative message to sticky bit permission denied exception.

HDFS-9033: dfsadmin -metasave prints "NaN" for cache used%.

HDFS-9034: StorageTypeStats Metric should not count failed storage.'.

HDFS-9038: DFS reserved space is erroneously counted towards non-DFS used.

HDFS-9042: Update document for the Storage policy name.

HDFS-9043: Doc updation for commands in HDFS Federation.

HDFS-9047: Retire libwebhdfs.

HDFS-9048: DistCp documentation is out-of-dated.

HDFS-9063: Correctly handle snapshot path for getContentSummary.

HDFS-9072: Fix random failures in TestJMXGet.

HDFS-9082: Change the log level in WebHdfsFileSystem.initialize() from INFO to DEBUG.

HDFS-9083: Replication violates block placement policy.

HDFS-9094: Add command line option to ask NameNode reload configuration.

HDFS-9106: Transfer failure during pipeline recovery causes permanent write failures.

HDFS-9107: Prevent NNs unrecoverable death spiral after full GC.

HDFS-9112: Improve error message for Haadmin when multiple name service IDs are configured.

HDFS-9128: TestWebHdfsFileContextMainOperations and TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows.

HDFS-9130: Use GenericTestUtils#setLogLevel to the logging level.

HDFS-9142: Separating Configuration object for namenode(s) in MiniDFSCluster.

HDFS-9145: Tracking methods that hold FSNamesytemLock for too long.

HDFS-9148: Incorrect assert message in TestWriteToReplica#testWriteToTemporary.

HDFS-9175: Change scope of 'AccessTokenProvider.getAccessToken()' and 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to public.

HDFS-9178: Slow datanode I/O can cause a wrong node to be marked bad.

HDFS-9184: Logging HDFS operation's caller context into audit logs.

HDFS-9188: Make block corruption related tests FsDataset- agnostic.

HDFS-9193: Fix incorrect references the usages of the DN in dfshealth.js.

HDFS-9198: Coalesce IBR processing in the NN.

HDFS-9205: Do not schedule corrupt blocks for replication. (szetszwo).

HDFS-9210: Fix some misuse of %n in VolumeScanner#printStats.

HDFS-9214: Support reconfiguring dfs.datanode.balance.max.concurrent.moves without DN restart.

HDFS-9220: Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum.

HDFS-9221: HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array.

HDFS-9226: Unit Tests failing due to mockito version issue.

HDFS-9239: DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness.

HDFS-9251: Refactor TestWriteToReplica and TestFsDatasetImpl to avoid explicitly creating Files in the tests code.

HDFS-9259: Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario.

HDFS-9264: Minor cleanup of operations on FsVolumeList#volumes. (Walter Su via lei).

HDFS-9273: ACLs on root directory may be lost after NN restart.

HDFS-9276: Failed to Update HDFS Delegation Token for long running application in HA mode.

HDFS-9289: Make DataStreamer#block thread safe and verify genStamp in commitBlock.

HDFS-9290: DFSClient#callAppend() is not backward compatible for slightly older NameNodes.

HDFS-9294: DFSClient deadlock when close file and failed to renew lease.

HDFS-9305: Delayed heartbeat processing causes storm of subsequent heartbeats.

HDFS-9311: Support optional offload of NameNode HA service health checks to a separate RPC server.

HDFS-9313: Possible NullPointerException in BlockManager if no excess replica can be chosen.

HDFS-9314: Improve BlockPlacementPolicyDefault's picking of excess replicas.

HDFS-9317: Document fsck -blockId and -storagepolicy options in branch-2.7.

HDFS-9343: Empty caller context considered invalid.

HDFS-9347: Invariant assumption in TestQuorumJournalManager.shutdown() is wrong.

HDFS-9349: Support reconfiguring fs.protected.directories without NN restart.

HDFS-9350: Avoid creating temprorary strings in Block.toString() and getBlockName() (Staffan Friberg via cmccabe).

HDFS-9354: Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows.

HDFS-9357: NN UI renders icons of decommissioned DN incorrectly.

HDFS-9358: TestNodeCount#testNodeCount timed out.

HDFS-9362: TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.

HDFS-9364: Unnecessary DNS resolution attempts when creating NameNodeProxies.

HDFS-9383: TestByteArrayManager#testByteArrayManager fails.

HDFS-9384: TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server.

HDFS-9395: Make HDFS audit logging consistant.

HDFS-9397: Fix typo for readChecksum() LOG.warn in BlockSender.java.

HDFS-9402: Switch DataNode.LOG to use slf4j.

HDFS-9406: FSImage may get corrupted after deleting snapshot.

HDFS-9412: getBlocks occupies FSLock and takes too long to complete.

HDFS-9413: getContentSummary() on standby should throw StandbyException.

HDFS-9414: Refactor reconfiguration of ClientDatanodeProtocol for reusability.

HDFS-9415: Document dfs.cluster.administrators and dfs.permissions.superusergroup.

HDFS-9426: Rollingupgrade finalization is not backward compatible.

HDFS-9428: Fix intermittent failure of TestDNFencing.testQueueingWithAppend.

HDFS-9431: DistributedFileSystem#concat fails if the target path is relative.

HDFS-9434: Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messags.

HDFS-9445: Datanode may deadlock while handling a bad volume.

HDFS-9466: TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky.

HDFS-9467: Fix data race accessing writeLockHeldTimeStamp in FSNamesystem.

HDFS-9470: Encryption zone on root not loaded from fsimage after NN restart. Xiao Chen via wang.

HDFS-9476: TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail.

HDFS-9478: Reason for failing ipc.FairCallQueue contruction should be thrown.

HDFS-9484: NNThroughputBenchmark$BlockReportStats should not send empty block reports.

HDFS-9493: Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk.

HDFS-9500: Fix software version counts for DataNodes during rolling upgrade.

HDFS-9505: HDFS Architecture documentation needs to be refreshed.

HDFS-9516: Truncate file fails with data dirs on multiple disks.

HDFS-9517: Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes.

HDFS-9521: TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk.

HDFS-9525: hadoop utilities need to support provided delegation tokens.

HDFS-9530: ReservedSpace is not cleared for abandoned Blocks.

HDFS-9533: seen_txid in the shared edits directory is modified during bootstrapping.

HDFS-9534: Add CLI command to clear storage policy from a path.

HDFS-9549: TestCacheDirectives#testExceedsCapacity is flaky.

HDFS-9555: LazyPersistFileScrubber should still sleep if there are errors in the clear progress.

HDFS-9557: Reduce object allocation in PB conversion. (Daryn Sharp via cnauroth).

HDFS-9566: Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method.

HDFS-9569: 's message change.

HDFS-9569: Log the name of the fsimage being loaded for better supportability.

HDFS-9572: Prevent DataNode log spam if a client connects on the data transfer port but sends no data.

HDFS-9574: Reduce client failures during datanode restart.

HDFS-9584: NPE in distcp when ssl configuration file does not exist in class path.

HDFS-9589: Block files which have been hardlinked should be duplicated before the DataNode appends to the them.

HDFS-9600: Do not check replication if the block is under construction.

HDFS-9601: NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block.

HDFS-9605: Add links to failed volumes to explorer.html in HDFS Web UI.

HDFS-9608: Disk IO imbalance in HDFS with heterogeneous storages.

HDFS-9612: DistCp worker threads are not terminated after jobs are done.

HDFS-9618: Fix mismatch between log level and guard in BlockManager#computeRecoveryWorkForBlocks.

HDFS-9619: SimulatedFSDataset sometimes can not find blockpool for the correct namenode.

HDFS-9623: Update example configuration of block state change log in log4j.properties.

HDFS-9624: DataNode start slowly due to the initial DU command operations.

HDFS-9625: set replication for empty file failed when set storage policy.

HDFS-9626: TestBlockReplacement#testBlockReplacement fails occasionally.

HDFS-9629: Update the footer of Web UI to show year 2016.

HDFS-9630: DistCp minor refactoring and clean up.

HDFS-9634: webhdfs client side exceptions don't provide enough details.

HDFS-9638: to branch-2.7.

HDFS-9644: Update encryption documentation to reflect nested EZs.

HDFS-9655: NN should start JVM pause monitor before loading fsimage.

HDFS-9661: Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage moveBlockAcrossStorage and createRbw.

HDFS-9669: TcpPeerServer should respect ipc.server.listen.queue.size.

HDFS-9670: DistCp throws NPE when source is root.

HDFS-9672: o.a.h.hdfs.TestLeaseRecovery2 fails intermittently.

HDFS-9688: Test the effect of nested encryption zones in HDFS downgrade.

HDFS-9696: Garbage snapshot records linger forever.

HDFS-9700: BUG-60868. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets for DataTransferProtocol.

HDFS-9701: DN may deadlock when hot-swapping under load.

HDFS-9706: Log more details in debug logs in BlockReceiver's constructor.

HDFS-9710: DN can be configured to send block receipt IBRs in batches.

HDFS-9711: Integrate CSRF prevention filter in WebHDFS.

HDFS-9713: DataXceiver#copyBlock should return if block is pinned.

HDFS-9715: Check storage ID uniqueness on datanode startup.

HDFS-9721: Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference.

HDFS-9724: Degraded performance in WebHDFS listing as it does not reuse ObjectMapper.

HDFS-9726: Refactor IBR code to a new class. (szetszwo).

HDFS-9730: Storage ID update does not happen when there is a layout change.

HDFS-9739: DatanodeStorage.isValidStorageId() is broken.

HDFS-9740: Use a reasonable limit in DFSTestUtil.waitForMetric().

HDFS-9743: Fix TestLazyPersistFiles#testFallbackToDiskFull in branch-2.7.

HDFS-9748: Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice.

HDFS-9752: Permanent write failures may happen to slow writers during datanode rolling upgrades.

HDFS-9760: WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler.

HDFS-9764: DistCp doesn't print value for several arguments including -numListstatusThreads.

HDFS-9765: TestBlockScanner#testVolumeIteratorWithCaching fails intermittently.

HDFS-9766: TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently.

HDFS-9768: Reuse ObjectMapper instance in HDFS to improve the performance.

HDFS-9772: TestBlockReplacement#testThrottler doesn't work as expected.

HDFS-9777: Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh).

HDFS-9779: . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value.

HDFS-9781: FsDatasetImpl#getBlockReports can occasionally throw NullPointerException.

HDFS-9784: Example usage is not correct in Transparent Encryption document.

HDFS-9790: HDFS Balancer should exit with a proper message if upgrade is not finalized.

HDFS-9797: Log Standby exceptions thrown by RequestHedgingProxyProvider at DEBUG Level.

HDFS-9799: Reimplement getCurrentTrashDir to remove incompatibility.

HDFS-9801: ReconfigurableBase should update the cached configuration.

HDFS-9812: Streamer threads leak if failure happens when closing DFSOutputStream.

HDFS-9839: Reduce verbosity of processReport logging.

HDFS-9842: dfs.datanode.balance.bandwidthPerSec should accept friendly size units.

HDFS-9844: Correct path creation in getTrashRoot to handle root dir.

HDFS-9851: NameNode throws NPE when setPermission is called on a path that does not exist.

HDFS-9854: Log cipher suite negotiation more verbosely.

HDFS-9864: Correct reference for RENEWDELEGATIONTOKEN and CANCELDELEGATIONTOKEN in webhdfs doc.

HDFS-9865: TestBlockReplacement fails intermittently in trunk.

HDFS-9871: "Bytes Being Moved" -ve(-1 B) when cluster was already balanced.

HDFS-9874: Long living DataXceiver threads cause volume shutdown to block.

HDFS-9880: TestDatanodeRegistration fails occasionally.

HDFS-9881: DistributedFileSystem#getTrashRoot returns incorrect path for encryption zones.

HDFS-9882: Add heartbeatsTotal in Datanode metrics.

HDFS-9902: Support different values of dfs.datanode.du.reserved per storage type.

HDFS-9904: testCheckpointCancellationDuringUpload occasionally fails.

HDFS-9905: WebHdfsFileSystem#runWithRetry should display original stack trace on error.

HDFS-9906: Remove spammy log spew when a datanode is restarted.

HDFS-9917: IBR accumulate more objects when SNN was down for sometime.

HDFS-9926: MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils.

HDFS-9941: Do not log StandbyException on NN, other minor logging fixes.

HDFS-9947: Block#toString should not output information from derived classes.

HDFS-9949: Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared.

HDFS-9958: BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages.

HDFS-10178: Permanent write failures can happen if pipeline recoveries occur for the first packet.

HDFS-10182: Hedged read might overwrite user's buf.

HDFS-10186: DirectoryScanner: Improve logs by adding full path of both actual and expected block directories. .

HDFS-10189: PacketResponder#toString should include the downstreams for PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE.

HDFS-10199: Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing.

HDFS-10207: Support enable Hadoop IPC backoff without namenode restart.

HDFS-10209: Support enable caller context in HDFS namenode audit log without restart namenode.

HDFS-10216: Distcp -diff throws exception when handling relative path.

HDFS-10217: show 'blockScheduled' tooltip in datanodes table.

HDFS-10223: peerFromSocketAndKey performs SASL exchange before setting connection timeouts.

HDFS-10224: Implement asynchronous rename for DistributedFileSystem.

HDFS-10225: DataNode hot swap drives should disallow storage type changes.

HDFS-10228: TestHDFSCLI fails.

HDFS-10235: Last contact for Live Nodes should be relative time.

HDFS-10239: Fsshell mv fails if port usage doesn't match in src and destination paths.

HDFS-10242: Cannot create space quota of zero.

HDFS-10245: Fix the findbugs warnings in branch-2.7.

HDFS-10253: Fix TestRefreshCallQueue failure.

HDFS-10261: TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports.

HDFS-10264: Logging improvements in FSImageFormatProtobuf.Saver.

HDFS-10270: TestJMXGet:testNameNode() fails.

HDFS-10271: Extra bytes are getting released from reservedSpace for append.

HDFS-10275: TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly.

HDFS-10277: PositionedReadable test testReadFullyZeroByteFile failing in HDFS.

HDFS-10279: Improve validation of the configured number of tolerated failed volumes.

HDFS-10281: TestPendingCorruptDnMessages fails intermittently.

HDFS-10287: MiniDFSCluster should implement AutoCloseable.

HDFS-10291: TestShortCircuitLocalRead failing.

HDFS-10301: BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order.

HDFS-10301: Remove FBR tracking state to fix false zombie storage detection for interleaving block reports.

HDFS-10309: Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga) (Amit Anand).

HDFS-10312: Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.

HDFS-10313: Distcp need to enforce the order of snapshot names passed to -diff.

HDFS-10319: Balancer should not try to pair storages with different types.

HDFS-10320: Rack failures may result in NN terminate.

HDFS-10324: Trash directory in an encryption zone should be pre-created with correct permissions.

HDFS-10329: Bad initialisation of StringBuffer in RequestHedgingProxyProvider.

HDFS-10330: Add Corrupt Blocks Information in Metasave output.

HDFS-10335: Mover$Processor#chooseTarget() always chooses the first matching target storage group. .

HDFS-10341: Add a metric to expose the timeout number of pending replication blocks.

HDFS-10342: BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt.

HDFS-10343: BlockManager#createLocatedBlocks may return blocks on failed storages.

HDFS-10344: DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash.

HDFS-10346: Implement asynchronous setPermission/setOwner for DistributedFileSystem.

HDFS-10347: Namenode report bad block method doesn't log the bad block or datanode.

HDFS-10360: DataNode may format directory and lose blocks if current/VERSION is missing.

HDFS-10367: TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception.

HDFS-10372: Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume.

HDFS-10377: CacheReplicationMonitor shutdown log message should use INFO level.

HDFS-10390: Implement asynchronous setAcl/getAclStatus for DistributedFileSystem.

HDFS-10396: Using -diff option with DistCp may get "Comparison method violates its general contract" exception.

HDFS-10397: Distcp should ignore -delete option if -diff option is provided instead of exiting.

HDFS-10410: RedundantEditLogInputStream.LOG is set to wrong class.

HDFS-10415: TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called.

HDFS-10417: Improve error message from checkBlockLocalPathAccess.

HDFS-10423: Increase default value of httpfs maxHttpHeaderSize.

HDFS-10424: DatanodeLifelineProtocol not able to use under security cluster.

HDFS-10431: Refactor and speedup TestAsyncDFSRename.

HDFS-10437: ReconfigurationProtocol not covered by HDFSPolicyProvider.

HDFS-10438: When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.

HDFS-10440: Improve DataNode web UI.

HDFS-10448: CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1.

HDFS-10455: Logging the username when deny the setOwner operation.

HDFS-10457: DataNode should not auto-format block pool directory if VERSION is missing.

HDFS-10458: getFileEncryptionInfo should return quickly for non-encrypted cluster.

HDFS-10468: HDFS read ends up ignoring an interrupt.

HDFS-10469: Add number of active xceivers to datanode metrics.

HDFS-10471: DFSAdmin#SetQuotaCommand's help msg is not correct.

HDFS-10474: hftp copy fails when file name with Chinese+special char in branch-2.

HDFS-10481: HTTPFS server should correctly impersonate as end user to open file.

HDFS-10485: Fix findbugs warning in FSEditLog.java.

HDFS-10488: Update WebHDFS documentation regarding CREATE and MKDIR default permissions.

HDFS-10493: Add links to datanode web UI in namenode datanodes page.

HDFS-10499: TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails Intermittently.

HDFS-10508: DFSInputStream should set thread's interrupt status after catching InterruptException from sleep.

HDFS-10512: VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks.

HDFS-10525: Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap.

HDFS-10544: Balancer doesn't work with IPFailoverProxyProvider.

HDFS-10556: DistCpOptions should be validated automatically.

HDFS-10569: A bug causes OutOfIndex error in BlockListAsLongs.

HDFS-10579: HDFS web interfaces lack configs for X-FRAME-OPTIONS protection.

HDFS-10589: Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params.

HDFS-10609: Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications.

HDFS-10617: PendingReconstructionBlocks.size() should be synchronized.

HDFS-10623: Remove unused import of httpclient.HttpConnection from TestWebHdfsTokens.

HDFS-10625: VolumeScanner to report why a block is found bad.

HDFS-10627: Volume Scanner marks a block as "suspect" even if the exception is network-related.

HDFS-10642: TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently.

HDFS-10643: [Unit Test] Namenode should use loginUser(hdfs) to generateEncryptedKey.

HDFS-10643: HDFS namenode should always use service user (hdfs) to generateEncryptedKey.

HDFS-10653: Optimize conversion from path string to components.

HDFS-10655: Fix path related byte array conversion bugs.

HDFS-10660: Expose storage policy apis via HDFSAdmin interface.

HDFS-10684: WebHDFS DataNode calls fail without parameter createparent.

HDFS-10688: BPServiceActor may run into a tight loop for sending block report when hitting IOException.

HDFS-10703: HA NameNode Web UI should show last checkpoint time.

HDFS-10710: In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.

HDFS-10713: Throttle FsNameSystem lock warnings.

HDFS-10715: NPE when applying AvailableSpaceBlockPlacementPolicy.

HDFS-10716: In Balancer, the target task should be removed when its size < 0. .

HDFS-10735: Distcp using webhdfs on secure HA clusters fails with StandbyException.

HDFS-10752: Several log refactoring/improvement suggestion in HDFS.

HDFS-10757: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.

HDFS-10763: Open files can leak permanently due to inconsistent lease update.

HDFS-10763: Open files can leak permanently due to inconsistent lease update.

HDFS-10764: Fix INodeFile#getBlocks to not return null.

HDFS-10793: Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184.

HDFS-10798: Make the threshold of reporting FSNamesystem lock contention configurable.

HDFS-10817: Add Logging for Long-held NN Read Locks.

HDFS-10822: Log DataNodes in the write pipeline. John Zhuge via Lei Xu.

HDFS-10892: Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'.

HDFS-10893: Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test.

HDFS-10906: Add unit tests for Trash with HDFS encryption zones.

HDFS-10913: Introduce fault injectors to simulate slow mirrors.

HDFS-10917: Collect peer performance statistics on DataNode.

HDFS-10922: Adding additional unit tests for Trash (II).

HDFS-10930: Refactor: Wrap Datanode IO related operations.

HDFS-10934: TestDFSShell#testStat fails intermittently.

HDFS-10956: Remove rename/delete performance penalty when not using snapshots.

HDFS-10959: Update DataNode to use DatasetVolumeChecker.

HDFS-10962: TestRequestHedgingProxyProvider is flaky.

HDFS-10963: Reduce log level when network topology cannot find enough datanodes.

HDFS-10965: Add unit test for HDFS command 'dfsadmin -printTopology'.

HDFS-10966: Enhance Dispatcher logic on deciding when to give up a source DataNode.

HDFS-10972: Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'.

HDFS-10984: Expose nntop output as metrics.

HDFS-10986: DFSAdmin should log detailed error message if any.

HDFS-10987: Make Decommission less expensive when lot of blocks present.

HDFS-10998: Add unit tests for HDFS command 'dfsadmin -fetchImage' in HA.

HDFS-11003: Expose XmitsInProgress through DataNodeMXBean.

HDFS-11008: Change unit test for testing parsing "-source" parameter in Balancer CLI.

HDFS-11011: Add unit tests for HDFS command 'dfsadmin -set/clrSpaceQuota'.

HDFS-11015: Enforce timeout in balancer.

HDFS-11018: Incorrect check and message in FsDatasetImpl#invalidate.

HDFS-11030: TestDataNodeVolumeFailure#testVolumeFailure is flaky (though passing).

HDFS-11031: Add additional unit test for DataNode startup behavior when volumes fail.

HDFS-11047: Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode.

HDFS-11054: Suppress verbose log message in BlockPlacementPolicyDefault.

HDFS-11056: Addendum fix for self-healing branch.

HDFS-11056: Concurrent append and read operations lead to checksum error.

HDFS-11065: Add space quota tests for heterogenous storages.

HDFS-11069: Tighten the authorization of datanode RPC.

HDFS-11076: Add unit test for extended Acls.

HDFS-11083: Add unit test for DFSAdmin -report command.

HDFS-11085: Add unit test for NameNode failing to start when name dir is unwritable.

HDFS-11087: NamenodeFsck should check if the output writer is still writable.

HDFS-11112: Journal Nodes should refuse to format non-empty directories.

HDFS-11114: Support for running async disk checks in DataNode.

HDFS-11119: Support for parallel checking of StorageLocations on DataNode startup.

HDFS-11122: TestDFSAdmin#testReportCommand fails due to timed out.

HDFS-11128: CreateEditsLog throws NullPointerException.

HDFS-11148: Update DataNode to use StorageLocationChecker at startup.

HDFS-11149: Fix build break.

HDFS-11149: Support for parallel checking of FsVolumes.

HDFS-11156: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API.

HDFS-11160: VolumeScanner reports write-in-progress replicas as corrupt incorrectly.

HDFS-11180: Intermittent deadlock in NameNode when failover happens.

HDFS-11182: Addendum fix for Fenton to fix a unit test.

HDFS-11182: Update DataNode to use DatasetVolumeChecker.

HDFS-11194: Maintain aggregated peer performance metrics on NameNode.

HDFS-11195: Return error when appending files by webhdfs rest api fails.

HDFS-11197: Listing encryption zones fails when deleting a EZ that is on a snapshotted directory.

HDFS-11209: SNN can't checkpoint when rolling upgrade is not finalized.

HDFS-11224: Lifeline message should be ignored for dead nodes.

HDFS-11229: HDFS-11056 failed to close meta file.

HDFS-11240: Remove snapshot version of SDK dependency from Azure Data Lake Store File System.

HDFS-11251: ConcurrentModificationException during DataNode#refreshVolumes.

HDFS-11253: FileInputStream leak on failure path in BlockSender.

HDFS-11258: File mtime change could not save to editlog.

HDFS-11263: ClassCastException when we use Bzipcodec for Fsimage compression.

HDFS-11267: Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage.

HDFS-11274: Datanode should only check the failed volume upon IO errors.

HDFS-11279: Cleanup unused DataNode#checkDiskErrorAsync().

HDFS-11280: Allow WebHDFS to reuse HTTP connections to NN.

HDFS-11282: Document the missing metrics of DataNode Volume IO operations.

HDFS-11292: log lastWrittenTxId etc info in logSyncAll.

HDFS-11299: Support multiple Datanode File IO hooks.

HDFS-11307: The rpc to portmap service for NFS has hardcoded timeout.

HDFS-11337: (HDFS-10958 backport). Add instrumentation hooks around Datanode disk IO.

HDFS-11339: Support File IO sampling for Datanode IO profiling hooks.

HDFS-11369: Change exception message in StorageLocationChecker.

HDFS-11379: DFSInputStream may infinite loop requesting block locations.

HDFS-11387: Socket reuse address option is not honored in PrivilegedNfsGatewayStarter.

HDFS-11391: Numeric usernames do no work with WebHDFS FS write access.

HDFS-11395: RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode.

HDFS-11403: Zookeper ACLs on NN HA enabled clusters should be handled consistently.

HDFS-11445: FSCK shows overall health stauts as corrupt even one replica is corrupt.

HDFS-11461: DataNode Disk Outlier Detection.

HDFS-11462: Fix occasional BindException in TestNameNodeMetricsLogger.

HDFS-11476: Fix NPE in FsDatasetImpl#checkAndUpdate.

HDFS-11479: Socket re-use address option should be used in SimpleUdpServer.

HDFS-11499: Decommissioning stuck because of failing recovery.

HDFS-11502: Datanode UI should display hostname based on JMX bean instead of window.location.hostname.

HDFS-11508: Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket is in TIME_WAIT state.

HDFS-11517: Expose slow disks via DataNode JMX.

HDFS-11533: reuseAddress option should be used for child channels in Portmap and SimpleTcpServer.

HDFS-11534: Add counters for number of blocks in pending IBR.

HDFS-11545: Propagate DataNode's slow disks info to the NameNode via Heartbeat.

HDFS-11547: Add logs for slow BlockReceiver while writing data to disk.

HDFS-11551: Handle SlowDiskReport from DataNode at the NameNode.

HDFS-11560: Expose slow disks via NameNode JMX.

HDFS-11570: Unit test for NameNodeStatusMXBean.

HDFS-11603: Improve slow mirror/disk warnings in BlockReceiver.

HDFS-11608: HDFS write crashed with block size greater than 2 GB.

HDFS-11634: Optimize BlockIterator when interating starts in the middle.

HDFS-11722: Change Datanode file IO profiling sampling to percentage.

HDFS-11842: TestDataNodeOutlierDetectionViaMetrics UT fails intermittently. Controbuted by Hanisha Koneru.

HDFS-12139: HTTPFS liststatus returns incorrect pathSuffix for path of file'.

HDFS-12158: Secondary Namenode's web interface lack configs for X-FRAME- OPTIONS protection.'.

HDFS-14676: Wrong default value for fs.df.interval.'.

MAPREDUCE-5485: Allow repeating job commit by extending OutputCommitter API.

MAPREDUCE-5621: mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time.

MAPREDUCE-6121: JobResourceUpdater#compareFs() doesn't handle HA namespaces.

MAPREDUCE-6165: [JDK8] TestCombineFileInputFormat failed on JDK8.

MAPREDUCE-6197: Cache MapOutputLocations in ShuffleHandler.

MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.

MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.

MAPREDUCE-6246: DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2.

MAPREDUCE-6273: HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state.

MAPREDUCE-6279: AM should explicity exit JVM after all services have stopped.

MAPREDUCE-6302:.

MAPREDUCE-6304: Specifying node labels when submitting MR jobs.

MAPREDUCE-6338: MR AppMaster does not honor ephemeral port range.

MAPREDUCE-6350: JobHistory doesn't support fully-functional search.

MAPREDUCE-6404: Allow AM to specify a port range for starting its webapp.

MAPREDUCE-6436: JobHistory cache issue.

MAPREDUCE-6436: JobHistory cache issue.

MAPREDUCE-6460: TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails.

MAPREDUCE-6478: Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob.

MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.

MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.

MAPREDUCE-6514: Job hangs as ask is not updated after ramping down of all reducers.

MAPREDUCE-6543: Migrate MR Client test cases part 2.

MAPREDUCE-6549: multibyte delimiters with LineRecordReader cause duplicate records.

MAPREDUCE-6565: Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.

MAPREDUCE-6566: Add retry support to mapreduce CLI tool.

MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.

MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.

MAPREDUCE-6579: TestNetworkedJob.testNetworkedJob failed due to YARN-3946.

MAPREDUCE-6593: TestJobHistoryEventHandler.testTimelineEventHandling fails because of NPE.

MAPREDUCE-6616: Fail to create jobhistory file if there are some multibyte characters in the job name.

MAPREDUCE-6618: YarnClientProtocolProvider leaking the YarnClient thread.(xgong).

MAPREDUCE-6621: Memory Leak in JobClient#submitJobInternal().

MAPREDUCE-6625: TestCLI#testGetJob fails occasionally (haibochen via rkanter).

MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.

MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.

MAPREDUCE-6639: Process hangs in LocatedFileStatusFetcher if FileSystem.get throws.

MAPREDUCE-6672: TestTeraSort fails on Windows.

MAPREDUCE-6684: High contention on scanning of user directory under immediate_done in Job History Server.

MAPREDUCE-6689: MapReduce job can infinitely increase number of reducer resource requests.

MAPREDUCE-6693: ArrayIndexOutOfBoundsException occurs when the length of the job name is equal to mapreduce.jobhistory.jobname.limit.

MAPREDUCE-6701: application master log can not be available when clicking jobhistory's am logs link.

MAPREDUCE-6702: Fixed failures in TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv.

MAPREDUCE-6711: After enabling preemption on cluster, jobs failed without making any attempts made after being preempted.

MAPREDUCE-6724: Single shuffle to memory must not exceed Integer#MAX_VALUE.

MAPREDUCE-6733: MapReduce JerseyTest tests failing with java.net.BindException: Address already in use.

MAPREDUCE-6762: ControlledJob#toString failed with NPE when job status is not successfully updated.

MAPREDUCE-6789: Fix TestAMWebApp failure. .

MAPREDUCE-6792: Allow user's full principal name as owner of MapReduce staging directory in JobSubmissionFiles#JobStagingDir().

MAPREDUCE-6797: Job history server scans can become blocked on a single, slow entry.

MAPREDUCE-6852: Job#updateStatus() failed with NPE due to race condition.txt.

MAPREDUCE-6895: Job end notification not send due to YarnRuntimeException.

MAPREDUCE-6897: Add Unit Test to ensure Job end notification gets sent even when appMaster stop gets YarnRuntimeException.

YARN-857: Localization failures should be available in container diagnostics.

YARN-998: Keep NM resource updated through dynamic resource config for RM/NM restart.

YARN-1376: NM need to notify the log aggregation status to RM through heartbeat.

YARN-1402: Update related Web UI and CLI with exposing client API to check log aggregation status.

YARN-1462: AHS API and other AHS changes to handle tags for completed MR jobs. .

YARN-1556: NPE getting application report with a null appId.

YARN-1728: Workaround guice3x-undecoded pathInfo in YARN WebApp.

YARN-1880: Cleanup TestApplicationClientProtocolOnHA.

YARN-1993: Cross-site scripting vulnerability in TextView.java.

YARN-2019: Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore.

YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.

YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.

YARN-2194: Fix bug causing CGroups functionality to fail on RHEL7.

YARN-2392: Add more diags about app retry limits on AM failures.

YARN-2495: Allow admin specify labels from each NM (Distributed configuration for node label).

YARN-2498: Respect labels in preemption policy of capacity scheduler for inter-queue preemption.

YARN-2513: Host framework UIs in YARN for use with the ATS.

YARN-2513: Host framework UIs in YARN for use with the ATS Note: Patch is in progress.

YARN-2571: RM to support YARN registry.

YARN-2619: Added NodeManager support for disk io isolation through cgroups.

YARN-2666: TestFairScheduler.testContinuousScheduling fails Intermittently.

YARN-2696: Queue sorting in CapacityScheduler should consider node label.

YARN-2740: Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled.

YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.

YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.

YARN-2868: FairScheduler: Metric for latency to allocate first container for an application.

YARN-2901: Add errors and warning metrics page to RM, NM web UI.

YARN-3021: YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp.

YARN-3026: Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp.

YARN-3102: Decommisioned Nodes not listed in Web UI.

YARN-3110: Few issues in ApplicationHistory web ui.

YARN-3136: Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers.

YARN-3148: Allow CORS related headers to passthrough in WebAppProxyServlet.

YARN-3215: Respect labels in CapacityScheduler when computing headroom.

YARN-3216: Max-AM-Resource-Percentage should respect node labels.

YARN-3225: New parameter of CLI for decommissioning node gracefully in RMAdmin CLI.

YARN-3243: CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits.

YARN-3248: Display count of nodes blacklisted by apps in the web UI.

YARN-3266: RMContext#inactiveNodes should have NodeId as map key.

YARN-3269: Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path.

YARN-3293: Track and display capacity scheduler health metrics in web UI.

YARN-3294: Allow dumping of Capacity Scheduler debug logs via web UI for a fixed time period.

YARN-3305: Normalize AM resource request on app submission.

YARN-3318: Create Initial OrderingPolicy Framework and FifoOrderingPolicy.

YARN-3319: Implement a FairOrderingPolicy.

YARN-3326: Support RESTful API for getLabelsToNodes.

YARN-3343: Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate timeout.

YARN-3345: Add non-exclusive node label API.

YARN-3347: Improve YARN log command to get AMContainer logs as well as running containers logs.

YARN-3348: Add a 'yarn top' tool to help understand cluster usage.

YARN-3354: Add node label expression in ContainerTokenIdentifier to support RM recovery.

YARN-3356: Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label.

YARN-3357: Move TestFifoScheduler to FIFO package.

YARN-3360: Add JMX metrics to TimelineDataManager.

YARN-3361: CapacityScheduler side changes to support non-exclusive node labels.

YARN-3362: Add node label usage in RM CapacityScheduler web UI.

YARN-3365: Enhanced NodeManager to support using the 'tc' tool via container-executor for outbound network traffic control.

YARN-3366: Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers.

YARN-3383: AdminService should use "warn" instead of "info" to log exception when operation fails.

YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.

YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.

YARN-3394: Enrich WebApplication proxy documentation.

YARN-3397: yarn rmadmin should skip -failover.

YARN-3404: Display queue name on application page.

YARN-3406: Display count of running containers in the RM's Web UI.

YARN-3410: YARN admin should be able to remove individual application records from RMStateStore.

YARN-3413: Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime.

YARN-3425: NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed.

YARN-3433: Jersey tests failing with Port in Use -again.

YARN-3435: AM container to be allocated Appattempt AM container shown as null.

YARN-3443: Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM.

YARN-3448: Added a rolling time-to-live LevelDB timeline store implementation.

YARN-3451: Display attempt start time and elapsed time on the web UI.

YARN-3453: Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing.

YARN-3459: Fix failiure of TestLog4jWarningErrorMetricsAppender.

YARN-3463: Integrate OrderingPolicy Framework with CapacityScheduler.

YARN-3467: Expose allocatedMB, allocatedVCores, and runningContainers metrics on running Applications in RM Web UI.

YARN-3480: Remove attempts that are beyond max-attempt limit from state store.

YARN-3494: Expose AM resource limit and usage in CS QueueMetrics.

YARN-3503: Expose disk utilization percentage and bad local and log dir counts in NM metrics.

YARN-3505: Node's Log Aggregation Report with SUCCEED should not cached in RMApps.

YARN-3511: Add errors and warnings page to ATS.

YARN-3517: RM web ui for dumping scheduler logs should be for admins only.

YARN-3521: Support return structured NodeLabel objects in REST API.

YARN-3530: ATS throws exception on trying to filter results without otherinfo.

YARN-3541: Add version info on timeline service / generic history web UI and REST API.

YARN-3543: ApplicationReport should be able to tell whether the Application is AM managed or not.

YARN-3552: RM Web UI shows -1 running containers for completed apps.

YARN-3565: NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String.

YARN-3579: CommonNodeLabelsManager should support NodeLabel instead of string label name when getting node-to-label/label-to-label mappings.

YARN-3580: [JDK8] TestClientRMService.testGetLabelsToNodes fails.

YARN-3581: Deprecate -directlyAccessNodeLabelStore in RMAdminCLI.

YARN-3583: Support of NodeLabel object instead of plain String in YarnClient side.

YARN-3589: RM and AH web UI display DOCTYPE wrongly.

YARN-3591: Resource localization on a bad disk causes subsequent containers failure.

YARN-3593: Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page.

YARN-3600: AM container link is broken -- on a killed application, at least.

YARN-3604: removeApplication in ZKRMStateStore should also disable watch.

YARN-3623: Add a config to indicate the Timeline Service version.

YARN-3623: Addendum: Improve the description for Timeline Service Version configuration in yarn-default.xml.

YARN-3632: Ordering policy should be allowed to reorder an application when demand changes.

YARN-3635: Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework.

YARN-3654: ContainerLogsPage web UI should not have meta-refresh.

YARN-3684: Changed ContainerExecutor's primary lifecycle methods to use a more extensible mechanism of context objects.

YARN-3695: ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.

YARN-3695: ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.

YARN-3700: Made generic history service load a number of latest applications according to the parameter or the configuration.

YARN-3707: RM Web UI queue filter doesn't work.

YARN-3717: Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API. (Naganarasimha G R via wangda).

YARN-3722: Merge multiple TestWebAppUtils into o.a.h.yarn.webapp.util.TestWebAppUtils.

YARN-3727: For better error recovery, check if the directory exists before using it for localization.

YARN-3740: Fixed the typo in the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS.

YARN-3749: We should make a copy of configuration when init MiniYARNCluster with multiple RMs.

YARN-3751: Fixed AppInfo to check if used resources are null.

YARN-3762: FairScheduler: CME on FSParentQueue#getQueueUserAclInfo.

YARN-3766: Fixed the apps table column error of generic history web UI.

YARN-3768: ArrayIndexOutOfBoundsException with empty environment variables.

YARN-3769: Consider user limit when calculating total pending resource for preemption policy in Capacity Scheduler.

YARN-3785: Support for Resource as an argument during submitApp call in MockRM test class.

YARN-3844: Make hadoop-yarn-project Native code -Wall-clean.

YARN-3846: RM Web UI queue filter is not working for sub queue.

YARN-3849: (Too much of preemption activity causing continuos killing of containers across queues) to 2.3-maint.

YARN-3852: Add docker container support to container-executor.

YARN-3853: Add docker container runtime support to LinuxContainterExecutor.

YARN-3873: PendingApplications in LeafQueue should also use OrderingPolicy.

YARN-3877: YarnClientImpl.submitApplication swallows exceptions.

YARN-3885: ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level.

YARN-3894: RM startup should fail for wrong CS xml NodeLabel capacity configuration.

YARN-3896: RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously.

YARN-3920: FairScheduler container reservation on a node should be configurable to limit it to large containers.

YARN-3932: SchedulerApplicationAttempt#getResourceUsageReport and UserInfo should based on total-used-resources.

YARN-3940: Application moveToQueue should check NodeLabel permission.

YARN-3941: Proportional Preemption policy should try to avoid sending duplicate PREEMPT_CONTAINER event to scheduler.

YARN-3946: Update exact reason as to why a submitted app is in ACCEPTED state to app's diagnostic message.

YARN-3948: Display Application Priority in RM Web UI.(Sunil G via rohithsharmaks).

YARN-3961: Expose pending, running and reserved containers of a queue in REST api and yarn top.

YARN-3963: AddNodeLabel on duplicate label addition shows success.

YARN-3966: Fix excessive loggings in CapacityScheduler.

YARN-3971: Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery.

YARN-3978: Configurably turn off the saving of container info in Generic AHS.

YARN-3982: container-executor parsing of container-executor.cfg broken in trunk and branch-2.

YARN-3983: Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic.

YARN-3986: getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface.

YARN-3987: am container complete msg ack to NM once RM receive it.

YARN-4000: RM crashes with NPE if leaf queue becomes parent queue during restart.

YARN-4004: container-executor should print output of docker logs if the docker container exits with non-0 exit status.

YARN-4007: Add support for different network setups when launching the docker container.

YARN-4017: container-executor overuses PATH_MAX.

YARN-4023: Publish Application Priority to TimelineServer.

YARN-4026: Refactored ContainerAllocator to accept a list of priorites rather than a single priority.

YARN-4082: Container shouldn't be killed when node's label updated.

YARN-4087: Followup fixes after YARN-2019 regarding RM behavior when state- store error occurs.

YARN-4092: Fixed UI redirection to print useful messages when both RMs are in standby mode.

YARN-4101: RM should print alert messages if ZooKeeper and Resourcemanager gets connection issue.

YARN-4108: CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request.

YARN-4109: Exception on RM scheduler page loading with labels.

YARN-4113: RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER.

YARN-4132: Separate configs for nodemanager to resourcemanager connection timeout and retries.

YARN-4140: RM container allocation delayed incase of app submitted to Nodelabel partition.

YARN-4149: yarn logs -am should provide an option to fetch all the log files.

YARN-4155: TestLogAggregationService.testLogAggregationServiceWithInterval failing.

YARN-4162: CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API.

YARN-4168: Fixed a failing test TestLogAggregationService.testLocalFileDeletionOnDiskFull.

YARN-4204: ConcurrentModificationException in FairSchedulerQueueInfo.

YARN-4215: RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request.

YARN-4225: Add preemption status to yarn queue -status for capacity scheduler.

YARN-4233: YARN Timeline Service plugin: ATS v1.5.

YARN-4243: Add retry on establishing ZooKeeper conenction in EmbeddedElectorService#serviceInit.

YARN-4245: Generalize config file handling in container- executor.

YARN-4252: Log container-executor invocation details when exit code is non- zero.

YARN-4253: Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor.

YARN-4255: container-executor does not clean up docker operation command files.

YARN-4258: Add support for controlling capabilities for docker containers.

YARN-4262: Allow whitelisted users to run privileged docker containers.

YARN-4267: Add additional logging to container launch implementations in container-executor.

YARN-4270: Limit application resource reservation on nodes for non- node/rack specific requests.

YARN-4285: Display resource usage as percentage of queue and cluster in the RM UI.

YARN-4287: Capacity Scheduler: Rack Locality improvement.

YARN-4288: Fixed RMProxy to retry on IOException from local host.

YARN-4290: Add -showDetails option to YARN Nodes CLI to print all nodes reports information.

YARN-4304: AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics.

YARN-4309: Add container launch related debug information to container logs when a container fails.

YARN-4313: Race condition in MiniMRYarnCluster when getting history server address.

YARN-4315: NaN in Queue percentage for cluster apps page.

YARN-4325: Nodemanager log handlers fail to send finished/failed events in some cases.

YARN-4329: [YARN-5437] Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler.

YARN-4345: yarn rmadmin -updateNodeResource doesn't work.

YARN-4347: Resource manager fails with Null pointer exception.

YARN-4349: YARN_APPLICATION call to ATS does not have YARN_APPLICATION_CALLER_CONTEXT.

YARN-4365: FileSystemNodeLabelStore should check for root dir existence on startup.

YARN-4384: updateNodeResource CLI should not accept negative values for resource.

YARN-4390: Do surgical preemption based on reserved container in CapacityScheduler.

YARN-4392: ApplicationCreatedEvent event time resets after RM restart/failover.

YARN-4393: TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently.

YARN-4403: (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period.

YARN-4405: Support node label store in non-appendable file system.

YARN-4411: RMAppAttemptImpl#createApplicationAttemptReport throws IllegalArgumentException.

YARN-4414: Nodemanager connection errors are retried at multiple levels.

YARN-4414: Nodemanager connection errors are retried at multiple levels.

YARN-4416: Deadlock due to synchronised get Methods in AbstractCSQueue.

YARN-4417: Make RM and Timeline-server REST APIs more consistent.

YARN-4418: AM Resource Limit per partition can be updated to ResourceUsage as well.

YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.

YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.

YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.

YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.

YARN-4439: Clarify NMContainerStatus#toString method.

YARN-4454: NM to nodelabel mapping going wrong after RM restart.

YARN-4465: SchedulerUtils#validateRequest for Label check should happen only when nodelabel enabled.

YARN-4477: FairScheduler: Handle condition which can result in an infinite loop in attemptScheduling.

YARN-4479: Change CS LeafQueue pendingOrderingPolicy to hornor recovered apps.

YARN-4484: Available Resource calculation for a queue is not correct when used with labels.

YARN-4496: Improve HA ResourceManager Failover detection on the client.

YARN-4497: RM might fail to restart when recovering apps whose attempts are missing.

YARN-4502: Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda).

YARN-4522: Queue acl can be checked at app submission.

YARN-4535: Fix checkstyle error in CapacityScheduler.java.

YARN-4546: YARN-4546. ResourceManager crash due to scheduling opportunity overflow.

YARN-4546: YARN-4546. ResourceManager crash due to scheduling opportunity overflow.

YARN-4553: Add cgroups support for docker containers.

YARN-4556: TestFifoScheduler.testResourceOverCommit fails.

YARN-4557: Fix improper Queues sorting in PartitionedQueueComparator when accessible-node-labels=*.

YARN-4565:.

YARN-4571: Make app id/name available to the yarn authorizer provider for better auditing.

YARN-4573: Fix test failure in TestRMAppTransitions#testAppRunningKill and testAppKilledKilled.

YARN-4577: Enable aux services to have their own custom classpath/jar file.

YARN-4578: Directories that are mounted in docker containers need to be more restrictive/container-specific.

YARN-4582: Label-related invalid resource request exception should be able to properly handled by application.

YARN-4584: RM startup failure when AM attempts greater than max-attempts.

YARN-4591: YARN Web UIs should provide a robots.txt. Junping Du.

YARN-4595: Add support for configurable read-only mounts when launching Docker containers.

YARN-4596: SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities.

YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.

YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.

YARN-4607: Pagination support for AppAttempt page TotalOutstandingResource Requests table.

YARN-4610: Reservations continue looking for one app causes other apps to starve.

YARN-4617: LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy.

YARN-4623: TestSystemMetricsPublisher#testPublishAppAttemptMetricsForUnmanagedAM fails with NPE on branch-2.7.

YARN-4624: NPE in PartitionQueueCapacitiesInfo while accessing Scheduler UI.

YARN-4625: Make ApplicationSubmissionContext and ApplicationSubmissionContextInfo more consistent.

YARN-4633: TestRMRestart.testRMRestartAfterPreemption fails intermittently.

YARN-4634: Scheduler UI/Metrics need to consider cases like non-queue label mappings.

YARN-4643: Container recovery is broken with delegating container runtime.

YARN-4667: RM Admin CLI for refreshNodesResources throws NPE when nothing is configured.

YARN-4680: Fix TimerTasks leak in ATS V1.5 Writer.

YARN-4696: TimelineClient to add flush operation for deterministic writes (including testing) and Changes to EntityGroupFSTimelineStore for testability.

YARN-4697: NM aggregation thread pool is not bound by limits.

YARN-4699: Scheduler UI and REST o/p is not in sync when -replaceLabelsOnNode is used to change label of a node.

YARN-4709: NMWebServices produces incorrect JSON for containers.

YARN-4709: NMWebServices produces incorrect JSON for containers.

YARN-4723: NodesListManager$UnknownNodeId ClassCastException.

YARN-4737: Add CSRF filter support in YARN.

YARN-4737: Add CSRF filter support in YARN.

YARN-4744: Too many signal to container failure in case of LCE.

YARN-4746: yarn web services should convert parse failures of appId, appAttemptId and containerId to 400.

YARN-4748: ApplicationHistoryManagerOnTimelineStore should not swallow exceptions on generateApplicationReport.

YARN-4759: Fix signal handling for docker containers.

YARN-4762: Fixed CgroupHandler's creation and usage to avoid NodeManagers crashing when LinuxContainerExecutor is enabled.

YARN-4764: Application submission fails when submitted queue is not available in scheduler xml.

YARN-4767: Network issues can cause persistent RM UI outage.

YARN-4769: Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI.

YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.

YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.

YARN-4811: Generate histograms in ContainerMetrics for actual container resource usage.

YARN-4814: ATS 1.5 timelineclient impl call flush after every event write.

YARN-4815: ATS 1.5 timelineclinet impl try to create attempt directory for every event call.

YARN-4817: TimelineClient ATSv1.5 logging is very noisy.

YARN-4820: ResourceManager web redirects in HA mode drops query parameters.

YARN-4822: Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates.

YARN-4832: NM side resource value should get updated if change applied in RM side.

YARN-4833: For Queue AccessControlException client retries multiple times on both RM.

YARN-4842: Fixed "yarn logs" command to guess (and thus not require) the appOwner argument when viewing another user's logs.

YARN-4844: Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource.

YARN-4846: Fix random failures for TestCapacitySchedulerPreemption #testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers.

YARN-4851: Metric improvements for ATS v1.5 storage components. Li Lu via junping_du.

YARN-4863: AHS Security login should be in serviceInit() instead of serviceStart().

YARN-4865: Track Reserved resources in ResourceUsage and QueueCapacities.

YARN-4890: Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate.

YARN-4893: Fix intermittent failures in TestRMAdminService.

YARN-4905: Improved "yarn logs" command-line to optionally show log metadata also.

YARN-4906: Capture container start/finish time in container metrics.

YARN-4909: Fix intermittent failures of TestRMWebServices And TestRMWithCSRFFilter.

YARN-4913: Yarn logs should take a -out option to write to a directory.

YARN-4916: TestNMProxy.tesNMProxyRPCRetry fails.

YARN-4920: ATS/NM should support a link to dowload/get the logs in text format.

YARN-4927: TestRMHA#testTransitionedToActiveRefreshFail fails with FairScheduler.

YARN-4928: Some yarn.server.timeline.* tests fail on Windows attempting to use a test root path containing a colon.

YARN-4934: Reserved Resource for QueueMetrics needs to be handled correctly in few cases.

YARN-4955: Add retry for SocketTimeoutException in TimelineClient.

YARN-4956: findbug issue on LevelDBCacheTimelineStore.

YARN-4966: Improve yarn logs to fetch container logs without specifying nodeId.

YARN-4983: JVM and UGI metrics disappear after RM transitioned to standby mode.

YARN-4984: LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak.

YARN-4987: Read cache concurrency issue between read and evict in EntityGroupFS timeline store.

YARN-4990: Re-direction of a particular log file within in a container in NM UI does not redirect properly.

YARN-5002: getApplicationReport call may raise NPE for removed queues.

YARN-5023: TestAMRestart#testShouldNotCountFailureToMaxAttemptRetry random failure.

YARN-5029: RM needs to send update event with YarnApplicationState as Running to ATS/AHS.

YARN-5048: DelegationTokenRenewer#skipTokenRenewal may throw NPE.

YARN-5069: TestFifoScheduler.testResourceOverCommit race condition.

YARN-5076: YARN web interfaces lack XFS (Cross-Frame Script) protection.

YARN-5080: Cannot obtain logs using YARN CLI -am for either KILLED or RUNNING AM.

YARN-5083: YARN CLI for AM logs does not give any error message if entered invalid am value.

YARN-5088: Improve "yarn log" command-line to read the last K bytes for the log files.

YARN-5089: Improve "yarn log" command-line "logFiles" option to support regex.

YARN-5098: Fixed ResourceManager's DelegationTokenRenewer to replace expiring system-tokens if RM stops and only restarts after a long time.

YARN-5100: The YarnApplicationState is always running in ATS even application is finished.

YARN-5103: With NM recovery enabled, restarting NM multiple times results in AM restart.

YARN-5112: Excessive log warnings for directory permission issue on NM recovery.

YARN-5115: Avoid setting CONTENT-DISPOSITION header in the container-logs web-service.

YARN-5121: fix some container-executor portability issues.

YARN-5122: "yarn logs" for running containers should print an explicit footer saying that the log may be incomplete.

YARN-5131: Distributed shell AM fails when extra container arrives during finishing. Wangda Tan via junping_du.

YARN-5141: Get Container logs for the Running application from Yarn Logs CommandLine.

YARN-5190: Registering/unregistering container metrics triggered by ContainerEvent and ContainersMonitorEvent are conflict which cause uncaught exception in ContainerMonitorImpl.

YARN-5191: Renamed the newly added “download=true” option for getting logs via NMWebServices and AHSWebServices to be a better "format" option.

YARN-5199: Close LogReader in in AHSWebServices#getStreamingOutput and FileInputStream in NMWebServices#getLogs.

YARN-5200: Enhanced "yarn logs" to be able to get a list of containers whose logs are aggregated via a "show_container_log_info" option.

YARN-5206: RegistrySecurity includes id:pass in exception text if considered invalid.

YARN-5208: Run TestAMRMClient TestNMClient TestYarnClient TestClientRMTokens TestAMAuthorization tests with hadoop.security.token.service.use_ip enabled.

YARN-5213: Fix a bug in LogCLIHelpers which cause TestLogsCLI#testFetchApplictionLogs fails intermittently.

YARN-5214: Fixed locking in DirectoryCollection to avoid hanging NMs when various code-paths hit slow disks.

YARN-5223: Container line in yarn logs output for a live application should include the hostname for the container.

YARN-5224: Added new web-services /containers/{containerid}/logs & /containers/{containerid}/logs/{filename} and using them in "yarn logs" CLI to get logs of finished containers of a running application.

YARN-5226: Remove AHS enable check from LogsCLI#fetchAMContainerLogs.

YARN-5227: Yarn logs command: no need to specify applicationId when specifying containerId.

YARN-5233: Support for specifying a path for ATS plugin jars.

YARN-5237: Fix missing log files issue in rolling log aggregation.

YARN-5246: NMWebAppFilter web redirects drop query parameters.

YARN-5251: Yarn CLI to obtain App logs for last 'n' bytes fails.

YARN-5266: Wrong exit code while trying to get app logs using regex via CLI.

YARN-5270: Solve miscellaneous issues caused by YARN-4844.

YARN-5294: Pass remote ip address down to YarnAuthorizationProvider.

YARN-5296: NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl.

YARN-5297: Avoid printing a stack trace when recovering an app after the RM restarts.

YARN-5298: Mount usercache and NM filecache directories into Docker container.

YARN-5299: Log Docker run command when container fails.

YARN-5314: Fixed a ConcurrentModificationException in ATS v1.5 EntityGroupFSTimelineStore.

YARN-5333: Some recovered apps are put into default queue when RM HA.

YARN-5337: Fix Dshell OOM issues.

YARN-5339: Passing file to -out for YARN log CLI doesnt give warning or error code.

YARN-5340: Fixed a race condition in RollingLevelDBTimelineStore that caused loss of Timeline events.

YARN-5342: Improve non-exclusive node partition resource allocation in Capacity Scheduler.

YARN-5353: ResourceManager can leak delegation tokens when they are shared across apps. (Jason Lowe via Varun Saxena).

YARN-5363: For AM containers, or for containers of running-apps, "yarn logs" incorrectly only (tries to) shows syslog file-type by default.

YARN-5368: Fixed a memory leak in Timeline server.

YARN-5432: Lock already held by another process while LevelDB cache store creation for dag.

YARN-5440: Use AHSClient in YarnClient when TimelineServer is running.

YARN-5456: container-executor support for FreeBSD, NetBSD, and others if conf path is absolute. .

YARN-5470: Differentiate exactly match with regex in yarn log CLI.

YARN-5476: Not existed application reported as ACCEPTED state by YarnClientImpl.

YARN-5523: Yarn running container log fetching causes OutOfMemoryError.

YARN-5524: Yarn live log aggregation does not throw if command line arg is wrong.

YARN-5539: TimelineClient failed to retry on java.net.SocketTimeoutException: Read timed out.

YARN-5549: AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately.

YARN-5551: : Ignore file backed pages from memory computation when smaps is enabled.

YARN-5659: getPathFromYarnURL should use standard methods.

YARN-5677: RM should transition to standby when connection is lost for an extended period.

YARN-5704: Provide config knobs to control enabling/disabling new/work in progress features in container-executor.

YARN-5711: Propogate exceptions back to client when using hedging RM failover provider. Create a new task attempt with failed map task.

YARN-5718: TimelineClient (and other places in YARN) shouldn't over-write HDFS client retry settings which could cause unexpected behavior.

YARN-5731: Preemption calculation is not accurate when reserved containers are present in queue.

YARN-5773: RM recovery too slow due to LeafQueue#activateApplications.

YARN-5864: Capacity Scheduler queue priority.

YARN-5873: RM crashes with NPE if generic application history is enabled.

YARN-5910: Support for multi-cluster delegation tokens.

YARN-5915: ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write.

YARN-5923: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled.

YARN-5923: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled.

YARN-5988: RM unable to start in secure setup.

YARN-5999: AMRMClientAsync will stop if any exceptions thrown on allocate call.

YARN-6011: Add a new web service to list the files on a container in AHSWebService.

YARN-6029: CapacityScheduler deadlock when ParentQueue#getQueueUserAclInfo is called by one thread and LeafQueue#assignContainers is releasing excessive reserved container by another thread.

YARN-6031: Application recovery has failed when node label feature is turned off during RM recovery.

YARN-6054: TimelineServer fails to start when some LevelDb state files are missing.

YARN-6068: Log aggregation get failed when NM restart even with recovery. .

YARN-6081: LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container.

YARN-6082: Invalid REST api response for getApps since queueUsagePercentage is coming as INF.

YARN-6099: Improve webservice to list aggregated log files.

YARN-6100: Improve YARN webservice to output aggregated container logs.

YARN-6104: RegistrySecurity overrides zookeeper sasl system properties.

YARN-6108: Improve AHS webservice to accept NM address as a parameter to get container logs.

YARN-6113: Re-direct NM Web Service to get container logs for finished applications.

YARN-6123: Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed.

YARN-6126: Obtaining app logs for Running application fails with json parse error.

YARN-6137: Yarn client implicitly invoke ATS client which accesses HDFS.

YARN-6145: Improve log message on fail over.

YARN-6174: Log files pattern should be same for both running and finished container.

YARN-6177: Yarn client should exit with an informative error message if an incompatible Jersey library is used at client.

YARN-6189: ApplicationNotFoundException due to RM failed over should throw more user friendly exception.

YARN-6228: EntityGroupFSTimelineStore should allow configurable cache stores.

YARN-6270: WebUtils.getRMWebAppURLWithScheme() needs to honor RM HA setting.

YARN-6271: yarn rmadin -getGroups returns information from standby RM.

YARN-6294: ATS client should better handle Socket closed case.

YARN-6304: Skip rm.transitionToActive call to RM if RM is already active.

YARN-6313: yarn logs cli should provide logs for a completed container even when application is still running.

YARN-6314: Potential infinite redirection on YARN log redirection web service.

YARN-6332: Make RegistrySecurity use short user names for ZK ACLs.

YARN-6339: Improve performance for createAndGetApplicationReport.

YARN-6367: YARN logs CLI needs alway check containerLogsInfo/containerLogInfo before parse the JSON object from NMWebService.

YARN-6726: Fix issues with docker commands executed by container-executor.

YARN-6890: Not display killApp button on UI if UI is unsecured but cluster is secured.

YARN-7090: testRMRestartAfterNodeLabelDisabled[1] UT Fails.

HADOOP-12173: NetworkTopology::add calls toString always.

HDFS-11499: Decommissioning stuck because of failing recovery.

HDFS-11545: Propagate DataNode's slow disks info to the NameNode via Heartbeat.

HDFS-11547: Add logs for slow BlockReceiver while writing data to disk.

HDFS-11551: Handle SlowDiskReport from DataNode at the NameNode. .

HDFS-11560: Expose slow disks via NameNode JMX.

HDFS-11570: Unit test for NameNodeStatusMXBean.

HDFS-11603: Improve slow mirror/disk warnings in BlockReceiver.

HDFS-11722: Change Datanode file IO profiling sampling to percentage.

HDFS-11842: TestDataNodeOutlierDetectionViaMetrics UT fails intermittently.

MAPREDUCE-6702: Fixed failures in TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv.

YARN-5368: Fixed a memory leak in Timeline server.

HADOOP-1023: Add more logging in WASB page blob read function.

HADOOP-1026: Logging bug in WASB causing out of index exception.

HADOOP-1540: breaks backward compatibilty.

HADOOP-1540: Support file exclusion list in distcp.

HADOOP-3733: "s3x:" URLs break when Secret Key contains a slash, even if encoded.

HADOOP-5323: Trash documentation should describe its directory structure and configurations.

HADOOP-5470: RunJar.unJar() should write the last modified time found in the jar entry to the uncompressed file.

HADOOP-7139: Allow appending to existing SequenceFiles

HADOOP-7256: Resource leak during failure scenario of closing of resources.

HADOOP-7713: dfs -count -q should label output column

HADOOP-7713: dfs -count -q should label output column .

HADOOP-7817: RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos()

HADOOP-7930: Kerberos relogin interval in UserGroupInformation should be configurable

HADOOP-8151: Error handling in snappy decompressor throws invalid exceptions.

HADOOP-8436: NPE In getLocalPathForWrite

HADOOP-8437: getLocalPathForWrite should throw IOException for invalid paths.

HADOOP-8751: NPE in Token.toString() when Token is constructed using null identifier.

HADOOP-8818: Use equals instead == in MD5MD5CRC32FileChecksum and TFileDumper.

HADOOP-8934: Shell command ls should include sort options

HADOOP-9121: InodeTree.java has redundant check for vName while throwing exception.

HADOOP-9242: Duplicate surefire plugin config in hadoop-common.

HADOOP-9477: Add posixGroups support for LDAP groups mapping service.

HADOOP-10035: Cleanup TestFilterFileSystem.

HADOOP-10048: LocalDirAllocator should avoid holding locks while accessing the filesystem.

HADOOP-10300: Allowed deferred sending of call responses.

HADOOP-10365: BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block.

HADOOP-10406: TestIPC.testIpcWithReaderQueuing may fail.

HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.

HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.

HADOOP-1062:added hadoop-azure-datalake and azure-data-lake-store-sdk to all the wasb locations.

HADOOP-10775: Shell operations to fail with meaningful errors on windows if winutils.exe not found.

HADOOP-10823: TestReloadingX509TrustManager is flaky.

HADOOP-10965: Print fully qualified path in CommandWithDestination error messages.

HADOOP-10971: Add -C flag to make `hadoop fs -ls` print filenames only.

HADOOP-11031: Design Document for Credential Provider API.

HADOOP-11098: [JDK8] Max Non Heap Memory default changed between JDK7 and 8.

HADOOP-11212: NetUtils.wrapException to handle SocketException explicitly.

HADOOP-11218: Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.

HADOOP-11252: RPC client does not time out by default

HADOOP-11262: Enable YARN to use S3A.

HADOOP-11361: Fix a race condition in MetricsSourceAdapter.updateJmxCache.

HADOOP-11404: Clarify the "expected client Kerberos principal is null" authorization message.

HADOOP-11491: HarFs incorrectly declared as requiring an authority.

HADOOP-11581: Multithreaded correctness Warnings #org.apache.hadoop.fs.shell.Ls

HADOOP-11613: Remove commons-httpclient dependency from hadoop-azure.

HADOOP-11628: SPNEGO auth does not work with CNAMEs in JDK8.

HADOOP-11685: StorageException complaining "no lease ID" during HBase distributed log splitting.

HADOOP-11687: Ignore x-* and response headers when copying an Amazon S3 object.

HADOOP-11692: Improve authentication failure WARN message to avoid user confusion.

HADOOP-11713: ViewFileSystem should support snapshot methods.

HADOOP-11717: Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth

HADOOP-11772: RPC Invoker relies on static ClientCache which has synchronized(this) blocks.

HADOOP-11780: Prevent IPC reader thread death.

HADOOP-11785: Reduce the number of listStatus operation in distcp buildListing

HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm

HADOOP-11812: Implement listLocatedStatus for ViewFileSystem to speed up split calculation

HADOOP-11827: Speed-up distcp buildListing() using threadpool

HADOOP-11859: PseudoAuthenticationHandler fails with httpcomponents v4.4.

HADOOP-11864: JWTRedirectAuthenticationHandler breaks java8 javadocs.

HADOOP-11876: Refactor code to make it more readable, minor maybePrintStats bug

HADOOP-11901: BytesWritable fails to support 2G chunks due to integer overflow.

HADOOP-11918: Listing an empty s3a root directory throws FileNotFound.

HADOOP-11924: Tolerate JDK-8047340-related exceptions in Shell#isSetSidAvailable preventing class init.

HADOOP-11932: MetricsSinkAdapter may hang when being stopped.

HADOOP-11960: Enable Azure-Storage Client Side logging.

HADOOP-12001: Fixed LdapGroupsMapping to include configurable Posix UID and GID attributes during the search.

HADOOP-12006: Remove unimplemented option for from document in branch-2.7.

HADOOP-12009: Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus. backport to 2.8.

HADOOP-12017: Hadoop archives command should use configurable replication factor when closing

HADOOP-12043: Display warning if defaultFs is not set when running fs commands.

HADOOP-12045: Enable LocalFileSystem#setTimes to change atime.

HADOOP-12049: Control http authentication cookie persistence via configuration.

HADOOP-12052:IPC client downgrades all exception types to IOE, breaks callers trying to use them.

HADOOP-12054: RPC client should not retry for InvalidToken exceptions.

HADOOP-12056: Use DirectoryStream in DiskChecker#checkDirs to detect errors when listing a directory.

HADOOP-12058: Fix dead links to DistCp and Hadoop Archives pages.

HADOOP-12059: S3Credentials should support use of CredentialProvider.

HADOOP-12061: Incorrect command in single cluster setup document.

HADOOP-12073: Azure FileSystem PageBlobInputStream does not return -1 on EOF.

HADOOP-12074: in Shell.java#runCommand() rethrow InterruptedException as InterruptedIOException

HADOOP-12076: Incomplete Cache Mechanism in CredentialProvider API.

HADOOP-12089: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.

HADOOP-12095: org.apache.hadoop.fs.shell.TestCount fails.

HADOOP-12097: Allow port range to be specified while starting webapp.

HADOOP-12100: ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermission.

HADOOP-12103: Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing.

HADOOP-12107: long running apps may have a huge number of StatisticsData instances under FileSystem

HADOOP-12119: hadoop fs -expunge does not work for federated namespace

HADOOP-12158: Improve error message in TestCryptoStreamsWithOpensslAesCtrCryptoCodec when OpenSSL is not installed.

HADOOP-12159: Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces

HADOOP-12161: Add getStoragePolicy API to the FileSystem interface

HADOOP-12169:ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse.

HADOOP-12185: NetworkTopology is not efficient adding/getting/removing nodes.

HADOOP-12186: ActiveStandbyElector shouldn't call monitorLockNodeAsync multiple times

HADOOP-12189: Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.

HADOOP-12191: Bzip2Factory is not thread safe.

HADOOP-12213: Interrupted exception can occur when Client#stop is called.

HADOOP-12229:Fix inconsistent subsection titles in filesystem.md.

HADOOP-12230: hadoop-project declares duplicate, conflicting curator dependencies.

HADOOP-12232: Upgrade Tomcat dependency to 6.0.44.

HADOOP-12239: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.

HADOOP-12242: Add in-page TOC to filesystem specification pages.

HADOOP-12252: LocalDirAllocator should not throw NPE with empty string configuration.

HADOOP-12258: Need translate java.nio.file.NoSuchFileException to FileNotFoundException to avoid regression.

HADOOP-12259: Utility to Dynamic port allocation

HADOOP-12269: Update aws-sdk dependency to 1.10.6

HADOOP-12280: Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded

HADOOP-12284: UserGroupInformation doAs can throw misleading exception

HADOOP-12291: Add support for nested groups in LdapGroupsMapping.

HADOOP-12292: Make use of DeleteObjects optional.

HADOOP-12296: when setnetgrent returns 0 in linux, exception should be thrown.

HADOOP-12304: Applications using FileContext fail with the default file system configured to be wasb/s3/etc.

HADOOP-12317: Applications fail on NM restart on some linux distro because NM container recovery declares AM container as LOST

HADOOP-12324: Better exception reporting in SaslPlainServer.

HADOOP-12325: RPC Metrics : Add the ability track and log slow RPCs.

HADOOP-12334: Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries.

HADOOP-12345: Pad hostname correctly in CredentialsSys.java.

HADOOP-12346: Increase some default timeouts / retries for S3a connector.

HADOOP-12348: MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.

HADOOP-12350: WASB Logging: Improve WASB Logging around deletes, reads and writes.

HADOOP-12358: Add -safely flag to rm to prompt when deleting many files.

HADOOP-12359: hadoop fs -getmerge doc is wrong.

HADOOP-12366: expose calculated paths.

HADOOP-12374: Description of hdfs expunge command is confusing.

HADOOP-12386: RetryPolicies.RETRY_FOREVER should be able to specify a retry interval.

HADOOP-12404: Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class.

HADOOP-12406: Fixed AbstractMapWritable.readFields to use the thread's ClassLoader to load class instead of System ClassLoader.

HADOOP-12407: Test failing: hadoop.ipc.TestSaslRPC.

HADOOP-12413: AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.

HADOOP-12415: Add io.netty dependency to hadoop-nfs and to hadoop-hdfs, needed to build Bigtop successfuly, see BIGTOP-2049.

HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.

HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.

HADOOP-12423: Handle failure of registering shutdownhook by ShutdownHookManager in static block

HADOOP-12426: Add Entry point for Kerberos health check.

HADOOP-12437: Allow SecurityUtil to lookup alternate hostnames.

HADOOP-12438: TestLocalFileSystem tests can fail on Windows after HDFS-8767 fix for handling pipe.

HADOOP-12440: TestRPC#testRPCServerShutdown did not produce the desired thread states before shutting down.

HADOOP-12441: Fixed kill-command behavior to work correctly across OSes by using bash shell built-in.

HADOOP-12442: Display help if the command option to 'hdfs dfs' is not valid

HADOOP-12444: Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel.

HADOOP-12449: TestDNS and TestNetUtils failing if no network.

HADOOP-12453: Support decoding KMS Delegation Token with its own Identifier.

HADOOP-12457: [JDK8] Fix a failure of compiling common by javadoc.

HADOOP-12463: Fix TestShell.testGetSignalKillCommand failure on windows.

HADOOP-12464: Interrupted client may try to fail-over and retry.

HADOOP-12465: Incorrect javadoc in WritableUtils.java.

HADOOP-12469: distcp should not ignore the ignoreFailures option.

HADOOP-12472: Make GenericTestUtils.assertExceptionContains robust.

HADOOP-12478: Shell.getWinUtilsPath() has been renamed Shell.getWinutilsPath().

HADOOP-12481: JWTRedirectAuthenticationHandler doesn't Retain Original Query String.

HADOOP-12482: Race condition in JMX cache update.

HADOOP-12483: Maintain wrapped SASL ordering for postponed IPC responses.

HADOOP-12484: Single File Rename Throws Incorrectly In Potential Race Condition Scenarios.

HADOOP-12508: delete fails with exception when lease is held on blob.

HADOOP-12520: Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests.

HADOOP-12526: there are duplicate dependency definitions in pom's

HADOOP-12533: Introduce FileNotFoundException in WASB for read and seek API.

HADOOP-12535: Run FileSystem contract tests with hadoop-azure.

HADOOP-12537: S3A to support Amazon STS temporary credentials.

HADOOP-12540: TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown.

HADOOP-12542: TestDNS fails on Windows after HADOOP-12437.

HADOOP-12545: Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp.

HADOOP-12548: Read s3a creds from a Credential Provider.

HADOOP-12551: Introduce FileNotFoundException for WASB FileSystem API

HADOOP-12555: WASB to read credentials from a credential provider.

HADOOP-12559: KMS connection failures should trigger TGT renewal.

HADOOP-12564: Upgrade JUnit3 TestCase to JUnit 4 in org.apache.hadoop.io package.

HADOOP-12565: Replace DSA with RSA for SSH key type in SingleCluster.md.

HADOOP-12568: Update core-default.xml to describe posixGroups support.

HADOOP-12570: HDFS Secure Mode Documentation updates.

HADOOP-12573: TestRPC.testClientBackOff failing.

HADOOP-12577: Bump up commons-collections version to 3.2.2 to address a security flaw.

HADOOP-12588: Fix intermittent test failure of TestGangliaMetrics.

HADOOP-12589: Fix intermittent test failure of TestCopyPreserveFlag

HADOOP-12590: TestCompressorDecompressor failing without stack traces

HADOOP-12602: TestMetricsSystemImpl#testQSize occasionally fails.

HADOOP-12603: TestSymlinkLocalFSFileContext#testSetTimesSymlinkToDir occasionally fail.

HADOOP-12604: Exception may be swallowed in KMSClientProvider.

HADOOP-12605: Fix intermittent failure of TestIPC.testIpcWithReaderQueuingi.

HADOOP-12608: Fix exception message in WASB when connecting with anonymous credential.

HADOOP-12609: Fix intermittent failure of TestDecayRpcScheduler.

HADOOP-12613: TestFind.processArguments occasionally fails.

HADOOP-12622: Improve the loggings in RetryPolicies and RetryInvocationHandler.

HADOOP-12634: Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy.

HADOOP-12635: Adding Append API support for WASB.

HADOOP-12636: Prevent ServiceLoader failure init for unused FileSystems.

HADOOP-12653: Use SO_REUSEADDR to avoid getting "Address already in use" when using kerberos and attempting to bind to any port on the local IP address.

HADOOP-12659: Incorrect usage of config parameters in token manager of KMS.

HADOOP-12663: Remove Hard-Coded Values From FileSystem.java.

HADOOP-12666: Support Microsoft Azure Data Lake - as a file system in Hadoop.

HADOOP-12667:s3a to support createNonRecursive API.

HADOOP-12668: Support excluding weak Ciphers in HttpServer2 through ssl- server.conf.

HADOOP-12672: RPC timeout should not override IPC ping interval

HADOOP-12675: Fix description about retention period in usage of expunge command.

HADOOP-12678: Handle empty rename pending metadata file during atomic rename in redo path.

HADOOP-12682: Fix TestKMS#testKMSRestart* failure.

HADOOP-12688: Fix deadlinks in Compatibility.md.

HADOOP-12689: S3 filesystem operations stopped working correctly

HADOOP-12691: Add CSRF Filter for REST APIs to Hadoop Common.

HADOOP-12691: Move files to correct location.

HADOOP-12696: Add tests for S3Filesystem Contract

HADOOP-12699: TestKMS#testKMSProvider intermittently fails during 'test rollover draining'.

HADOOP-12700: Remove unused import in TestCompressorDecompressor.java.

HADOOP-12706: TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally.

HADOOP-12711: Remove dependency on commons-httpclient for ServletUtil

HADOOP-12715: TestValueQueue#testgetAtMostPolicyALL fails intermittently.

HADOOP-12716: KerberosAuthenticator#doSpnegoSequence use incorrect class to determine isKeyTab in JDK8.

HADOOP-12717: NPE when trying to rename a directory in Windows Azure Storage FileSystem.

HADOOP-12723:S3A: Add ability to plug in any AWSCredentialsProvider.

HADOOP-12731: Remove useless boxing/unboxing code.

HADOOP-12736: TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out.

HADOOP-12738: Create unit test to automatically compare Common related classes and core-default.xml.

HADOOP-12751: While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple.

HADOOP-12752: Improve diagnostics/use of envvar/sysprop credential propagation.

HADOOP-12758: Extend CSRF Filter with UserAgent Checks.

HADOOP-12761: incremental maven build is not really incremental

HADOOP-12765: HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections.

HADOOP-12767: Update apache httpclient version to 4.5.2; httpcore to 4.4.4.

HADOOP-12772: NetworkTopologyWithNodeGroup.getNodeGroup() can loop infinitely for invalid 'loc' values.

HADOOP-12773: HBase classes fail to load with client/job classloader enabled

HADOOP-12774: s3a should use UGI.getCurrentUser.getShortname() for username.

HADOOP-12776: Remove getaclstatus call for non-acl commands in getfacl.

HADOOP-12780: During WASB atomic rename handle crash when one directory has been renamed but not file under it.

HADOOP-12782: Faster LDAP group name resolution with ActiveDirectory.

HADOOP-12786: "hadoop key" command usage is not documented.

HADOOP-12787: KMS SPNEGO sequence does not work with WEBHDFS.

HADOOP-12788: OpensslAesCtrCryptoCodec should log which random number generator is used.

HADOOP-12789: log classpath of ApplicationClassLoader at INFO level.

HADOOP-12792: TestUserGroupInformation#testGetServerSideGroups fails in chroot.

HADOOP-12793: Write a new group mapping service guide

HADOOP-12794: Support additional compression levels for GzipCodec.

HADOOP-12795: KMS does not log detailed stack trace for unexpected errors.

HADOOP-12800: Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs.

HADOOP-12801: Suppress obsolete S3FileSystem tests.

HADOOP-12804: Read Proxy Password from Credential Providers in S3 FileSystem.

HADOOP-12805: Annotate CanUnbuffer with @InterfaceAudience.Public

HADOOP-12807: S3AFileSystem should read AWS credentials from environment variables.

HADOOP-12810: FileSystem#listLocatedStatus causes unnecessary RPC calls

HADOOP-12817: Enable TLS v1.1 and 1.2

HADOOP-12825: Log slow name resolutions.

HADOOP-12828: Print user when services are started.

HADOOP-12829: StatisticsDataReferenceCleaner swallows interrupt exceptions

HADOOP-12831: LocalFS/FSOutputSummer NPEs in constructor if bytes per checksum set to 0

HADOOP-12844:Recover when S3A fails on IOException in read().

HADOOP-12846: Credential Provider Recursive Dependencies.

HADOOP-12847: hadoop daemonlog should support https and SPNEGO for Kerberized cluster.

HADOOP-12849: TestSymlinkLocalFSFileSystem fails intermittently.

HADOOP-12851: S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders.

HADOOP-12853: Change WASB documentation regarding page blob support.

HADOOP-12870: Fix typo admininistration in CommandsManual.md.

HADOOP-12871: Fix dead link to NativeLibraries.html in CommandsManual.md.

HADOOP-12872: Fix formatting in ServiceLevelAuth.md.

HADOOP-12875: [Azure Data Lake] Support for contract test and unit test cases.

HADOOP-12878: KMS SPNEGO sequence does not work with WEBHDFS.

HADOOP-12886: Exclude weak ciphers in SSLFactory through ssl-server.xml.

HADOOP-12891: S3AFileSystem should configure Multipart Copy threshold and chunk size.

HADOOP-12893: Update LICENSE.txt and NOTICE.txt.

HADOOP-12893: Update LICENSE.txt and NOTICE.txt.

HADOOP-12893: Update LICENSE.txt and NOTICE.txt".

HADOOP-12895: SSLFactory#createSSLSocketFactory exception message is wrong.

HADOOP-12901: Add warning log when KMSClientProvider cannot create a connection to the KMS server.

HADOOP-12902: JavaDocs for SignerSecretProvider are out-of-date in AuthenticationFilter.

HADOOP-12903: IPC Server should allow suppressing exception logging by type,.

HADOOP-12903: IPC Server should allow suppressing exception logging by type, not log 'server too busy' messages.

HADOOP-12906: AuthenticatedURL should convert a 404/Not Found into an FileNotFoundException.

HADOOP-12916: Allow RPC scheduler/callqueue backoff using response times.

HADOOP-12926: lz4.c does not detect 64-bit mode properly.

HADOOP-12929: JWTRedirectAuthenticationHandler must accommodate null expiration time.

HADOOP-12943: Add -w -r options in dfs -test command.

HADOOP-12947: Update documentation Hadoop Groups Mapping to add static group mapping, negative cache.

HADOOP-12950: ShutdownHookManager should have a timeout for each of the Registered shutdown hook.

HADOOP-12954: Add a way to change hadoop.security.token.service.use_ip

HADOOP-12958: PhantomReference for filesystem statistics can trigger OOM.

HADOOP-12962: KMS key names are incorrectly encoded when creating key.

HADOOP-12963:Allow using path style addressing for accessing the s3 endpoint.

HADOOP-12964: Http server vulnerable to clickjacking.

HADOOP-12973: Make DU pluggable.

HADOOP-12975: Add jitter to CachingGetSpaceUsed's thread

HADOOP-12977:s3a to handle delete("/", true) robustly.

HADOOP-12982:Document missing S3A and S3 properties.

HADOOP-12984: Add GenericTestUtils.getTestDir method and use it for emporary directory in tests.

HADOOP-12985: Support MetricsSource interface for DecayRpcScheduler Metrics.

HADOOP-12989: Some tests in org.apache.hadoop.fs.shell.find occasionally time out.

HADOOP-12993: Change ShutdownHookManger complete shutdown log from INFO to DEBUG.

HADOOP-12994: Specify PositionedReadable, add contract tests, fix problems.

HADOOP-13008: Add XFS Filter for UIs to Hadoop Common.

HADOOP-13011: Clearly Document the Password Details for Keystore-based Credential Providers

HADOOP-13018: Make Kdiag check whether hadoop.token.files points to existent and valid files.

HADOOP-13024: Distcp with -delete feature on raw data not implemented.

HADOOP-13026: Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator.

HADOOP-13028:add low level counter metrics for S3A; use in read performance tests.

HADOOP-13030: Handle special characters in passwords in KMS startup script.

HADOOP-13037: Refactor Azure Data Lake Store as an independent FileSystem.

HADOOP-13039: Add documentation for configuration property ipc.maximum.data.length

HADOOP-13042: Restore lost leveldbjni LICENSE and NOTICE changes.

HADOOP-13047:S3a Forward seek in stream length to be configurable.

HADOOP-13052: ChecksumFileSystem mishandles crc file permissions.

HADOOP-13058:S3A FS fails during init against a read-only FS if multipart purge.

HADOOP-13065: Add a new interface for retrieving FS and FC Statistics

HADOOP-13072: WindowsGetSpaceUsed constructor should be public

HADOOP-13079: Add -q option to Ls to print ? instead of non-printable characters.

HADOOP-13084: Fix ASF License warnings in branch-2.7.

HADOOP-13098: Dynamic LogLevel setting page should accept case-insensitive log level string.

HADOOP-13103: Group resolution from LDAP may fail on javax.naming.ServiceUnavailableException.

HADOOP-13105: Support timeouts in LDAP queries in LdapGroupsMapping.

HADOOP-13113:Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel.

HADOOP-13116:Jets3tNativeS3FileSystemContractTest does not run.

HADOOP-13122:Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.

HADOOP-13130: s3a failures can surface as RTEs, not IOEs.

HADOOP-13131: Add tests to verify that S3A supports SSE-S3 encryption.

HADOOP-13138: Unable to append to a SequenceFile with Compression.NONE.

HADOOP-13139:Branch-2: S3a to use thread pool that blocks clients.

HADOOP-13140: FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes.

HADOOP-13145:In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.

HADOOP-13146: Refactor RetryInvocationHandler.

HADOOP-13155: Implement TokenRenewer to renew and cancel delegation tokens in KMS.

HADOOP-13158:S3AFileSystem#toString might throw NullPointerException due to null cannedACL.

HADOOP-13159: Fix potential NPE in Metrics2 source for DecayRpcScheduler.

HADOOP-13162: Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs.

HADOOP-13163:Reuse pre-computed filestatus in Distcp-CopyMapper

HADOOP-13164:Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories.

HADOOP-13168: Support Future.get with timeout in ipc async calls

HADOOP-13169: Randomize file list in SimpleCopyListing.

HADOOP-13171: Add StorageStatistics to S3A; instrument some more operations.

HADOOP-13179: GenericOptionsParser is not thread-safe because commons-cli OptionBuilder is not thread-safe.

HADOOP-13183: S3A proxy tests fail after httpclient/httpcore upgrade.

HADOOP-13188:S3A file-create should throw error rather than overwrite directories.

HADOOP-13192: org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly.

HADOOP-13197: Add non-decayed call metrics for DecayRpcScheduler.

HADOOP-13202: Avoid possible overflow in org.apache.hadoop.util.bloom.BloomFilter#getNBytes.

HADOOP-13203: S3a: Consider reducing the number of connection aborts by setting correct length in s3 request.

HADOOP-13207: Specify FileSystem listStatus, listFiles and RemoteIterator.

HADOOP-13208: S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories.

HADOOP-13212:Provide an option to set the socket buffers in S3AFileSystem

HADOOP-13213: Small Documentation bug with AuthenticatedURL in hadoop-auth.

HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials.

HADOOP-13239:Deprecate s3:// in branch-2.

HADOOP-13241: document s3a better.

HADOOP-13242: Authenticate to Azure Data Lake using client ID and keys.

HADOOP-13244: o.a.h.ipc.Server#Server should honor handlerCount when queueSizePerHandler is specified in consturctor.

HADOOP-13249: RetryInvocationHandler need wrap InterruptedException in IOException when call Thread.sleep.

HADOOP-13251: Authenticate with Kerberos credentials when renewing KMS delegation token.

HADOOP-13252: Tune S3A provider plugin mechanism.

HADOOP-13254: Create framework for configurable disk checkers

HADOOP-13255: KMSClientProvider should check and renew tgt when doing delegation token operations.

HADOOP-13257: Improve Azure Data Lake contract tests.

HADOOP-13263: Reload cached groups in background after expiry.

HADOOP-13270: BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks.

HADOOP-13280: FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps

HADOOP-13283: Support reset operation for new global storage statistics and per FS storage stats.

HADOOP-13285: DecayRpcScheduler MXBean should only report decayed CallVolumeSummary.

HADOOP-13287: TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +.

HADOOP-13288: Guard null stats key in FileSystemStorageStatistics

HADOOP-13290: Appropriate use of generics in FairCallQueue.

HADOOP-13291: Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented.

HADOOP-13299: JMXJsonServlet is vulnerable to TRACE.

HADOOP-13305: Define common statistics names across schemes.

HADOOP-13309: Document S3A known limitations in file ownership and permission model.

HADOOP-13315: FileContext#umask is not initialized properly.

HADOOP-13316: Enforce Kerberos authentication for required ops in DelegationTokenAuthenticator.

HADOOP-13320: Fix arguments check in documentation for WordCount v2.0.

HADOOP-13323: Downgrade stack trace on FS load from Warn to debug.

HADOOP-13324: s3a tests don't authenticate with S3 frankfurt

HADOOP-13336:S3A to support per-bucket configuration.

HADOOP-13350: Additional fix to LICENSE and NOTICE.

HADOOP-13351: TestDFSClientSocketSize buffer size tests are flaky

HADOOP-13352: Make X-FRAME-OPTIONS configurable in HttpServer2.

HADOOP-13353: LdapGroupsMapping getPassward shouldn't return null when IOException throws.

HADOOP-13354: Update WASB driver to use the latest version

HADOOP-13368: DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation.

HADOOP-13381: KMS clients should use KMS Delegation Tokens from current UGI.

HADOOP-13382: Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, HADOOP-openstack.

HADOOP-13387:users always get told off for using S3 —even when not using it.

HADOOP-13389:TestS3ATemporaryCredentials.testSTS error when using IAM credentials.

HADOOP-13392: [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured.

HADOOP-13393: Omit unsupported fs.defaultFS setting in ADLS documentation.

HADOOP-13403: AzureNativeFileSystem rename/delete performance improvements.

HADOOP-13405:doc for fs.s3a.acl.default indicates incorrect values.

HADOOP-13406:S3AFileSystem: Consider reusing filestatus in delete() and mkdirs().

HADOOP-13422: ZKDelegationTokenSecretManager JaasConfig does not work well with other ZK users in process.

HADOOP-13433:Race in UGI.reloginFromKeytab.

HADOOP-13434: Add quoting to Shell class.

HADOOP-13443: KMS should check the type of underlying keyprovider of KeyProviderExtension before falling back to default.

HADOOP-13446: Support running isolated unit tests separate from AWS integration tests.

HADOOP-13447: Refactor S3AFileSystem to support introduction of separate metadata repository and tests.

HADOOP-13458: LoadBalancingKMSClientProvider#doOp should log IOException stacktrace.

HADOOP-13459: hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time.

HADOOP-13461: NPE in KeyProvider.rollNewVersion.

HADOOP-13466: Add an AutoCloseableLock class.

HADOOP-13467: Shell#getSignalKillCommand should use the bash builtin on Linux.

HADOOP-13470: GenericTestUtils$LogCapturer is flaky.

HADOOP-13494: ReconfigurableBase can log sensitive information.

HADOOP-13502: Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests.

HADOOP-13512: ReloadingX509TrustManager should keep reloading in case of exception.

HADOOP-13513: Java 1.7 support for org.apache.hadoop.fs.azure testcases.

HADOOP-13535: Add jetty6 acceptor startup issue workaround to branch-2.

HADOOP-13540:improve section on troubleshooting s3a auth problems.

HADOOP-13541:explicitly declare the Joda time version S3A depends on.

HADOOP-13546: Override equals and hashCode to avoid connection leakage

HADOOP-13552: RetryInvocationHandler logs all remote exceptions.

HADOOP-13552: RetryInvocationHandler logs all remote exceptions.

HADOOP-13558: UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket.

HADOOP-13560: S3ABlockOutputStream to support huge

HADOOP-13565: KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request.

HADOOP-13590: Retry until TGT expires even if the UGI renewal thread encountered exception.

HADOOP-13599: s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown.

HADOOP-13614: Purge some superfluous/obsolete S3 FS tests that are slowing test runs down.

HADOOP-13638: KMS should set UGI's Configuration object properly.

HADOOP-13674: S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.

HADOOP-13675: Bug in return value for delete() calls in WASB.

HADOOP-13680: fs.s3a.readahead.range to use getLongBytes.

HADOOP-13686: Adding additional unit test for Trash

HADOOP-13692: hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.

HADOOP-13707: If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed.

HADOOP-13710: Supress CachingGetSpaceUsed from logging interrupted exception stacktrace.

HADOOP-13716: Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup.

HADOOP-13727: S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider.

HADOOP-13735:ITestS3AFileContextStatistics.testStatistics() failing.

HADOOP-13737: Cleanup DiskChecker interface.

HADOOP-13738: DiskChecker should perform some disk IO.

HADOOP-13742: Expose NumOpenConnectionsPerUser as a metric. Brahma Reddy Battula.

HADOOP-13749: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.

HADOOP-13782: Make MutableRates metrics thread-local write, aggregate-on-read.

HADOOP-13792: Stackoverflow for schemeless defaultFS with trailing slash.

HADOOP-13801:regression: ITestS3AMiniYarnCluster failing.

HADOOP-13804: MutableStat mean loses accuracy if add(long, long) is used.

HADOOP-13822: Use GlobalStorageStatistics.INSTANCE.reset() at FileSystem#clearStatistics(). Contribute by Brahma Reddy Battula .

HADOOP-13823: s3a rename: fail if dest file exists.

HADOOP-13824: FsShell can suppress the real error if no error message is present.

HADOOP-13826: S3A Deadlock in multipart copy due to thread pool limits.

HADOOP-13830:Intermittent failure of ITestS3NContractRootDir.testRecursiveRootListing "Can not create a Path from an empty string".

HADOOP-13831: Correct check for error code to detect Azure Storage Throttling and provide retries.

HADOOP-13838: KMSTokenRenewer should close providers

HADOOP-13847: KMSWebApp should close KeyProviderCryptoExtension.

HADOOP-13855: Fix a couple of the s3a statistic names to be consistent with the rest.

HADOOP-13857: S3AUtils.translateException to map

HADOOP-13863: Azure: Add a new SAS key mode for WASB.

HADOOP-13871: ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful.

HADOOP-13890: Maintain HTTP/host as SPNEGO SPN support and fix KerberosName parsing.

HADOOP-13930: Azure: Add Authorization support to WASB.

HADOOP-13945: Azure: Add Kerberos and Delegation token support to WASB client.

HADOOP-13956: Read ADLS credentials from Credential Provider.

HADOOP-13962: Update ADLS SDK to 2.1.4.

HADOOP-13988: KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser.

HADOOP-14017: User friendly name for ADLS user and group.

HADOOP-14019: Fix some typos in the s3a docs.

HADOOP-14028: S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.

HADOOP-14029: Fix KMSClientProvider for non-secure proxyuser use case.

HADOOP-14048: REDO operation of WASB#AtomicRename should create placeholder blob for destination folder.

HADOOP-14049: Honour AclBit flag associated to file/folder permission for Azure datalake account.

HADOOP-14055: SwiftRestClient includes pass length in exception if auth fails.

HADOOP-14081:S3A: Consider avoiding array copy in S3ABlockOutputStream

HADOOP-14091: AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari.

HADOOP-14100: Upgrade Jsch jar to latest version to fix vulnerability in old versions.

HADOOP-14138:Remove S3A ref from META-INF service discovery, rely on existing core-default entry.

HADOOP-14150: Implement getHomeDirectory() method in NativeAzureFileSystem.

HADOOP-14214: DomainSocketWatcher::add()/delete() should not self interrupt while looping await().

HDFS-2043: TestHFlush failing intermittently.

HDFS-2580: NameNode#main(...) can make use of GenericOptionsParser.

HDFS-27: HDFS CLI with --config set to default config complains log file not found error.

HDFS-3677: dfs.namenode.edits.dir.required is missing from hdfs- default.xml.

HDFS-3918: EditLogTailer shouldn't log WARN when other node is in standby mode.

HDFS-4015: Safemode should count and report orphaned blocks.

HDFS-4176: EditLogTailer should call rollEdits with a timeout.

HDFS-4185: Add a metric for number of active leases

HDFS-4366: Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks.

HDFS-4396: Add START_MSG/SHUTDOWN_MSG for ZKFC

HDFS-4937: ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom.

HDFS-4946: Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable.

HDFS-5640: Add snapshot methods to FileContext.

HDFS-6054: MiniQJMHACluster should not use static port to avoid binding failure in unit test.

HDFS-6101: TestReplaceDatanodeOnFailure fails occasionally.

HDFS-6184: Capture NN's thread dump when it fails over.

HDFS-6407: Add sorting and pagination in the datanode tab of the NN Web UI.

HDFS-6481: DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs.

HDFS-6533: TestBPOfferService#testBasicFunctionalitytest fails intermittently.

HDFS-6581: Support for writing to single replica in RAM. Applied HDFS-6581.merge.15.patch to Champlain. This is a consolidated patch containing all feature changes and was applied to trunk.

HDFS-6666: Abort NameNode and DataNode startup if security is enabled but block access token is not enabled.

HDFS-6763: Initialize file system-wide quota once on transitioning to active

HDFS-6832: Fix the usage of 'hdfs namenode' command.

HDFS-6860: BlockStateChange logs are too noisy.

HDFS-6945: BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed.

HDFS-7163: WebHdfsFileSystem should retry reads according to the configured retry policy.

HDFS-7166: SbNN Web UI shows #Under replicated blocks and #pending deletion blocks.

HDFS-7284: Add more debug info to BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas.

HDFS-7314: When the DFSClient lease cannot be renewed, abort open-for- write files rather than the entire DFSClient.

HDFS-7390: Provide JMX metrics per storage type.

HDFS-7452: skip StandbyException log for getCorruptFiles()

HDFS-7483: Display information per tier on the Namenode UI.

HDFS-7597: DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping.

HDFS-7608: hdfs dfsclient newConnectedPeer has no write timeout

HDFS-7609: Avoid retry cache collision when Standby NameNode loading edits.

HDFS-7645: Rolling upgrade is restoring blocks from trash multiple times

HDFS-7701: Support reporting per storage type quota and usage with hadoop/hdfs shell.

HDFS-7725: Incorrect "nodes in service" metrics caused all writes to fail.

HDFS-7758: Retire FsDatasetSpi#getVolumes() and use FsDatasetSpi#getVolumeRefs() instead

HDFS-7890: Improve information on Top users for metrics in RollingWindowsManager and lower log level.

HDFS-7923: The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages

HDFS-7928: Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy.

HDFS-7933: fsck should also report decommissioning replicas.

HDFS-7964: Add support for async edit logging.

HDFS-7978: Add LOG.isDebugEnabled() guard for some LOG.debug(..).

HDFS-7990: IBR delete ack should not be delayed.

HDFS-8008: Support client-side back off when the datanodes are congested.

HDFS-8008: Support client-side back off when the datanodes are congested.

HDFS-8009: Signal congestion on the DataNode.

HDFS-8009: Signal congestion on the DataNode.

HDFS-8046: Allow better control of getContentSummary

HDFS-8055: NullPointerException when topology script is missing.

HDFS-8071:(Contributed by Tsz Wo Nicholas Sze).

HDFS-8099: Change "DFSInputStream has been closed already" message to debug log level.

HDFS-8101: DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS classes at runtime.

HDFS-8113: Add check for null BlockCollection pointers in BlockInfoContiguous structures.

HDFS-8131: Implement a space balanced block placement policy.

HDFS-8144: Split TestLazyPersistFiles into multiple tests.

HDFS-8152: Refactoring of lazy persist storage cases.

HDFS-8155: Support OAuth2 in WebHDFS.

HDFS-8180: AbstractFileSystem Implementation for WebHdfs.

HDFS-8205: CommandFormat#parse() should not parse option as value of option.

HDFS-8209: Support different number of datanode directories in MiniDFSCluster.

HDFS-8211: DataNode UUID is always null in the JMX counter.

HDFS-8219: setStoragePolicy with folder behavior is different after cluster restart.

HDFS-8224: Schedule a block for scanning if its metadata file is corrupt.

HDFS-8229: LAZY_PERSIST file gets deleted after NameNode restart.

HDFS-8232: Missing datanode counters when using Metrics2 sink interface.

HDFS-8247: TestDiskspaceQuotaUpdate#testAppendOverTypeQuota is failing.

HDFS-8276: LazyPersistFileScrubber should be disabled if scrubber interval configured zero.

HDFS-8278: When computing max-size-to-move in Balancer, count only the storage with remaining >= default block size.

HDFS-8309: Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows.

HDFS-8310: Fix TestCLI.testAll "help: help for find" on Windows.

HDFS-8311: DataStreamer.transfer() should timeout the socket InputStream.

HDFS-8337: Accessing httpfs via webhdfs doesn't work from a jar with kerberos.

HDFS-8345: Storage policy APIs must be exposed via the FileSystem interface.

HDFS-8384: Allow NN to startup if there are files having a lease but are not under construction.

HDFS-8431: hdfs crypto class not found in Windows.

HDFS-8432: Introduce a minimum compatible layout version to allow downgrade in more rolling upgrade use cases.

HDFS-8435: Support CreateFlag in WebHDFS.

HDFS-8492: DN should notify NN when client requests a missing block

HDFS-8539: Hdfs doesnt have class 'debug' in windows.

HDFS-8542: WebHDFS getHomeDirectory behavior does not match specification.

HDFS-8546: Use try with resources in DataStorage and Storage.

HDFS-8548: Minicluster throws NPE on shutdown.

HDFS-8549: Abort the balancer if an upgrade is in progress.

HDFS-8554: TestDatanodeLayoutUpgrade fails on Windows.

HDFS-8573: Move creation of restartMeta file logic from BlockReceiver to ReplicaInPipeline.

HDFS-8578: On upgrade, Datanode should process all storage/data dirs in parallel.

HDFS-8581: ContentSummary on / skips further counts on yielding lock

HDFS-8593: Calculation of effective layout version mishandles comparison to current layout version in storage.

HDFS-8615: Correct HTTP method in WebHDFS document.

HDFS-8633: Fix setting of dfs.datanode.readahead.bytes in hdfs-default.xml to match DFSConfigKeys.

HDFS-8642: Make TestFileTruncate more reliable.

HDFS-8647: Abstract BlockManager's rack policy into BlockPlacementPolicy.

HDFS-8656: Preserve compatibility of ClientProtocol#rollingUpgrade after finalization.

HDFS-8659: Block scanner INFO message is spamming logs.

HDFS-8674: Improve performance of postponed block scans.

HDFS-8676: Delayed rolling upgrade finalization can cause heartbeat expiration.

HDFS-8696: Make the lower and higher watermark in the DN Netty server configurable.

HDFS-8709: Clarify automatic sync in FSEditLog#logEdit.

HDFS-8722: Optimize datanode writes for small writes and flushes.

HDFS-8729: Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately which occasionally failed.

HDFS-8767: RawLocalFileSystem.listStatus() returns null for UNIX pipefile.

HDFS-8772: Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails.

HDFS-8778: TestBlockReportRateLimiting#testLeaseExpiration can deadlock.

HDFS-8785: TestDistributedFileSystem is failing in trunk.

HDFS-8792: BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory.

HDFS-8797: WebHdfsFileSystem creates too many connections for pread.

HDFS-8806: Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared

HDFS-8809: HDFS fsck reports under construction blocks as CORRUPT.

HDFS-8815: DFS getStoragePolicy implementation using single RPC call

HDFS-8816: Improve visualization for the Datanode tab in the NN UI.

HDFS-8818: Changes the global moveExecutor to per datanode executors and changes MAX_SIZE_TO_MOVE to be configurable.

HDFS-8824: Do not use small blocks for balancing the cluster.

HDFS-8826: In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes.

HDFS-8828: Utilize Snapshot diff report to build diff copy list in distcp.

HDFS-8829: Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning.

HDFS-8831: Trash Support for deletion in HDFS encryption zone.

HDFS-8844: TestHDFSCLI does not cleanup the test directory

HDFS-8845: DiskChecker should not traverse the entire tree

HDFS-8846: Add a unit test for INotify functionality across a layout version upgrade

HDFS-8850: VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks.

HDFS-8852: HDFS architecture documentation of version 2.x is outdated about append write support.

HDFS-8855: Webhdfs client leaks active NameNode connections.

HDFS-8856: Make LeaseManager#countPath O(1).

HDFS-8859: Improve DataNode ReplicaMap memory footprint to save about 45%.

HDFS-8860: Remove unused Replica copyOnWrite code

HDFS-8863: The remaining space check in BlockPlacementPolicyDefault is flawed.

HDFS-8867: Enable optimized block reports.

HDFS-8879: Quota by storage type usage incorrectly initialized upon namenode restart.

HDFS-8880: NameNode metrics logging.

HDFS-8883: NameNode Metrics : Add FSNameSystem lock Queue Length.

HDFS-8884: Fail-fast check in BlockPlacementPolicyDefault#chooseTarget.

HDFS-8885: ByteRangeInputStream used in webhdfs does not override available().

HDFS-8887: Expose storage type and storage ID in BlockLocation.

HDFS-8891: HDFS concat should keep srcs order.

HDFS-8896: DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager.

HDFS-8898: Create API and command-line argument to get quota and quota usage without detailed content summary.

HDFS-8911: NameNode Metric : Add Editlog counters as a JMX metric.

HDFS-8914: Document HA support in the HDFS HdfsDesign.md.

HDFS-8923: Add -source flag to balancer usage message.

HDFS-8930: Block report lease may leak if the 2nd full block report comes when NN is still in safemode

HDFS-8932: NPE thrown in NameNode when try to get TotalSyncCount metric before editLogStream initialization.

HDFS-8939: Test(S)WebHdfsFileContextMainOperations failing on branch-2.

HDFS-8946: Improve choosing datanode storage for block placement.

HDFS-8950: NameNode refresh doesn't remove DataNodes that are no longer in the allowed list.

HDFS-8964: When validating the edit log, do not read at or beyond the file offset that is being written

HDFS-8965: Harden edit log reading code against out of memory errors

HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.

HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.

HDFS-8983: NameNode support for protected directories.

HDFS-8983: NameNode support for protected directories.

HDFS-8995: Flaw in registration bookeeping can make DN die on reconnect.

HDFS-9008: Balancer#Parameters class could use a builder pattern.

HDFS-9008:was not backported correctly.

HDFS-9009: Send metrics logs to NullAppender by default.

HDFS-9019: Adding informative message to sticky bit permission denied exception.

HDFS-9033: dfsadmin -metasave prints "NaN" for cache used%.

HDFS-9038: DFS reserved space is erroneously counted towards non-DFS used.

HDFS-9042: Update document for the Storage policy name

HDFS-9043: Doc updation for commands in HDFS Federation

HDFS-9047: Retire libwebhdfs.

HDFS-9048: DistCp documentation is out-of-dated

HDFS-9063: Correctly handle snapshot path for getContentSummary.

HDFS-9072: Fix random failures in TestJMXGet.

HDFS-9082: Change the log level in WebHdfsFileSystem.initialize() from INFO to DEBUG.

HDFS-9083: Replication violates block placement policy.

HDFS-9106: Transfer failure during pipeline recovery causes permanent write failures.

HDFS-9107: Prevent NNs unrecoverable death spiral after full GC

HDFS-9112: Improve error message for Haadmin when multiple name service IDs are configured.

HDFS-9128: TestWebHdfsFileContextMainOperations and TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows.

HDFS-9130: Use GenericTestUtils#setLogLevel to the logging level.

HFDS-9137:should be backported correctly.

HDFS-9142: Separating Configuration object for namenode(s) in MiniDFSCluster.

HDFS-9145: Tracking methods that hold FSNamesytemLock for too long.

HDFS-9148: Incorrect assert message in TestWriteToReplica#testWriteToTemporary

HDFS-9175: Change scope of 'AccessTokenProvider.getAccessToken()' and 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to public.

HDFS-9178: Slow datanode I/O can cause a wrong node to be marked bad.

HDFS-9184: Logging HDFS operation's caller context into audit logs.

HDFS-9198: Coalesce IBR processing in the NN.

HDFS-9205: Do not schedule corrupt blocks for replication.

HDFS-9210: Fix some misuse of %n in VolumeScanner#printStats.

HDFS-9220: Reading small file

HDFS-9221: HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array.

HDFS-9226: Unit Tests failing due to mockito version issue.

HDFS-9239: DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness.

HDFS-9251: Refactor TestWriteToReplica and TestFsDatasetImpl to avoid explicitly creating Files in the tests code.

HDFS-9259: Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario.

HDFS-9264: Minor cleanup of operations on FsVolumeList#volumes.

HDFS-9273: ACLs on root directory may be lost after NN restart.

HDFS-9276: Failed to Update HDFS Delegation Token for long running application in HA mode.

HDFS-9289: Make DataStreamer#block thread safe and verify genStamp in commitBlock.

HDFS-9290: DFSClient#callAppend() is not backward compatible for slightly older NameNodes.

HDFS-9294: DFSClient deadlock when close file and failed to renew lease.

HDFS-9305: Delayed heartbeat processing causes storm of subsequent heartbeats.

HDFS-9311: Support optional offload of NameNode HA service health checks to a separate RPC server.

HDFS-9313: Possible NullPointerException in BlockManager if no excess replica can be chosen.

HDFS-9314: Improve BlockPlacementPolicyDefault's picking of excess replicas.

HDFS-9317: Document fsck -blockId and -storagepolicy options in branch-2.7.

HDFS-9343: Empty caller context considered invalid.

HDFS-9347: Invariant assumption in TestQuorumJournalManager.shutdown() is wrong

HDFS-9350: Avoid creating temprorary strings in Block.toString() and getBlockName()

HDFS-9354: Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows.

HDFS-9358: TestNodeCount#testNodeCount timed out.

HDFS-9362: TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.

HDFS-9364: Unnecessary DNS resolution attempts when creating NameNodeProxies.

HDFS-9383: TestByteArrayManager#testByteArrayManager fails.

HDFS-9384: TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server.

HDFS-9395: Make HDFS audit logging consistant.

HDFS-9397: Fix typo for readChecksum() LOG.warn in BlockSender.java.

HDFS-9402: Switch DataNode.LOG to use slf4j.

HDFS-9406: FSImage may get corrupted after deleting snapshot.

HDFS-9412: getBlocks occupies FSLock and takes too long to complete.

HDFS-9413: getContentSummary() on standby should throw StandbyException.

HDFS-9415: Document dfs.cluster.administrators and dfs.permissions.superusergroup.

HDFS-9426: Rollingupgrade finalization is not backward compatible.

HDFS-9428: Fix intermittent failure of TestDNFencing.testQueueingWithAppend.

HDFS-9431: DistributedFileSystem#concat fails if the target path is relative.

HDFS-9434: Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messags.

HDFS-9445: Datanode may deadlock while handling a bad volume.

HDFS-9466: TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky

HDFS-9467: Fix data race accessing writeLockHeldTimeStamp in FSNamesystem.

HDFS-9470: Encryption zone on root not loaded from fsimage after NN restart. Xiao Chen via wang.

HDFS-9476: TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail

HDFS-9478: Reason for failing ipc.FairCallQueue contruction should be thrown.

HDFS-9484: NNThroughputBenchmark$BlockReportStats should not send empty block reports.

HDFS-9493: Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk.

HDFS-9500: Fix software version counts for DataNodes during rolling upgrade.

HDFS-9505: HDFS Architecture documentation needs to be refreshed.

HDFS-9516: Truncate file fails with data dirs on multiple disks.

HDFS-9517: Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes

HDFS-9521: TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk.

HDFS-9525: hadoop utilities need to support provided delegation tokens.

HDFS-9530: ReservedSpace is not cleared for abandoned Blocks

HDFS-9533: seen_txid in the shared edits directory is modified during bootstrapping.

HDFS-9534: Add CLI command to clear storage policy from a path.

HDFS-9549: TestCacheDirectives#testExceedsCapacity is flaky

HDFS-9555: LazyPersistFileScrubber should still sleep if there are errors in the clear progress.

HDFS-9557: Reduce object allocation in PB conversion.

HDFS-9566: Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method

HDFS-9569: Log the name of the fsimage being loaded for better supportability.

HDFS-9572: Prevent DataNode log spam if a client connects on the data transfer port but sends no data.

HDFS-9574: Reduce client failures during datanode restart.

HDFS-9584: NPE in distcp when ssl configuration file does not exist in class path.

HDFS-9589: Block files which have been hardlinked should be duplicated before the DataNode appends to the them

HDFS-9600: Do not check replication if the block is under construction.

HDFS-9605: Add links to failed volumes to explorer.html in HDFS Web UI.

HDFS-9608: Disk IO imbalance in HDFS with heterogeneous storages.

HDFS-9612: DistCp worker threads are not terminated after jobs are done.

HDFS-9618: Fix mismatch between log level and guard in BlockManager#computeRecoveryWorkForBlocks

HDFS-9619: SimulatedFSDataset sometimes can not find blockpool for the correct namenode

HDFS-9623: Update example configuration of block state change log in log4j.properties.

HDFS-9624: DataNode start slowly due to the initial DU command operations.

HDFS-9625: set replication for empty file failed when set storage policy

HDFS-9626: TestBlockReplacement#testBlockReplacement fails occasionally.

HDFS-9629: Update the footer of Web UI to show year 2016

HDFS-9630: DistCp minor refactoring and clean up.

HDFS-9634: webhdfs client side exceptions don't provide enough details.

HDFS-9638: to branch-2.7.

HDFS-9644: Update encryption documentation to reflect nested EZs.

HDFS-9655: NN should start JVM pause monitor before loading fsimage.

HDFS-9661: Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage moveBlockAcrossStorage and createRbw.

HDFS-9669: TcpPeerServer should respect ipc.server.listen.queue.size

HDFS-9670: DistCp throws NPE when source is root.

HDFS-9672: o.a.h.hdfs.TestLeaseRecovery2 fails intermittently

HDFS-9688: Test the effect of nested encryption zones in HDFS downgrade.

HDFS-9696: Garbage snapshot records linger forever.

HDFS-9701: DN may deadlock when hot-swapping under load.

HDFS-9706: Log more details in debug logs in BlockReceiver's constructor.

HDFS-9710: DN can be configured to send block receipt IBRs in batches.

HDFS-9711: Integrate CSRF prevention filter in WebHDFS.

HDFS-9713: DataXceiver#copyBlock should return if block is pinned.

HDFS-9715: Check storage ID uniqueness on datanode startup

HDFS-9721: Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference

HDFS-9724: Degraded performance in WebHDFS listing as it does not reuse ObjectMapper.

HDFS-9726: Refactor IBR code to a new class.

HDFS-9730: Storage ID update does not happen when there is a layout change.

HDFS-9739: DatanodeStorage.isValidStorageId() is broken.

HDFS-9740: Use a reasonable limit in DFSTestUtil.waitForMetric().

HDFS-9743: Fix TestLazyPersistFiles#testFallbackToDiskFull in branch-2.7.

HDFS-9748: Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice.

HDFS-9752: Permanent write failures may happen to slow writers during datanode rolling upgrades

HDFS-9760: WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler

HDFS-9764: DistCp doesn't print value for several arguments including -numListstatusThreads.

HDFS-9765: TestBlockScanner#testVolumeIteratorWithCaching fails intermittently.

HDFS-9766: TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently.

HDFS-9768: Reuse ObjectMapper instance in HDFS to improve the performance.

HDFS-9772: TestBlockReplacement#testThrottler doesn't work as expected.

HDFS-9777: Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh).

HDFS-9779: . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value.

HDFS-9781: FsDatasetImpl#getBlockReports can occasionally throw NullPointerException.

HDFS-9784: Example usage is not correct in Transparent Encryption document.

HDFS-9790: HDFS Balancer should exit with a proper message if upgrade is not finalized

HDFS-9797: Log Standby exceptions thrown by RequestHedgingProxyProvider at DEBUG Level

HDFS-9799: Reimplement getCurrentTrashDir to remove incompatibility.

HDFS-9812: Streamer threads leak if failure happens when closing DFSOutputStream.

HDFS-9839: Reduce verbosity of processReport logging.

HDFS-9842: dfs.datanode.balance.bandwidthPerSec should accept friendly size units.

HDFS-9844: Correct path creation in getTrashRoot to handle root dir.

HDFS-9851: NameNode throws NPE when setPermission is called on a path that does not exist.

HDFS-9854: Log cipher suite negotiation more verbosely.

HDFS-9864: Correct reference for RENEWDELEGATIONTOKEN and CANCELDELEGATIONTOKEN in webhdfs doc.

HDFS-9865: TestBlockReplacement fails intermittently in trunk

HDFS-9871: "Bytes Being Moved" -ve(-1 B) when cluster was already balanced.

HDFS-9874: Long living DataXceiver threads cause volume shutdown to block.

HDFS-9880: TestDatanodeRegistration fails occasionally.

HDFS-9881: DistributedFileSystem#getTrashRoot returns incorrect path for encryption zones.

HDFS-9882: Add heartbeatsTotal in Datanode metrics.

HDFS-9902: Support different values of dfs.datanode.du.reserved per storage type.

HDFS-9904: testCheckpointCancellationDuringUpload occasionally fails.

HDFS-9905: WebHdfsFileSystem#runWithRetry should display original stack trace on error.

HDFS-9906: Remove spammy log spew when a datanode is restarted.

HDFS-9917: IBR accumulate more objects when SNN was down for sometime.

HDFS-9926: MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils.

HDFS-9941: Do not log StandbyException on NN, other minor logging fixes.

HDFS-9947: Block#toString should not output information from derived classes

HDFS-9949: Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared

HDFS-9958: BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages.

HDFS-10178: Permanent write failures can happen if pipeline recoveries occur for the first packet.

HDFS-10182: Hedged read might overwrite user's buf.

HDFS-10186: DirectoryScanner: Improve logs by adding full path of both actual and expected block directories.

HDFS-10189: PacketResponder#toString should include the downstreams for PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE

HDFS-10199: Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing.

HDFS-10216: Distcp -diff throws exception when handling relative path.

HDFS-10217: show 'blockScheduled' tooltip in datanodes table.

HDFS-10223: peerFromSocketAndKey performs SASL exchange before setting connection timeouts.

HDFS-10225: DataNode hot swap drives should disallow storage type changes.

HDFS-10228: TestHDFSCLI fails.

HDFS-10235: Last contact for Live Nodes should be relative time.

HDFS-10239: Fsshell mv fails if port usage doesn't match in src and destination paths.

HDFS-10245: Fix the findbugs warnings in branch-2.7.

HDFS-10253: Fix TestRefreshCallQueue failure

HDFS-10261: TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports.

HDFS-10264: Logging improvements in FSImageFormatProtobuf.Saver.

HDFS-10270: TestJMXGet:testNameNode() fails.

HDFS-10271: Extra bytes are getting released from reservedSpace for append

HDFS-10275: TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly.

HDFS-10277: PositionedReadable test testReadFullyZeroByteFile failing in HDFS.

HDFS-10279: Improve validation of the configured number of tolerated failed volumes.

HDFS-10281: TestPendingCorruptDnMessages fails intermittently.

HDFS-10291: TestShortCircuitLocalRead failing

HDFS-10301: BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order .

HDFS-10301: Remove FBR tracking state to fix false zombie storage detection for interleaving block reports.

HDFS-10309: Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga).

HDFS-10312: Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.

HDFS-10313: Distcp need to enforce the order of snapshot names passed to -diff.

HDFS-10319: Balancer should not try to pair storages with different types.

HDFS-10320: Rack failures may result in NN terminate.

HDFS-10324: Trash directory in an encryption zone should be pre-created with correct permissions.

HDFS-10329: Bad initialisation of StringBuffer in RequestHedgingProxyProvider.

HDFS-10330: Add Corrupt Blocks Information in Metasave output.

HDFS-10335:Mover$Processor#chooseTarget() always chooses the first matching target storage group.

HDFS-10341: Add a metric to expose the timeout number of pending replication blocks

HDFS-10342: BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt.

HDFS-10343: BlockManager#createLocatedBlocks may return blocks on failed storages.

HDFS-10344: DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash.

HDFS-10347: Namenode report bad block method doesn't log the bad block or datanode.

HDFS-10360: DataNode may format directory and lose blocks if current/VERSION is missing.

HDFS-10367: TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception.

HDFS-10372: Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume.

HDFS-10377: CacheReplicationMonitor shutdown log message should use INFO level

HDFS-10390: Implement asynchronous setAcl/getAclStatus for DistributedFileSystem

HDFS-10396: Using -diff option with DistCp may get "Comparison method violates its general contract" exception.

HDFS-10397: Distcp should ignore -delete option if -diff option is provided instead of exiting.

HDFS-10410: RedundantEditLogInputStream.LOG is set to wrong class.

HDFS-10415: TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called

HDFS-10417: Improve error message from checkBlockLocalPathAccess.

HDFS-10423: Increase default value of httpfs maxHttpHeaderSize.

HDFS-10424: DatanodeLifelineProtocol not able to use under security cluster

HDFS-10431: Refactor and speedup TestAsyncDFSRename

HDFS-10437: ReconfigurationProtocol not covered by HDFSPolicyProvider.

HDFS-10438: When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.

HDFS-10440: Improve DataNode web UI

HDFS-10448: CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1

HDFS-10455: Logging the username when deny the setOwner operation.

HDFS-10457: DataNode should not auto-format block pool directory if VERSION is missing.

HDFS-10458: getFileEncryptionInfo should return quickly for non-encrypted cluster.

HDFS-10468: HDFS read ends up ignoring an interrupt.

HDFS-10469: Add number of active xceivers to datanode metrics.

HDFS-10471: DFSAdmin#SetQuotaCommand's help msg is not correct.

HDFS-10474: hftp copy fails when file name with Chinese+special char in branch-2

HDFS-10481: HTTPFS server should correctly impersonate as end user to open file.

HDFS-10485: Fix findbugs warning in FSEditLog.java

HDFS-10488: Update WebHDFS documentation regarding CREATE and MKDIR default permissions.

HDFS-10493: Add links to datanode web UI in namenode datanodes page.

HDFS-10499: TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails Intermittently.

HDFS-10508: DFSInputStream should set thread's interrupt status after catching InterruptException from sleep.

HDFS-10512: VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks.

HDFS-10525: Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap

HDFS-10544: Balancer doesn't work with IPFailoverProxyProvider.

HDFS-10556: DistCpOptions should be validated automatically.

HDFS-10569: A bug causes OutOfIndex error in BlockListAsLongs.

HDFS-10579: HDFS web interfaces lack configs for X-FRAME-OPTIONS protection.

HDFS-10589: Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params.

HDFS-10609: Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications.

HDFS-10617: PendingReconstructionBlocks.size() should be synchronized.

HDFS-10623: Remove unused import of httpclient.HttpConnection from TestWebHdfsTokens.

HDFS-10625: VolumeScanner to report why a block is found bad.

HDFS-10627: Volume Scanner marks a block as "suspect" even if the exception is network-related.

HDFS-10642: TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently.

HDFS-10643: [Unit Test] Namenode should use loginUser(hdfs) to generateEncryptedKey.

HDFS-10643: HDFS namenode should always use service user

HDFS-10653: Optimize conversion from path string to components.

HDFS-10655: Fix path related byte array conversion bugs.

HDFS-10660: Expose storage policy apis via HDFSAdmin interface.

HDFS-10684: WebHDFS DataNode calls fail without parameter createparent.

HDFS-10688: BPServiceActor may run into a tight loop for sending block report when hitting IOException.

HDFS-10703: HA NameNode Web UI should show last checkpoint time.

HDFS-10710: In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui.

HDFS-10713: Throttle FsNameSystem lock warnings.

HDFS-10715: NPE when applying AvailableSpaceBlockPlacementPolicy.

HDFS-10716: In Balancer, the target task should be removed when its size < 0.

HDFS-10735:Distcp using webhdfs on secure HA clusters fails with StandbyException .

HDFS-10752: Several log refactoring/improvement suggestion in HDFS.

HDFS-10757: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.

HDFS-10763: Open files can leak permanently due to inconsistent lease update.

HDFS-10763: Open files can leak permanently due to inconsistent lease update TestTrashWithSecureEncryptionZones diverged from HDFS-10906.

HDFS-10764: Fix INodeFile#getBlocks to not return null.

HDFS-10793: Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184. HDFS-9700. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets for DataTransferProtocol.

HDFS-10798: Make the threshold of reporting FSNamesystem lock contention configurable.

HDFS-10817: Add Logging for Long-held NN Read Locks.

HDFS-10822: Log DataNodes in the write pipeline. John Zhuge via Lei Xu.

HDFS-10913: Introduce fault injectors to simulate slow mirrors.

HDFS-10917: Collect peer performance statistics on DataNode.

HDFS-10922: Adding additional unit tests for Trash

HDFS-10930: Refactor: Wrap Datanode IO related operations.

HDFS-10934: TestDFSShell#testStat fails intermittently.

HDFS-10956: Remove rename/delete performance penalty when not using snapshots.

HDFS-10959: Update DataNode to use DatasetVolumeChecker.

HDFS-10962: TestRequestHedgingProxyProvider is flaky.

HDFS-10963: Reduce log level when network topology cannot find enough datanodes.

HDFS-10965: Add unit test for HDFS command 'dfsadmin -printTopology'.

HDFS-10966: Enhance Dispatcher logic on deciding when to give up a source DataNode.

HDFS-10984: Expose nntop output as metrics.

HDFS-10987: Make Decommission less expensive when lot of blocks present.

HDFS-11003: Expose XmitsInProgress through DataNodeMXBean.

HDFS-11008: Change unit test for testing parsing "-source" parameter in Balancer CLI.

HDFS-11011: Add unit tests for HDFS command 'dfsadmin -set/clrSpaceQuota'.

HDFS-11015: Enforce timeout in balancer.

HDFS-11018: Incorrect check and message in FsDatasetImpl#invalidate.

HDFS-11031: Add additional unit test for DataNode startup behavior when volumes fail.

HDFS-11047: Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode.

HDFS-11054: Suppress verbose log message in BlockPlacementPolicyDefault.

HDFS-11056: Addendum fix for self-healing branch.

HDFS-11056: Concurrent append and read operations lead to checksum error.

HDFS-11069: Tighten the authorization of datanode RPC.

HDFS-11076: Add unit test for extended Acls.

HDFS-11083: Add unit test for DFSAdmin -report command.

HDFS-11087: NamenodeFsck should check if the output writer is still writable.

HDFS-11112: Journal Nodes should refuse to format non-empty directories.

HDFS-11114: Support for running async disk checks in DataNode.

HDFS-11119: Support for parallel checking of StorageLocations on DataNode startup.

HDFS-11122: TestDFSAdmin#testReportCommand fails due to timed out.

HDFS-11128: CreateEditsLog throws NullPointerException.

HDFS-11148: Update DataNode to use StorageLocationChecker at startup.

HDFS-11149: Fix build break.

HDFS-11149: Support for parallel checking of FsVolumes.

HDFS-11160: VolumeScanner reports write-in-progress replicas as corrupt incorrectly.

HDFS-11180: Intermittent deadlock in NameNode when failover happens.

HDFS-11182: Addendum fix for Fenton to fix a unit test.

HDFS-11182: Update DataNode to use DatasetVolumeChecker.

HDFS-11194: Maintain aggregated peer performance metrics on NameNode.

HDFS-11195: Return error when appending files by webhdfs rest api fails.

HDFS-11197: Listing encryption zones fails when deleting a EZ that is on a snapshotted directory.

HDFS-11209: SNN can't checkpoint when rolling upgrade is not finalized.

HDFS-11224: Lifeline message should be ignored for dead nodes

HDFS-11229: HDFS-11056 failed to close meta file.

HDFS-11240: Remove snapshot version of SDK dependency from Azure Data Lake Store File System.

HDFS-11251: ConcurrentModificationException during DataNode#refreshVolumes .

HDFS-11253: FileInputStream leak on failure path in BlockSender.

HDFS-11258: File mtime change could not save to editlog.

HDFS-11263: ClassCastException when we use Bzipcodec for Fsimage compression.

HDFS-11267: Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage.

HDFS-11274: Datanode should only check the failed volume upon IO errors.

HDFS-11279: Cleanup unused DataNode#checkDiskErrorAsync().

HDFS-11280: Allow WebHDFS to reuse HTTP connections to NN.

HDFS-11282: Document the missing metrics of DataNode Volume IO operations.

HDFS-11292: log lastWrittenTxId etc info in logSyncAll.

HDFS-11299: Support multiple Datanode File IO hooks.

HDFS-11307: The rpc to portmap service for NFS has hardcoded timeout.

HDFS-11337:

HDFS-11339: Support File IO sampling for Datanode IO profiling hooks.

HDFS-11369: Change exception message in StorageLocationChecker.

HDFS-11379: DFSInputStream may infinite loop requesting block locations.

HDFS-11387: Socket reuse address option is not honored in PrivilegedNfsGatewayStarter.

HDFS-11391: Numeric usernames do no work with WebHDFS FS write access.

HDFS-11395: RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode.

HDFS-11403: Zookeper ACLs on NN HA enabled clusters should be handled consistently.

HDFS-11461: DataNode Disk Outlier Detection.

HDFS-11462: Fix occasional BindException in TestNameNodeMetricsLogger.

HDFS-11476: Fix NPE in FsDatasetImpl#checkAndUpdate

HDFS-11477: Simplify file IO profiling configuration.

HDFS-11479: Socket re-use address option should be used in SimpleUdpServer.

HDFS-11508: Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket is in TIME_WAIT state.

HDFS-11517: Expose slow disks via DataNode JMX.

HDFS-11533: reuseAddress option should be used for child channels in Portmap and SimpleTcpServer.

MAPREDUCE-5485: Allow repeating job commit by extending OutputCommitter API

MAPREDUCE-6197: Cache MapOutputLocations in ShuffleHandler.

MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.

MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.

MAPREDUCE-6273: HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state.

MAPREDUCE-6302:

MAPREDUCE-6338: MR AppMaster does not honor ephemeral port range.

MAPREDUCE-6350: JobHistory doesn't support fully-functional search.

MAPREDUCE-6404: Allow AM to specify a port range for starting its webapp.

MAPREDUCE-6436: JobHistory cache issue.

MAPREDUCE-6436: JobHistory cache issue.

MAPREDUCE-6460: TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails.

MAPREDUCE-6485: Create a new task attempt with failed map task priority if in-progress attempts are unassigned.

MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.

MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.

MAPREDUCE-6514: Job hangs as ask is not updated after ramping down of all reducers.

MAPREDUCE-6543: Migrate MR Client test cases part 2.

MAPREDUCE-6549: multibyte delimiters with LineRecordReader cause duplicate records

MAPREDUCE-6566: Add retry support to mapreduce CLI tool.

MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.

MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.

MAPREDUCE-6579: TestNetworkedJob.testNetworkedJob failed due to YARN-3946.

MAPREDUCE-6593: TestJobHistoryEventHandler.testTimelineEventHandling fails because of NPE.

MAPREDUCE-6616: Fail to create jobhistory file if there are some multibyte characters in the job name.

MAPREDUCE-6618: YarnClientProtocolProvider leaking the YarnClient thread.(xgong).

MAPREDUCE-6621: Memory Leak in JobClient#submitJobInternal().

MAPREDUCE-6625: TestCLI#testGetJob fails occasionally

MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.

MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.

MAPREDUCE-6639: Process hangs in LocatedFileStatusFetcher if FileSystem.get throws.

MAPREDUCE-6672: TestTeraSort fails on Windows.

MAPREDUCE-6684: High contention on scanning of user directory under immediate_done in Job History Server.

MAPREDUCE-6689: MapReduce job can infinitely increase number of reducer resource requests.

MAPREDUCE-6693: ArrayIndexOutOfBoundsException occurs when the length of the job name is equal to mapreduce.jobhistory.jobname.limit.

MAPREDUCE-6701: application master log can not be available when clicking jobhistory's am logs link.

MAPREDUCE-6711: After enabling preemption on cluster, jobs failed without making any attempts made after being preempted.

MAPREDUCE-6724: Single shuffle to memory must not exceed Integer#MAX_VALUE.

MAPREDUCE-6762: ControlledJob#toString failed with NPE when job status is not successfully updated.

MAPREDUCE-6789: Fix TestAMWebApp failure.

MAPREDUCE-6792: Allow user's full principal name as owner of MapReduce staging directory in JobSubmissionFiles#JobStagingDir().

MAPREDUCE-6797: Job history server scans can become blocked on a single, slow entry.

MAPREDUCE-6852: Job#updateStatus() failed with NPE due to race condition.

YARN-857: Localization failures should be available in container diagnostics.

YARN-998: Keep NM resource updated through dynamic resource config for RM/NM restart.

YARN-1376: NM need to notify the log aggregation status to RM through heartbeat.

YARN-1402: Update related Web UI and CLI with exposing client API to check log aggregation status.

YARN-1462: AHS API and other AHS changes to handle tags for completed MR jobs.

YARN-1556: NPE getting application report with a null appId.

YARN-1880: Cleanup TestApplicationClientProtocolOnHA

YARN-1993: Cross-site scripting vulnerability in TextView.java.

YARN-2019: Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore.

YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.

YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.

YARN-2194: Fix bug causing CGroups functionality to fail on RHEL7.

YARN-2392: Add more diags about app retry limits on AM failures.

YARN-2495: Allow admin specify labels from each NM

YARN-2498: Respect labels in preemption policy of capacity scheduler for inter-queue preemption.

YARN-2571:RM to support YARN registry.

YARN-2619: Added NodeManager support for disk io isolation through cgroups.

YARN-2666: TestFairScheduler.testContinuousScheduling fails Intermittently.

YARN-2696: Queue sorting in CapacityScheduler should consider node label.

YARN-2740: Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled.

YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.

YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.

YARN-2868: FairScheduler: Metric for latency to allocate first container for an application.

YARN-2901: Add errors and warning metrics page to RM, NM web UI.

YARN-3021: YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp.

YARN-3026: Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp.

YARN-3102: Decommisioned Nodes not listed in Web UI.

YARN-3102: Decommisioned Nodes not listed in Web UI.

YARN-3110: Few issues in ApplicationHistory web ui.

YARN-3136: Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers.

YARN-3148: Allow CORS related headers to passthrough in WebAppProxyServlet.

YARN-3215: Respect labels in CapacityScheduler when computing headroom.

YARN-3216: Max-AM-Resource-Percentage should respect node labels.

YARN-3225: New parameter of CLI for decommissioning node gracefully in RMAdmin CLI.

YARN-3243: CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits.

YARN-3248: Display count of nodes blacklisted by apps in the web UI.

YARN-3266: RMContext#inactiveNodes should have NodeId as map key.

YARN-3269: Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path.

YARN-3293: Track and display capacity scheduler health metrics in web UI.

YARN-3294: Allow dumping of Capacity Scheduler debug logs via web UI for a fixed time period.

YARN-3305: Normalize AM resource request on app submission.

YARN-3318: Create Initial OrderingPolicy Framework and FifoOrderingPolicy.

YARN-3319: Implement a FairOrderingPolicy.

YARN-3326: Support RESTful API for getLabelsToNodes.

YARN-3343: Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate timeout.

YARN-3345: Add non-exclusive node label API.

YARN-3347: Improve YARN log command to get AMContainer logs as well as running containers logs.