HADOOP-7930: Kerberos relogin interval in UserGroupInformation should be configurable.

HADOOP-11180: Change log message "token.Token: Cannot find class for token kind kms-dt" to debug.

HADOOP-11780: Prevent IPC reader thread death.

HADOOP-12017: Hadoop archives command should use configurable replication factor when closing.

HADOOP-12054: RPC client should not retry for InvalidToken exceptions.

HADOOP-12097: Allow port range to be specified while starting webapp.

HADOOP-12185: NetworkTopology is not efficient adding/getting/removing nodes.

HADOOP-12386: RetryPolicies.RETRY_FOREVER should be able to specify a retry interval.

HADOOP-12404: Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class.

HADOOP-12442: Display help if the command option to 'hdfs dfs' is not valid.

HADOOP-12453: Support decoding KMS Delegation Token with its own Identifier.

HADOOP-12483: Maintain wrapped SASL ordering for postponed IPC responses.

HADOOP-12666: Support Microsoft Azure Data Lake - as a file system in Hadoop.

HADOOP-12735: core-default.xml misspells hadoop.workaround.non.threadsafe.getpwuid.

HADOOP-12738: Create unit test to automatically compare Common related classes and core-default.xml.

HADOOP-12875: [Azure Data Lake] Support for contract test and unit test cases.

HADOOP-13018: Make Kdiag check whether hadoop.token.files points to existent and valid files.

HADOOP-13037: Refactor Azure Data Lake Store as an independent FileSystem.

HADOOP-13202: Avoid possible overflow in org.apache.hadoop.util.bloom.BloomFilter#getNBytes.

HADOOP-13208: S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories.

HADOOP-13242: Authenticate to Azure Data Lake using client ID and keys.

HADOOP-13257: Improve Azure Data Lake contract tests.

HADOOP-13299: JMXJsonServlet is vulnerable to TRACE.

HADOOP-13392: [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured.

HADOOP-13393: Omit unsupported fs.defaultFS setting in ADLS documentation.

HADOOP-13433: Race in UGI.reloginFromKeytab.

HADOOP-13443: KMS should check the type of underlying keyprovider of KeyProviderExtension before falling back to default.

HADOOP-13458: LoadBalancingKMSClientProvider#doOp should log IOException stacktrace.

HADOOP-13461: NPE in KeyProvider.rollNewVersion.

HADOOP-13466: Add an AutoCloseableLock class.

HADOOP-13494: ReconfigurableBase can log sensitive information.

HADOOP-13552: RetryInvocationHandler logs all remote exceptions.

HADOOP-13558: UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket.

HADOOP-13558: UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket.

HADOOP-13558: UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket.

HADOOP-13590: Retry until TGT expires even if the UGI renewal thread encountered exception.

HADOOP-13638: KMS should set UGI's Configuration object properly.

HADOOP-13707: If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed.

HADOOP-13710: Supress CachingGetSpaceUsed from logging interrupted exception stacktrace.

HADOOP-13742: Expose NumOpenConnectionsPerUser as a metric. Brahma Reddy Battula.

HADOOP-13749: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.

HADOOP-13792: Stackoverflow for schemeless defaultFS with trailing slash.

HADOOP-13812: Upgrade Tomcat to 6.0.48.

HADOOP-13822: Use GlobalStorageStatistics.INSTANCE.reset() at FileSystem#clearStatistics(). Contribute by Brahma Reddy Battula.

HADOOP-13823: s3a rename: fail if dest file exists.

HADOOP-13824: FsShell can suppress the real error if no error message is present.

HADOOP-13838: KMSTokenRenewer should close providers.

HADOOP-13838: KMSTokenRenewer should close providers.

HADOOP-13847: KMSWebApp should close KeyProviderCryptoExtension.

HADOOP-13890: Maintain HTTP/host as SPNEGO SPN support and fix KerberosName parsing.

HADOOP-13930: Azure: Add Authorization support to WASB.

HADOOP-13930: Azure: Add Authorization support to WASB.

HADOOP-13943: TestCommonConfigurationFields#testCompareXmlAgainstConfigurationClass fails after HADOOP-13863.

HADOOP-13945: Azure: Add Kerberos and Delegation token support to WASB client.

HADOOP-13945: Azure: Add Kerberos and Delegation token support to WASB client.

HADOOP-13956: Read ADLS credentials from Credential Provider.

HADOOP-13962: Update ADLS SDK to 2.1.4.

HADOOP-13988: KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser.

HADOOP-14017: User friendly name for ADLS user and group.

HADOOP-14029: Fix KMSClientProvider for non-secure proxyuser use case.

HADOOP-14048: REDO operation of WASB#AtomicRename should create placeholder blob for destination folder.

HADOOP-14049: Honour AclBit flag associated to file/folder permission for Azure datalake account.

HADOOP-14055: SwiftRestClient includes pass length in exception if auth fails.

HADOOP-14059: typo in s3a rename(self, subdir) error message.

HADOOP-14100: Upgrade Jsch jar to latest version to fix vulnerability in old versions.

HDFS-3918: EditLogTailer shouldn't log WARN when other node is in standby mode.

HDFS-4176: EditLogTailer should call rollEdits with a timeout.

HDFS-6184: Capture NN's thread dump when it fails over.

HDFS-6763: Initialize file system-wide quota once on transitioning to active.

HDFS-7284: Add more debug info to BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas.

HDFS-7964: Add support for async edit logging.

HDFS-8224: Schedule a block for scanning if its metadata file is corrupt.

HDFS-8492: DN should notify NN when client requests a missing block.

HDFS-8674: Improve performance of postponed block scans.

HDFS-8792: BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory.

HDFS-9145: Tracking methods that hold FSNamesytemLock for too long.

HDFS-9193: Fix incorrect references the usages of the DN in dfshealth.js.

HDFS-9350: Avoid creating temprorary strings in Block.toString() and getBlockName().

HDFS-9467: Fix data race accessing writeLockHeldTimeStamp in FSNamesystem.

HDFS-9500: Fix software version counts for DataNodes during rolling upgrade.

HDFS-9525: hadoop utilities need to support provided delegation tokens.

HDFS-9696: Garbage snapshot records linger forever.

HDFS-9947: Block#toString should not output information from derived classes.

HDFS-10225: DataNode hot swap drives should disallow storage type changes.

HDFS-10301: Remove FBR tracking state to fix false zombie storage detection for interleaving block reports.

HDFS-10330: Add Corrupt Blocks Information in Metasave output.

HDFS-10342: BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt.

HDFS-10423: Increase default value of httpfs maxHttpHeaderSize.

HDFS-10455: Logging the username when deny the setOwner operation.

HDFS-10457: DataNode should not auto-format block pool directory if VERSION is missing.

HDFS-10512: VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks.

HDFS-10544: Balancer doesn't work with IPFailoverProxyProvider.

HDFS-10609: Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications.

HDFS-10625: VolumeScanner to report why a block is found bad.

HDFS-10627: Volume Scanner marks a block as "suspect" even if the exception is network-related.

HDFS-10684: WebHDFS DataNode calls fail without parameter createparent.

HDFS-10703: HA NameNode Web UI should show last checkpoint time.

HDFS-10710: In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.

HDFS-10713: Throttle FsNameSystem lock warnings.

HDFS-10722: Fix race condition in TestEditLog#testBatchedSyncWithClosedLogs.

HDFS-10735: Distcp using webhdfs on secure HA clusters fails with StandbyException.

HDFS-10752: Several log refactoring/improvement suggestion in HDFS.

HDFS-10763: Open files can leak permanently due to inconsistent lease update.

HDFS-10793: Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184.

HDFS-10798: Make the threshold of reporting FSNamesystem lock contention configurable.

HDFS-10817: Add Logging for Long-held NN Read Locks.

HDFS-10822: Log DataNodes in the write pipeline. John Zhuge via Lei Xu.

HDFS-10966: Enhance Dispatcher logic on deciding when to give up a source DataNode.

HDFS-10987: Make Decommission less expensive when lot of blocks present.

HDFS-11003: Expose XmitsInProgress through DataNodeMXBean.

HDFS-11015: Enforce timeout in balancer.

HDFS-11018: Incorrect check and message in FsDatasetImpl#invalidate.

HDFS-11047: Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode.

HDFS-11054: Suppress verbose log message in BlockPlacementPolicyDefault.

HDFS-11054: Suppress verbose log message in BlockPlacementPolicyDefault.

HDFS-11069: Tighten the authorization of datanode RPC.

HDFS-11087: NamenodeFsck should check if the output writer is still writable.

HDFS-11112: Journal Nodes should refuse to format non-empty directories.

HDFS-11128: CreateEditsLog throws NullPointerException.

HDFS-11160: VolumeScanner reports write-in-progress replicas as corrupt incorrectly.

HDFS-11195: Return error when appending files by webhdfs rest api fails.

HDFS-11240: Remove snapshot version of SDK dependency from Azure Data Lake Store File System.

HDFS-11258: File mtime change could not save to editlog.

HDFS-11263: ClassCastException when we use Bzipcodec for Fsimage compression.

HDFS-11280: Allow WebHDFS to reuse HTTP connections to NN.

HDFS-11391: Numeric usernames do no work with WebHDFS FS write access.

HDFS-11395: RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode.

MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.

MAPREDUCE-6338: MR AppMaster does not honor ephemeral port range.

MAPREDUCE-6350: JobHistory doesn't support fully-functional search.

MAPREDUCE-6404: Allow AM to specify a port range for starting its webapp.

MAPREDUCE-6485: Create a new task attempt with failed map task priority if in-progress attempts are unassigned.

MAPREDUCE-6616: Fail to create jobhistory file if there are some multibyte characters in the job name.

MAPREDUCE-6684: High contention on scanning of user directory under immediate_done in Job History Server.

MAPREDUCE-6693: ArrayIndexOutOfBoundsException occurs when the length of the job name is equal to mapreduce.jobhistory.jobname.limit.

MAPREDUCE-6698: Increase timeout on TestUnnecessaryBlockingOnHist oryFileInfo.testTwoThreadsQueryingDifferentJobOfSameUser.

MAPREDUCE-6701: application master log can not be available when clicking jobhistory's am logs link.

MAPREDUCE-6724: Single shuffle to memory must not exceed Integer#MAX_VALUE.

MAPREDUCE-6789: Fix TestAMWebApp failure.

MAPREDUCE-6797: Job history server scans can become blocked on a single, slow entry.

YARN-857: Localization failures should be available in container diagnostics.

YARN-1993: Cross-site scripting vulnerability in TextView.java.

YARN-3148: Allow CORS related headers to passthrough in WebAppProxyServlet.

YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.

YARN-3589: RM and AH web UI display DOCTYPE wrongly. Contbituted by Rohith.

YARN-3591: Resource localization on a bad disk causes subsequent containers failure.

YARN-4109: Exception on RM scheduler page loading with labels.

YARN-4113: RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER.

YARN-4403: (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period.

YARN-4496: Improve HA ResourceManager Failover detection on the client.

YARN-4697: NM aggregation thread pool is not bound by limits.

YARN-4767: Network issues can cause persistent RM UI outage.

YARN-5121: fix some container-executor portability issues.

YARN-5206: RegistrySecurity includes id:pass in exception text if considered invalid.

YARN-5208: Run TestAMRMClient TestNMClient TestYarnClient TestClientRMTokens TestAMAuthorization tests with hadoop.security.token.service.use_ip enabled.

YARN-5318: Fix intermittent test failure of TestRMAdminService#te stRefreshNodesResourceWithFileSystemBasedConfigurationProvider.

YARN-5353: ResourceManager can leak delegation tokens when they are shared across apps.

YARN-5456: container-executor support for FreeBSD, NetBSD, and others if conf path is absolute.

YARN-5539: TimelineClient failed to retry on java.net.SocketTimeoutException: Read timed out.

YARN-5549: AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately.

YARN-5555: Scheduler UI: "% of Queue" is inaccurate if leaf queue is hierarchically nested.

YARN-5659: getPathFromYarnURL should use standard methods.

YARN-5677: RM should transition to standby when connection is lost for an extended period.

YARN-5711: Propogate exceptions back to client when using hedging RM failover provider.

YARN-5718: TimelineClient (and other places in YARN) shouldn't over-write HDFS client retry settings which could cause unexpected behavior.

YARN-5873: RM crashes with NPE if generic application history is enabled.

YARN-5923: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled.

YARN-6145: Improve log message on fail over.

HADOOP-12954: Add a way to change hadoop.security.token.service.use_ip.

HADOOP-1540: Support file exclusion list in distcp.

HADOOP-3733: "s3x:" URLs break when Secret Key contains a slash, even if encoded.

HADOOP-5323: Trash documentation should describe its directory structure and configurations.

HADOOP-5470: RunJar.unJar() should write the last modified time found in the jar entry to the uncompressed file.

HADOOP-7139: Allow appending to existing SequenceFiles

HADOOP-7713: dfs -count -q should label output column.

HADOOP-7817: RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos().

HADOOP-8151: Error handling in snappy decompressor throws invalid exceptions.

HADOOP-8818: Use equals instead == in MD5MD5CRC32FileChecksum and TFileDumper.

HADOOP-8934: Shell command ls should include sort options.

HADOOP-9121: InodeTree.java has redundant check for vName while throwing exception.

HADOOP-9242: Duplicate surefire plugin config in HADOOP-common.

HADOOP-9477: Add posixGroups support for LDAP groups mapping service.

HADOOP-10035: Cleanup TestFilterFileSystem.

HADOOP-1023: Add more logging in WASB page blob read function.

HADOOP-1026: Logging bug in WASB causing out of index exception.

HADOOP-10300: Allowed deferred sending of call responses.

HADOOP-10365: BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block.

HADOOP-10406: TestIPC.testIpcWithReaderQueuing may fail.

HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.

HADOOP-10775: Shell operations to fail with meaningful errors on windows if winutils.exe not found.

HADOOP-10823: TestReloadingX509TrustManager is flaky.

HADOOP-10965: Print fully qualified path in CommandWithDestination error messages.

HADOOP-10971: Add -C flag to make `hadoop fs -ls` print filenames only.

HADOOP-11031: Design document for credential provider API.

HADOOP-11098: [JDK8] Max Non Heap Memory default changed between JDK7 and 8.

HADOOP-11212: NetUtils.wrapException to handle SocketException explicitly.

HADOOP-11218: Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.

HADOOP-11252: RPC client does not time out by default.

HADOOP-11262: Enable YARN to use S3A.

HADOOP-11361: Fix a race condition in MetricsSourceAdapter.updateJmxCache.

HADOOP-11404: Clarify the "expected client Kerberos principal is null" authorization message.

HADOOP-11491: HarFs incorrectly declared as requiring an authority.

HADOOP-11581: Multithreaded correctness Warnings #org.apache.hadoop.fs.shell.Ls.

HADOOP-11613: Remove commons-httpclient dependency from HADOOP-azure.

HADOOP-11628: SPNEGO auth does not work with CNAMEs in JDK8.

HADOOP-11685: StorageException complaining "no lease ID" during HBase distributed log splitting.

HADOOP-11687: Ignore x-* and response headers when copying an Amazon S3 object.

HADOOP-11692: Improve authentication failure WARN message to avoid user confusion.

HADOOP-11713: ViewFileSystem should support snapshot methods.

HADOOP-11717: Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth.

HADOOP-11772: RPC Invoker relies on static ClientCache which has synchronized(this) blocks.

HADOOP-11785: Reduce the number of listStatus operation in distcp buildListing.

HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm.

HADOOP-11812: Implement listLocatedStatus for ViewFileSystem to speed up split calculation.

HADOOP-11827: Speed-up distcp buildListing() using threadpool.

HADOOP-11859: PseudoAuthenticationHandler fails with httpcomponents v4.4.

HADOOP-11864: JWTRedirectAuthenticationHandler breaks java8 javadocs.

HADOOP-11876: Refactor code to make it more readable, minor maybePrintStats bug.

HADOOP-11901: BytesWritable fails to support 2G chunks due to integer overflow.

HADOOP-11918: Listing an empty s3a root directory throws FileNotFound.

HADOOP-11924: Tolerate JDK-8047340-related exceptions in Shell#isSetSidAvailable preventing class init.

HADOOP-11932: MetricsSinkAdapter may hang when being stopped.

HADOOP-11960: Enable Azure-Storage Client Side logging.

HADOOP-12001: Fixed LdapGroupsMapping to include configurable Posix UID and GID attributes during the search.

HADOOP-12006: Remove unimplemented option for from document.

HADOOP-12043: Display warning if defaultFs is not set when running fs commands.

HADOOP-12045: Enable LocalFileSystem#setTimes to change atime.

HADOOP-12049: Control http authentication cookie persistence via configuration.

HADOOP-12052: IPC client downgrades all exception types to IOE, breaks callers trying to use them.

HADOOP-12058: Fix dead links to DistCp and Hadoop Archives pages.

HADOOP-12059: S3Credentials should support use of CredentialProvider.

HADOOP-12061: Incorrect command in single cluster setup document.

HADOOP-12073: Azure FileSystem PageBlobInputStream does not return -1 on EOF.

HADOOP-12074: In Shell.java#runCommand() rethrow InterruptedException as InterruptedIOException.

HADOOP-12076: Incomplete cache mechanism in CredentialProvider API.

HADOOP-12089: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.

HADOOP-12095: org.apache.hadoop.fs.shell.TestCount fails.

HADOOP-12100: ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermission.

HADOOP-12103: Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing.

HADOOP-12107: Long running apps may have a huge number of StatisticsData instances under FileSystem.

HADOOP-12119: Hadoop fs -expunge does not work for federated namespace.

HADOOP-12159: Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces.

HADOOP-12161: Add getStoragePolicy API to the FileSystem interface.

HADOOP-12169: ListStatus on empty dir in S3A lists itself instead of returning an empty list.

HADOOP-12186: ActiveStandbyElector shouldn't call monitorLockNodeAsync multiple times.

HADOOP-12189: Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.

HADOOP-12191: Bzip2Factory is not thread safe.

HADOOP-12213: Interrupted exception can occur when Client#stop is called.

HADOOP-12230: HADOOP-project declares duplicate, conflicting curator dependencies.

HADOOP-12232: Upgrade Tomcat dependency to 6.0.44.

HADOOP-12239: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.

HADOOP-12258: Need translate java.nio.file.NoSuchFileException to FileNotFoundException to avoid regression.

HADOOP-12259: Utility to Dynamic port allocation.

HADOOP-12269: Update aws-sdk dependency to 1.10.6.

HADOOP-12280: Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded.

HADOOP-12284: UserGroupInformation doAs can throw misleading exception.

HADOOP-12291: Add support for nested groups in LdapGroupsMapping.

HADOOP-12292: Make use of DeleteObjects optional.

HADOOP-12296: When setnetgrent returns 0 in Linux, exception should be thrown.

HADOOP-12304: Applications using FileContext fail with the default file system configured to be wasb/s3/etc.

HADOOP-12317: Applications fail on NM restart on some Linux distro because NM container recovery declares AM container as LOST.

HADOOP-12324: Better exception reporting in SaslPlainServer.

HADOOP-12325: RPC Metrics : Add the ability track and log slow RPCs.

HADOOP-12334: Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries.

HADOOP-12345: Pad hostname correctly in CredentialsSys.java.

HADOOP-12346: Increase some default timeouts / retries for S3a connector.

HADOOP-12348: MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.

HADOOP-12350: WASB Logging: Improve WASB Logging around deletes, reads and writes.

HADOOP-12358: Add -safely flag to rm to prompt when deleting many files.

HADOOP-12359: hadoop fs -getmerge doc is wrong.

HADOOP-12366: Expose calculated paths (aw).

HADOOP-12374: Description of hdfs expunge command is confusing.

HADOOP-12406: Fixed AbstractMapWritable.readFields to use the thread's ClassLoader to load class instead of System ClassLoader.

HADOOP-12407: Test failing: hadoop.ipc.TestSaslRPC.

HADOOP-12413: AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.

HADOOP-12415: Add io.netty dependency to HADOOP-nfs and to HADOOP-hdfs, needed to build Bigtop successfully, see BIGTOP-2049.

HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.

HADOOP-12423: Handle failure of registering shutdownhook by ShutdownHookManager in static block

HADOOP-12426: Add Entry point for Kerberos health check.

HADOOP-12437: Allow SecurityUtil to lookup alternate hostnames.

HADOOP-12438: TestLocalFileSystem tests can fail on Windows after HDFS-8767 fix for handling pipe.

HADOOP-12440: TestRPC#testRPCServerShutdown did not produce the desired thread states before shutting down.

HADOOP-12441: Fixed kill-command behavior to work correctly across OSes by using bash shell built-in.

HADOOP-12444: Support lazy seek in S3AInputStream.

HADOOP-12463: Fix TestShell.testGetSignalKillCommand failure on windows.

HADOOP-12464: Interrupted client may try to fail-over and retry.

HADOOP-12465: Incorrect javadoc in WritableUtils.java.

HADOOP-12469: distcp should not ignore the ignoreFailures option.

HADOOP-12472: Make GenericTestUtils.assertExceptionContains robust.

HADOOP-12478: Shell.getWinUtilsPath() has been renamed Shell.getWinutilsPath().

HADOOP-12481: JWTRedirectAuthenticationHandler doesn't Retain Original Query String.

HADOOP-12482: Race condition in JMX cache update.

HADOOP-12484: Single File Rename Throws Incorrectly In Potential Race Condition Scenarios.

HADOOP-12508: Delete fails with exception when lease is held on blob.

HADOOP-12520: Use XInclude in HADOOP-azure test configuration to isolate Azure Storage account keys for service integration tests.

HADOOP-12526: There are duplicate dependency definitions in poms.

HADOOP-12533: Introduce FileNotFoundException in WASB for read and seek API.

HADOOP-12535: Run FileSystem contract tests with HADOOP-azure.

HADOOP-12537: S3A to support Amazon STS temporary credentials.

HADOOP-12540: TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown.

HADOOP-12542: TestDNS fails on Windows after HADOOP-12437.

HADOOP-12545: Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp.

HADOOP-12548: Read s3a creds from a Credential Provider.

HADOOP-12551: Introduce FileNotFoundException for WASB FileSystem API.

HADOOP-12555: WASB to read credentials from a credential provider.

HADOOP-12559: KMS connection failures should trigger TGT renewal.

HADOOP-12564: Upgrade JUnit3 TestCase to JUnit 4 in org.apache.hadoop.io package.

HADOOP-12565: Replace DSA with RSA for SSH key type in SingleCluster.md.

HADOOP-12568: Update core-default.xml to describe posixGroups support.

HADOOP-12570: HDFS Secure Mode Documentation updates.

HADOOP-12573: TestRPC.testClientBackOff failing.

HADOOP-12577: Bump up commons-collections version to 3.2.2 to address a security flaw.

HADOOP-12588: Fix intermittent test failure of TestGangliaMetrics.

HADOOP-12589: Fix intermittent test failure of TestCopyPreserveFlag.

HADOOP-12590: TestCompressorDecompressor failing without stack traces.

HADOOP-12602: TestMetricsSystemImpl#testQSize occasionally fails.

HADOOP-12603: TestSymlinkLocalFSFileContext#testSetTimesSymlinkToDir occasionally fail.

HADOOP-12604: Exception may be swallowed in KMSClientProvider.

HADOOP-12605: Fix intermittent failure of TestIPC.testIpcWithReaderQueuingi.

HADOOP-12608: Fix exception message in WASB when connecting with anonymous credential.

HADOOP-12609: Fix intermittent failure of TestDecayRpcScheduler.

HADOOP-12613: TestFind.processArguments occasionally fails.

HADOOP-12622: Improve the loggings in RetryPolicies and RetryInvocationHandler.

HADOOP-12635: Adding Append API support for WASB.

HADOOP-12636: Prevent ServiceLoader failure init for unused FileSystems.

HADOOP-12653: Use SO_REUSEADDR to avoid getting "Address already in use" when using kerberos and attempting to bind to any port on the local IP address.

HADOOP-12659: Incorrect usage of config parameters in token manager of KMS.

HADOOP-12663: Remove Hard-Coded Values From FileSystem.java.

HADOOP-12668: Support excluding weak Ciphers in HttpServer2 through ssl- server.conf.

HADOOP-12672: RPC timeout should not override IPC ping interval.

HADOOP-12675: Fix description about retention period in usage of expunge command.

HADOOP-12678: Handle empty rename pending metadata file during atomic rename in redo path.

HADOOP-12682: Fix TestKMS#testKMSRestart* failure.

HADOOP-12688: Fix dead links in Compatibility.md.

HADOOP-12689: S3 filesystem operations stopped working correctly.

HADOOP-12691: Add CSRF Filter for REST APIs to Hadoop Common.

HADOOP-12691: Move files to correct location.

HADOOP-12696: Add tests for S3Filesystem Contract.

HADOOP-12699: TestKMS#testKMSProvider intermittently fails during 'test rollover draining'.

HADOOP-12700: Remove unused import in TestCompressorDecompressor.java.

HADOOP-12706: TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally.

HADOOP-12711: Remove dependency on commons-httpclient for ServletUtil.

HADOOP-12715: TestValueQueue#testgetAtMostPolicyALL fails intermittently.

HADOOP-12716: KerberosAuthenticator#doSpnegoSequence use incorrect class to determine isKeyTab in JDK8.

HADOOP-12717: NPE when trying to rename a directory in Windows Azure Storage FileSystem.

HADOOP-12723: S3A: Add ability to plug in any AWSCredentialsProvider.

HADOOP-12731: Remove useless boxing/unboxing code.

HADOOP-12736: TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out.

HADOOP-12751: While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple.

HADOOP-12752: Improve diagnostics/use of envvar/sysprop credential propagation.

HADOOP-12758: Extend CSRF Filter with UserAgent Checks.

HADOOP-12761: Incremental maven build is not really incremental.

HADOOP-12767: Update apache httpclient version to 4.5.2; httpcore to 4.4.4.

HADOOP-12772: NetworkTopologyWithNodeGroup.getNodeGroup() can loop infinitely for invalid 'loc' values.

HADOOP-12773: HBase classes fail to load with client/job classloader enabled.

HADOOP-12776: Remove getaclstatus call for non-acl commands in getfacl.

HADOOP-12780: During WASB atomic rename handle crash when one directory has been renamed but not file under it.

HADOOP-12782: Faster LDAP group name resolution with ActiveDirectory.

HADOOP-12786: "hadoop key" command usage is not documented.

HADOOP-12787: KMS SPNEGO sequence does not work with WEBHDFS.

HADOOP-12788: OpensslAesCtrCryptoCodec should log which random number generator is used.

HADOOP-12789: Log classpath of ApplicationClassLoader at INFO level.

HADOOP-12792: TestUserGroupInformation#testGetServerSideGroups fails in chroot.

HADOOP-12793: Write a new group mapping service guide.

HADOOP-12794: Support additional compression levels for GzipCodec.

HADOOP-12795: KMS does not log detailed stack trace for unexpected errors.

HADOOP-12800: Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs.

HADOOP-12801: Suppress obsolete S3FileSystem tests.

HADOOP-12805: Annotate CanUnbuffer with @InterfaceAudience.Public

HADOOP-12807: S3AFileSystem should read AWS credentials from environment variables.

HADOOP-12810: FileSystem#listLocatedStatus causes unnecessary RPC calls

HADOOP-12817: Enable TLS v1.1 and 1.2.

HADOOP-12825: Log slow name resolutions.

HADOOP-12828: Print user when services are started.

HADOOP-12829: StatisticsDataReferenceCleaner swallows interrupt exceptions.

HADOOP-12831: LocalFS/FSOutputSummer NPEs in constructor if bytes per checksum set to 0.

HADOOP-12844: Recover when S3A fails on IOException in read().

HADOOP-12846: Credential Provider Recursive Dependencies.

HADOOP-12847: hadoop daemonlog should support https and SPNEGO for Kerberized cluster.

HADOOP-12849: TestSymlinkLocalFSFileSystem fails intermittently.

HADOOP-12851: S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders.

HADOOP-12853: Change WASB documentation regarding page blob support.

HADOOP-12870: Fix typo admininistration in CommandsManual.md.

HADOOP-12871: Fix dead link to NativeLibraries.html in CommandsManual.md.

HADOOP-12872: Fix formatting in ServiceLevelAuth.md.

HADOOP-12878: KMS SPNEGO sequence does not work with WEBHDFS.

HADOOP-12886: Exclude weak ciphers in SSLFactory through ssl-server.xml.

HADOOP-12891: S3AFileSystem should configure Multipart Copy threshold and chunk size.

HADOOP-12893: Update LICENSE.txt and NOTICE.txt.

HADOOP-12895: SSLFactory#createSSLSocketFactory exception message is wrong.

HADOOP-12901: Add warning log when KMSClientProvider cannot create a connection to the KMS server.

HADOOP-12902: JavaDocs for SignerSecretProvider are out-of-date in AuthenticationFilter.

HADOOP-12903: IPC Server should allow suppressing exception logging by type.

HADOOP-12903: IPC Server should allow suppressing exception logging by type, not log 'server too busy' messages.

HADOOP-12906: AuthenticatedURL should convert a 404/Not Found into an FileNotFoundException.

HADOOP-12909: Change ipc.Client to support asynchronous calls.

HADOOP-12916: Allow RPC scheduler/callqueue backoff using response times.

HADOOP-12926: lz4.c does not detect 64-bit mode properly.

HADOOP-12929: JWTRedirectAuthenticationHandler must accommodate null expiration time.

HADOOP-12943: Add -w -r options in dfs -test command.

HADOOP-12947: Update documentation Hadoop Groups Mapping to add static group mapping, negative cache.

HADOOP-12950: ShutdownHookManager should have a timeout for each of the Registered shutdown hook.

HADOOP-12957: Limit the number of outstanding async calls.

HADOOP-12958: PhantomReference for filesystem statistics can trigger OOM.

HADOOP-12962: KMS key names are incorrectly encoded when creating key.

HADOOP-12963: Allow using path style addressing for accessing the s3 endpoint.

HADOOP-12964: HTTP server vulnerable to clickjacking.

HADOOP-12973: Make DU pluggable.

HADOOP-12975: Add jitter to CachingGetSpaceUsed's thread.

HADOOP-12982: Document missing S3A and S3 properties.

HADOOP-12984: Add GenericTestUtils.getTestDir method and use it for temporary directory in tests.

HADOOP-12985: Support MetricsSource interface for DecayRpcScheduler Metrics.

HADOOP-12989: Some tests in org.apache.hadoop.fs.shell.find occasionally time out.

HADOOP-12993: Change ShutdownHookManger complete shutdown log from INFO to DEBUG.

HADOOP-12994: Specify PositionedReadable, add contract tests, fix problems.

HADOOP-13008: Add XFS Filter for UIs to Hadoop Common.

HADOOP-13011: Clearly Document the Password Details for Keystore-based Credential Providers.

HADOOP-13026: Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator.

HADOOP-13028: Add low level counter metrics for S3A; use in read performance tests.

HADOOP-13030: Handle special characters in passwords in KMS startup script.

HADOOP-13039: Add documentation for configuration property ipc.maximum.data.length

HADOOP-13042: Restore lost leveldbjni LICENSE and NOTICE changes.

HADOOP-13047: S3a Forward seek in stream length to be configurable.

HADOOP-13052: ChecksumFileSystem mishandles crc file permissions.

HADOOP-13058: S3A FS fails during init against a read-only FS if multipart purge.

HADOOP-13065: Add a new interface for retrieving FS and FC Statistics.

HADOOP-13072: WindowsGetSpaceUsed constructor should be public.

HADOOP-13079: Add -q option to Ls to print ? instead of non-printable characters.

HADOOP-13084: Fix ASF License warnings.

HADOOP-13098: Dynamic LogLevel setting page should accept case-insensitive log level string.

HADOOP-13103: Group resolution from LDAP may fail on javax.naming.ServiceUnavailableException.

HADOOP-13105: Support timeouts in LDAP queries in LdapGroupsMapping.

HADOOP-13113: Enable parallel test execution for HADOOP-aws.

HADOOP-13116: Jets3tNativeS3FileSystemContractTest does not run.

HADOOP-13122: Customize User-Agent header sent in HTTP requests by S3A.

HADOOP-13130: s3a failures can surface as RTEs, not IOEs.

HADOOP-13131: Add tests to verify that S3A supports SSE-S3 encryption.

HADOOP-13138: Unable to append to a SequenceFile with Compression.NONE.

HADOOP-13140: FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes.

HADOOP-13145: In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.

HADOOP-13146: Refactor RetryInvocationHandler.

HADOOP-13155: Implement TokenRenewer to renew and cancel delegation tokens in KMS.

HADOOP-13158: S3AFileSystem#toString might throw NullPointerException due to null cannedACL.

HADOOP-13159: Fix potential NPE in Metrics2 source for DecayRpcScheduler.

HADOOP-13162: Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs.

HADOOP-13163: Reuse pre-computed filestatus in Distcp-CopyMapper.

HADOOP-13168: Support Future.get with timeout in ipc async calls.

HADOOP-13171: Add StorageStatistics to S3A; instrument some more operations.

HADOOP-13179: GenericOptionsParser is not thread-safe because commons-cli OptionBuilder is not thread-safe.

HADOOP-13183: S3A proxy tests fail after httpclient/httpcore upgrade.

HADOOP-13192: org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly.

HADOOP-13197: Add non-decayed call metrics for DecayRpcScheduler.

HADOOP-13203: S3a: Consider reducing the number of connection aborts by setting correct length in s3 request.

HADOOP-13213: Small Documentation bug with AuthenticatedURL in HADOOP-auth.

HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials.

HADOOP-13241: Document s3a better.

HADOOP-13244: o.a.h.ipc.Server#Server should honor handlerCount when queueSizePerHandler is specified in constructor.

HADOOP-13249: RetryInvocationHandler need wrap InterruptedException in IOException when call Thread.sleep.

HADOOP-13251: Authenticate with Kerberos credentials when renewing KMS delegation token.

HADOOP-13255: KMSClientProvider should check and renew tgt when doing delegation token operations.

HADOOP-13263: Reload cached groups in background after expiry.

HADOOP-13270: BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks.

HADOOP-13280: FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps.

HADOOP-13283: Support reset operation for new global storage statistics and per FS storage stats.

HADOOP-13285: DecayRpcScheduler MXBean should only report decayed CallVolumeSummary.

HADOOP-13287: TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +.

HADOOP-13288: Guard null stats key in FileSystemStorageStatistics.

HADOOP-13290: Appropriate use of generics in FairCallQueue.

HADOOP-13291: Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented.

HADOOP-13305: Define common statistics names across schemes.

HADOOP-13315: FileContext#umask is not initialized properly.

HADOOP-13316: Enforce Kerberos authentication for required ops in DelegationTokenAuthenticator.

HADOOP-13320: Fix arguments check in documentation for WordCount v2.0.

HADOOP-13350: Additional fix to LICENSE and NOTICE.

HADOOP-13351: TestDFSClientSocketSize buffer size tests are flaky.

HADOOP-13352: Make X-FRAME-OPTIONS configurable in HttpServer2.

HADOOP-13368: DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation.

HADOOP-13382: Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, HADOOP-openstack.

HADOOP-13422: ZKDelegationTokenSecretManager JaasConfig does not work well with other ZK users in process.

HADOOP-13434: Add quoting to Shell class.

HADOOP-13467: Shell#getSignalKillCommand should use the bash built-in on Linux.

HDFS-1477: Support reconfiguring dfs.heartbeat.interval and dfs.namenode.heartbeat.recheck-interval without NN restart.

HDFS-2043: TestHFlush failing intermittently.

HDFS-2580: NameNode#main(.) can make use of GenericOptionsParser.

HDFS-27: HDFS CLI with --config set to default config complains log file not found error.

HDFS-3677: dfs.namenode.edits.dir.required is missing from HDFS- default.xml.

HDFS-4015: Safemode should count and report orphaned blocks.

HDFS-4185: Add a metric for number of active leases.

HDFS-4366: Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks.

HDFS-4396: Add START_MSG/SHUTDOWN_MSG for ZKFC.

HDFS-4937: ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom.

HDFS-4946: Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable.

HDFS-5640: Add snapshot methods to FileContext.

HDFS-6054: MiniQJMHACluster should not use static port to avoid binding failure in unit test.

HDFS-6101: TestReplaceDatanodeOnFailure fails occasionally.

HDFS-6407: Add sorting and pagination in the datanode tab of the NN Web UI.

HDFS-6481: DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs.

HDFS-6533: TestBPOfferService#testBasicFunctionalitytest fails intermittently.

HDFS-6581: Support for writing to single replica in RAM. Applied HDFS-6581 merge.15.patch to Champlain. This is a consolidated patch containing all feature changes and was applied to trunk.

HDFS-6666: Abort NameNode and DataNode startup if security is enabled but block access token is not enabled.

HDFS-6832: Fix the usage of 'hdfs namenode' command.

HDFS-6860: BlockStateChange logs are too noisy.

HDFS-6945: BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed.

HDFS-7163: WebHdfsFileSystem should retry reads according to the configured retry policy.

HDFS-7166: SbNN Web UI shows #Under replicated blocks and #pending deletion blocks.

HDFS-7314: When the DFSClient lease cannot be renewed, abort open-for- write files rather than the entire DFSClient.

HDFS-7390: Provide JMX metrics per storage type.

HDFS-7452: skip StandbyException log for getCorruptFiles()

HDFS-7483: Display information per tier on the Namenode UI.

HDFS-7597: DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping.

HDFS-7608: HDFS dfsclient newConnectedPeer has no write timeout.

HDFS-7609: Avoid retry cache collision when Standby NameNode loading edits.

HDFS-7645: Fix CHANGES.txt.

HDFS-7645: Rolling upgrade is restoring blocks from trash multiple times.

HDFS-7701: Support reporting per storage type quota and usage with hadoop/hdfs shell.

HDFS-7725: Incorrect "nodes in service" metrics caused all writes to fail.

HDFS-7833: DataNode reconfiguration does not recalculate valid volumes required, based on configured failed volumes tolerated.

HDFS-7858: Improve HA Namenode Failover detection on the client.

HDFS-7890: Improve information on Top users for metrics in RollingWindowsManager and lower log level.

HDFS-7923: The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages.

HDFS-7928: Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy.

HDFS-7933: fsck should also report decommissioning replicas.

HDFS-7978: Add LOG.isDebugEnabled() guard for some LOG.debug.

HDFS-7990: IBR delete ack should not be delayed.

HDFS-8008: Support client-side back off when the datanodes are congested.

HDFS-8009: Signal congestion on the DataNode.

HDFS-8046: Allow better control of getContentSummary.

HDFS-8055: NullPointerException when topology script is missing.

HDFS-8071: Redundant checkFileProgress() in PART II of getAdditionalBlock().

HDFS-8099: Change "DFSInputStream has been closed already" message to debug log level.

HDFS-8101: DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS classes at runtime.

HDFS-8113: Add check for null BlockCollection pointers in BlockInfoContiguous structures.

HDFS-8131: Implement a space balanced block placement policy.

HDFS-8144: Split TestLazyPersistFiles into multiple tests.

HDFS-8152: Refactoring of lazy persist storage cases.

HDFS-8155: Support OAuth2 in WebHDFS.

HDFS-8180: AbstractFileSystem Implementation for WebHdfs.

HDFS-8205: CommandFormat#parse() should not parse option as value of option.

HDFS-8209: Support different number of datanode directories in MiniDFSCluster.

HDFS-8211: DataNode UUID is always null in the JMX counter.

HDFS-8219: setStoragePolicy with folder behavior is different after cluster restart.

HDFS-8229: LAZY_PERSIST file gets deleted after NameNode restart.

HDFS-8232: Missing datanode counters when using Metrics2 sink interface.

HDFS-8276: LazyPersistFileScrubber should be disabled if scrubber interval configured zero.

HDFS-8278: When computing max-size-to-move in Balancer, count only the storage with remaining >= default block size.

HDFS-8309: Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows.

HDFS-8310: Fix TestCLI.testAll "help: help for find" on Windows.

HDFS-8311: DataStreamer.transfer() should timeout the socket InputStream.

HDFS-8337: Accessing httpfs via webhdfs doesn't work from a jar with Kerberos.

HDFS-8345: Storage policy APIs must be exposed via the FileSystem interface.

HDFS-8384: Allow NN to startup if there are files having a lease but are not under construction.

HDFS-8431: HDFS crypto class not found in Windows.

HDFS-8432: Introduce a minimum compatible layout version to allow downgrade in more rolling upgrade use cases.

HDFS-8435: Support CreateFlag in WebHDFS.

HDFS-8539: Hdfs doesn't have class 'debug' in windows.

HDFS-8542: WebHDFS getHomeDirectory behavior does not match specification.

HDFS-8546: Use try with resources in DataStorage and Storage.

HDFS-8548: Minicluster throws NPE on shutdown.

HDFS-8549: Abort the balancer if an upgrade is in progress.

HDFS-8554: TestDatanodeLayoutUpgrade fails on Windows.

HDFS-8578: On upgrade, Datanode should process all storage/data dirs in parallel.

HDFS-8581: ContentSummary on / skips further counts on yielding lock.

HDFS-8582: Support getting a list of reconfigurable config properties and do not generate spurious reconfig warnings.

HDFS-8593: Calculation of effective layout version mishandles comparison to current layout version in storage.

HDFS-8615: Correct HTTP method in WebHDFS document.

HDFS-8633: Fix setting of dfs.datanode.readahead.bytes in HDFS-default.xml to match DFSConfigKeys.

HDFS-8642: Make TestFileTruncate more reliable.

HDFS-8647: Abstract BlockManager's rack policy into BlockPlacementPolicy.

HDFS-8656: Preserve compatibility of ClientProtocol#rollingUpgrade after finalization.

HDFS-8659: Block scanner INFO message is spamming logs.

HDFS-8676: Delayed rolling upgrade finalization can cause heartbeat expiration.

HDFS-8696: Make the lower and higher watermark in the DN Netty server configurable.

HDFS-8709: Clarify automatic sync in FSEditLog#logEdit.

HDFS-8722: Optimize datanode writes for small writes and flushes.

HDFS-8729: Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately which occasionally failed.

HDFS-8767: RawLocalFileSystem.listStatus() returns null for UNIX pipefile.

HDFS-8772: Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails.

HDFS-8778: TestBlockReportRateLimiting#testLeaseExpiration can deadlock.

HDFS-8785: TestDistributedFileSystem is failing in trunk.

HDFS-8797: WebHdfsFileSystem creates too many connections for pread.

HDFS-8806: Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared.

HDFS-8809: HDFS fsck reports under construction blocks as CORRUPT.

HDFS-8815: DFS getStoragePolicy implementation using single RPC call.

HDFS-8816: Improve visualization for the Datanode tab in the NN UI.

HDFS-8818: Changes the global moveExecutor to per datanode executors and changes MAX_SIZE_TO_MOVE to be configurable.

HDFS-8824: Do not use small blocks for balancing the cluster.

HDFS-8826: In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes.

HDFS-8828: Utilize Snapshot diff report to build diff copy list in distcp.

HDFS-8829: Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning.

HDFS-8831: Trash Support for deletion in HDFS encryption zone.

HDFS-8844: TestHDFSCLI does not cleanup the test directory.

HDFS-8845: DiskChecker should not traverse the entire tree.

HDFS-8846: Add a unit test for INotify functionality across a layout version upgrade.

HDFS-8850: VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks.

HDFS-8852: HDFS architecture documentation of version 2.x is outdated about append write support.

HDFS-8855: Webhdfs client leaks active NameNode connections.

HDFS-8856: Make LeaseManager#countPath O(1).

HDFS-8863: The remaining space check in BlockPlacementPolicyDefault is flawed.

HDFS-8867: Enable optimized block reports.

HDFS-8879: Quota by storage type usage incorrectly initialized upon namenode restart.

HDFS-8880: NameNode metrics logging.

HDFS-8883: NameNode Metrics : Add FSNameSystem lock Queue Length.

HDFS-8885: ByteRangeInputStream used in webhdfs does not override available().

HDFS-8887: Expose storage type and storage ID in BlockLocation.

HDFS-8891: HDFS concat should keep srcs order.

HDFS-8896: DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager.

HDFS-8911: NameNode Metric : Add Editlog counters as a JMX metric.

HDFS-8914: Document HA support in the HDFS HdfsDesign.md.

HDFS-8923: Add -source flag to balancer usage message.

HDFS-8930: Block report lease may leak if the 2nd full block report comes when NN is still in safemode.

HDFS-8932: NPE thrown in NameNode when try to get TotalSyncCount metric before editLogStream initialization.

HDFS-8939: Test(S)WebHdfsFileContextMainOperations failing.

HDFS-8950: NameNode refresh doesn't remove DataNodes that are no longer in the allowed list.

HDFS-8964: When validating the edit log, do not read at or beyond the file offset that is being written.

HDFS-8965: Harden edit log reading code against out of memory errors.

HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.

HDFS-8983: NameNode support for protected directories.

HDFS-8995: Flaw in registration bookkeeping can make DN die on reconnect.

HDFS-9008: Balancer#Parameters class could use a builder pattern.

HDFS-9009: Send metrics logs to NullAppender by default.

HDFS-9019: Adding informative message to sticky bit permission denied exception.

HDFS-9033: dfsadmin -metasave prints "NaN" for cache used%.

HDFS-9042: Update document for the Storage policy name.

HDFS-9043: Doc update for commands in HDFS Federation.

HDFS-9047: Retire libwebhdfs.

HDFS-9048: DistCp documentation is out-of-dated.

HDFS-9063: Correctly handle snapshot path for getContentSummary.

HDFS-9072: Fix random failures in TestJMXGet.

HDFS-9082: Change the log level in WebHdfsFileSystem.initialize() from INFO to DEBUG.

HDFS-9083: Replication violates block placement policy.

HDFS-9094: Add command line option to ask NameNode reload configuration.

HDFS-9106: Transfer failure during pipeline recovery causes permanent write failures.

HDFS-9107: Prevent NNs unrecoverable death spiral after full GC.

HDFS-9112: Improve error message for Haadmin when multiple name service IDs are configured.

HDFS-9128: TestWebHdfsFileContextMainOperations and TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows.

HDFS-9130: Use GenericTestUtils#setLogLevel to the logging level.

HDFS-9142: Separating Configuration object for namenode(s) in MiniDFSCluster.

HDFS-9175: Change scope of 'AccessTokenProvider.getAccessToken()' and 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to public.

HDFS-9178: Slow datanode I/O can cause a wrong node to be marked bad.

HDFS-9184: Logging HDFS operation's caller context into audit logs.

HDFS-9198: Coalesce IBR processing in the NN.

HDFS-9205: Do not schedule corrupt blocks for replication.

HDFS-9210: Fix some misuse of %n in VolumeScanner#printStats.

HDFS-9214: Support reconfiguring dfs.datanode.balance.max.concurrent.moves without DN restart.

HDFS-9220: Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum.

HDFS-9221: HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array.

HDFS-9226: Unit Tests failing due to mockito version issue.

HDFS-9239: DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness.

HDFS-9259: Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario.

HDFS-9273: ACLs on root directory may be lost after NN restart.

HDFS-9289: Make DataStreamer#block thread safe and verify genStamp in commitBlock.

HDFS-9290: DFSClient#callAppend() is not backward compatible for slightly older NameNodes.

HDFS-9294: DFSClient deadlock when close file and failed to renew lease.

HDFS-9305: Delayed heartbeat processing causes storm of subsequent heartbeats.

HDFS-9311: Support optional offload of NameNode HA service health checks to a separate RPC server.

HDFS-9313: Possible NullPointerException in BlockManager if no excess replica can be chosen.

HDFS-9314: Improve BlockPlacementPolicyDefault's picking of excess replicas.

HDFS-9317: Document fsck -blockId and -storagepolicy options.

HDFS-9343: Empty caller context considered invalid.

HDFS-9347: Invariant assumption in TestQuorumJournalManager.shutdown() is wrong.

HDFS-9349: Support reconfiguring fs.protected.directories without NN restart.

HDFS-9354: Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows.

HDFS-9358: TestNodeCount#testNodeCount timed out.

HDFS-9362: TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.

HDFS-9364: Unnecessary DNS resolution attempts when creating NameNodeProxies.

HDFS-9383: TestByteArrayManager#testByteArrayManager fails.

HDFS-9384: TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server.

HDFS-9395: Make HDFS audit logging consistent.

HDFS-9397: Fix typo for readChecksum() LOG.warn in BlockSender.java.

HDFS-9402: Switch DataNode.LOG to use slf4j.

HDFS-9406: FSImage may get corrupted after deleting snapshot.

HDFS-9412: getBlocks occupies FSLock and takes too long to complete.

HDFS-9413: getContentSummary() on standby should throw StandbyException.

HDFS-9414: Refactor reconfiguration of ClientDatanodeProtocol for reusability.

HDFS-9415: Document dfs.cluster.administrators and dfs.permissions.superusergroup.

HDFS-9426: Rollingupgrade finalization is not backward compatible.

HDFS-9428: Fix intermittent failure of TestDNFencing.testQueueingWithAppend.

HDFS-9431: DistributedFileSystem#concat fails if the target path is relative.

HDFS-9434: Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messages.

HDFS-9445: Datanode may deadlock while handling a bad volume.

HDFS-9466: TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky.

HDFS-9470: Encryption zone on root not loaded from fsimage after NN restart.

HDFS-9476: TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail.

HDFS-9478: Reason for failing ipc.FairCallQueue construction should be thrown.

HDFS-9493: Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk.

HDFS-9505: HDFS Architecture documentation needs to be refreshed.

HDFS-9516: Truncate file fails with data dirs on multiple disks.

HDFS-9517: Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes.

HDFS-9521: TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk.

HDFS-9530: ReservedSpace is not cleared for abandoned Blocks.

HDFS-9533: seen_txid in the shared edits directory is modified during bootstrapping.

HDFS-9534: Add CLI command to clear storage policy from a path.

HDFS-9549: TestCacheDirectives#testExceedsCapacity is flaky.

HDFS-9555: LazyPersistFileScrubber should still sleep if there are errors in the clear progress.

HDFS-9557: Reduce object allocation in PB conversion.

HDFS-9566: Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method

HDFS-9569: Log the name of the fsimage being loaded for better supportability.

HDFS-9572: Prevent DataNode log spam if a client connects on the data transfer port but sends no data.

HDFS-9574: Reduce client failures during datanode restart.

HDFS-9584: NPE in distcp when ssl configuration file does not exist in class path.

HDFS-9600: Do not check replication if the block is under construction.

HDFS-9605: Add links to failed volumes to explorer.html in HDFS Web UI.

HDFS-9608: Disk IO imbalance in HDFS with heterogeneous storages.

HDFS-9612: DistCp worker threads are not terminated after jobs are done.

HDFS-9619: SimulatedFSDataset sometimes can not find blockpool for the correct namenode.

HDFS-9623: Update example configuration of block state change log in log4j.properties.

HDFS-9624: DataNode start slowly due to the initial DU command operations.

HDFS-9625: Set replication for empty file failed when set storage policy.

HDFS-9626: TestBlockReplacement#testBlockReplacement fails occasionally.

HDFS-9629: Update the footer of Web UI to show year 2016.

HDFS-9630: DistCp minor refactoring and clean up.

HDFS-9634: WebHDFS client side exceptions don't provide enough details.

HDFS-9638: Improve DistCp Help and documentation.

HDFS-9644: Update encryption documentation to reflect nested EZs.

HDFS-9655: NN should start JVM pause monitor before loading fsimage.

HDFS-9661: Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage and createRbw.

HDFS-9669: TcpPeerServer should respect ipc.server.listen.queue.size.

HDFS-9670: DistCp throws NPE when source is root.

HDFS-9672: o.a.h.hdfs.TestLeaseRecovery2 fails intermittently.

HDFS-9688: Test the effect of nested encryption zones in HDFS downgrade.

HDFS-9706: Log more details in debug logs in BlockReceiver's constructor.

HDFS-9710: DN can be configured to send block receipt IBRs in batches.

HDFS-9711: Integrate CSRF prevention filter in WebHDFS.

HDFS-9713: DataXceiver#copyBlock should return if block is pinned.

HDFS-9715: Check storage ID uniqueness on datanode startup

HDFS-9721: Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference.

HDFS-9724: Degraded performance in WebHDFS listing as it does not reuse ObjectMapper.

HDFS-9726: Refactor IBR code to a new class.

HDFS-9730: Storage ID update does not happen when there is a layout change.

HDFS-9739: DatanodeStorage.isValidStorageId() is broken.

HDFS-9740: Use a reasonable limit in DFSTestUtil.waitForMetric().

HDFS-9743: Fix TestLazyPersistFiles#testFallbackToDiskFull.

HDFS-9748: Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice.

HDFS-9752: Permanent write failures may happen to slow writers during datanode rolling upgrades.

HDFS-9760: WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler.

HDFS-9764: DistCp doesn't print value for several arguments including -numListstatusThreads.

HDFS-9765: TestBlockScanner#testVolumeIteratorWithCaching fails intermittently.

HDFS-9766: TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently.

HDFS-9768: Reuse ObjectMapper instance in HDFS to improve the performance.

HDFS-9772: TestBlockReplacement#testThrottler doesn't work as expected.

HDFS-9777: Fix typos in DFSAdmin command line and documentation.

HDFS-9779: TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value.

HDFS-9784: Example usage is not correct in Transparent Encryption document.

HDFS-9790: HDFS Balancer should exit with a proper message if upgrade is not finalized.

HDFS-9797: Log Standby exceptions thrown by RequestHedgingProxyProvider at DEBUG Level.

HDFS-9799: Reimplement getCurrentTrashDir to remove incompatibility.

HDFS-9801: ReconfigurableBase should update the cached configuration.

HDFS-9812: Streamer threads leak if failure happens when closing DFSOutputStream.

HDFS-9839: Reduce verbosity of processReport logging.

HDFS-9842: dfs.datanode.balance.bandwidthPerSec should accept friendly size units.

HDFS-9844: Correct path creation in getTrashRoot to handle root dir.

HDFS-9851: NameNode throws NPE when setPermission is called on a path that does not exist.

HDFS-9854: Log cipher suite negotiation more verbosely.

HDFS-9864: Correct reference for RENEWDELEGATIONTOKEN and CANCELDELEGATIONTOKEN in webhdfs doc.

HDFS-9865: TestBlockReplacement fails intermittently in trunk.

HDFS-9871: "Bytes Being Moved" -ve(-1 B) when cluster was already balanced.

HDFS-9874: Long living DataXceiver threads cause volume shutdown to block.

HDFS-9880: TestDatanodeRegistration fails occasionally.

HDFS-9881: DistributedFileSystem#getTrashRoot returns incorrect path for encryption zones.

HDFS-9882: Add heartbeatsTotal in Datanode metrics.

HDFS-9902: Support different values of dfs.datanode.du.reserved per storage type.

HDFS-9904: testCheckpointCancellationDuringUpload occasionally fails.

HDFS-9905: WebHdfsFileSystem#runWithRetry should display original stack trace on error.

HDFS-9906: Remove spammy log spew when a datanode is restarted.

HDFS-9917: IBR accumulate more objects when SNN was down for sometime.

HDFS-9926: MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils.

HDFS-9941: Do not log StandbyException on NN, other minor logging fixes.

HDFS-9949: Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared.

HDFS-9958: BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages.

HDFS-10178: Permanent write failures can happen if pipeline recoveries occur for the first packet.

HDFS-10182: Hedged read might overwrite user's buf.

HDFS-10186: DirectoryScanner: Improve logs by adding full path of both actual and expected block directories.

HDFS-10189: PacketResponder#toString should include the downstreams for PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE.

HDFS-10199: Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing.

HDFS-10207: Support enable Hadoop IPC backoff without namenode restart.

HDFS-10209: Support enable caller context in HDFS namenode audit log without restart namenode.

HDFS-10216: Distcp -diff throws exception when handling relative path.

HDFS-10217: Show 'blockScheduled' tooltip in datanodes table.

HDFS-10223: peerFromSocketAndKey performs SASL exchange before setting connection timeouts.

HDFS-10224: Implement asynchronous rename for DistributedFileSystem.

HDFS-10228: TestHDFSCLI fails.

HDFS-10235: Last contact for Live Nodes should be relative time.

HDFS-10239: Fsshell mv fails if port usage doesn't match in src and destination paths.

HDFS-10245: Fix the findbugs warnings.

HDFS-10253: Fix TestRefreshCallQueue failure.

HDFS-10261: TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports.

HDFS-10264: Logging improvements in FSImageFormatProtobuf.Saver.

HDFS-10270: TestJMXGet:testNameNode() fails.

HDFS-10271: Extra bytes are getting released from reservedSpace for append.

HDFS-10275: TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly.

HDFS-10277: PositionedReadable test testReadFullyZeroByteFile failing in HDFS.

HDFS-10281: TestPendingCorruptDnMessages fails intermittently.

HDFS-10291: TestShortCircuitLocalRead failing.

HDFS-10309: Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga).

HDFS-10312: Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.

HDFS-10313: Distcp need to enforce the order of snapshot names passed to -diff.

HDFS-10319: Balancer should not try to pair storages with different types.

HDFS-10324: Trash directory in an encryption zone should be pre-created with correct permissions.

HDFS-10329: Bad initialisation of StringBuffer in RequestHedgingProxyProvider.

HDFS-10335: Mover$Processor#chooseTarget() always chooses the first matching target storage group.

HDFS-10341: Add a metric to expose the timeout number of pending replication blocks.

HDFS-10344: DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash.

HDFS-10346: Implement asynchronous setPermission/setOwner for DistributedFileSystem.

HDFS-10347: Namenode report bad block method doesn't log the bad block or datanode.

HDFS-10360: DataNode may format directory and lose blocks if current/VERSION is missing.

HDFS-10367: TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception.

HDFS-10372: Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume.

HDFS-10377: CacheReplicationMonitor shutdown log message should use INFO level.

HDFS-10390: Implement asynchronous setAcl/getAclStatus for DistributedFileSystem.

HDFS-10396: Using -diff option with DistCp may get "Comparison method violates its general contract" exception.

HDFS-10397: Distcp should ignore -delete option if -diff option is provided instead of exiting.

HDFS-10410: RedundantEditLogInputStream.LOG is set to wrong class.

HDFS-10415: TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called.

HDFS-10417: Improve error message from checkBlockLocalPathAccess.

HDFS-10424: DatanodeLifelineProtocol not able to use under security cluster.

HDFS-10431: Refactor and speedup TestAsyncDFSRename.

HDFS-10437: ReconfigurationProtocol not covered by HDFSPolicyProvider.

HDFS-10438: When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.

HDFS-10440: Improve DataNode web UI.

HDFS-10448: CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1.

HDFS-10458: getFileEncryptionInfo should return quickly for non-encrypted cluster.

HDFS-10468: HDFS read ends up ignoring an interrupt.

HDFS-10469: Add number of active xceivers to datanode metrics.

HDFS-10471: DFSAdmin#SetQuotaCommand's help msg is not correct.

HDFS-10474: hftp copy fails when file name with Chinese+special char.

HDFS-10481: HTTPFS server should correctly impersonate as end user to open file.

HDFS-10485: Fix findbugs warning in FSEditLog.java.

HDFS-10488: Update WebHDFS documentation regarding CREATE and MKDIR default permissions.

HDFS-10493: Add links to datanode web UI in namenode datanodes page.

HDFS-10508: DFSInputStream should set thread's interrupt status after catching InterruptException from sleep.

HDFS-10525: Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap.

HDFS-10556: DistCpOptions should be validated automatically.

HDFS-10579: HDFS web interfaces lack configs for X-FRAME-OPTIONS protection.

HDFS-10589: Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params.

HDFS-10617: PendingReconstructionBlocks.size() should be synchronized.

HDFS-10623: Remove unused import of httpclient.HttpConnection from TestWebHdfsTokens.

HDFS-10642: TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently.

HDFS-10643: HDFS namenode should always use service user (hdfs) to generateEncryptedKey.

HDFS-10660: Expose storage policy APIs via HDFSAdmin interface.

HDFS-10688: BPServiceActor may run into a tight loop for sending block report when hitting IOException.

MAPREDUCE-5485: Allow repeating job commit by extending OutputCommitter API.

MAPREDUCE-6197: Cache MapOutputLocations in ShuffleHandler.

MAPREDUCE-6273: HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state.

MAPREDUCE-6302: Backport preempt reducers after a configurable timeout irrespective of headroom.

MAPREDUCE-6436: JobHistory cache issue.

MAPREDUCE-6460: TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails.

MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.

MAPREDUCE-6514: Job hangs as ask is not updated after ramping down of all reducers.

MAPREDUCE-6543: Migrate MR Client test cases part 2.

MAPREDUCE-6549: multibyte delimiters with LineRecordReader cause duplicate records.

MAPREDUCE-6566: Add retry support to mapreduce CLI tool.

MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.

MAPREDUCE-6579: TestNetworkedJob.testNetworkedJob failed due to YARN-3946.

MAPREDUCE-6593: TestJobHistoryEventHandler.testTimelineEventHandling fails because of NPE.

MAPREDUCE-6618: YarnClientProtocolProvider leaking the YarnClient thread.

MAPREDUCE-6621: Memory Leak in JobClient#submitJobInternal().

MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.

MAPREDUCE-6639: Process hangs in LocatedFileStatusFetcher if FileSystem.get throws.

MAPREDUCE-6672: TestTeraSort fails on Windows.

MAPREDUCE-6689: MapReduce job can infinitely increase number of reducer resource requests.

YARN-998: Keep NM resource updated through dynamic resource config for RM/NM restart.

YARN-1376: NM need to notify the log aggregation status to RM through heartbeat.

YARN-1402: Update related Web UI and CLI with exposing client API to check log aggregation status.

YARN-1462: AHS API and other AHS changes to handle tags for completed MR jobs.

YARN-1556: NPE getting application report with a null appId.

YARN-1880: Cleanup TestApplicationClientProtocolOnHA.

YARN-2019: EAR-1682. Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore.

YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.

YARN-2194: Fix bug causing CGroups functionality to fail on RHEL7.

YARN-2392: Add more diags about app retry limits on AM failures.

YARN-2495: Allow admin specify labels from each NM (Distributed configuration for node label).

YARN-2498: Respect labels in preemption policy of capacity scheduler for inter-queue preemption.

YARN-2513: Host framework UIs in YARN for use with the ATS Note: Patch is in progress.

YARN-2571: RM to support YARN registry.

YARN-2619: Added NodeManager support for disk IO isolation through cgroups.

YARN-2666: TestFairScheduler.testContinuousScheduling fails intermittently.

YARN-2696: Queue sorting in CapacityScheduler should consider node label.

YARN-2740: Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled.

YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.

YARN-2868: FairScheduler: Metric for latency to allocate first container for an application.

YARN-2901: Add errors and warning metrics page to RM, NM web UI.

YARN-3021: YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp.

YARN-3026: Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp.

YARN-3102: Decommissioned Nodes not listed in Web UI.

YARN-3110: Few issues in ApplicationHistory web UI.

YARN-3136: Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers.

YARN-3215: Respect labels in CapacityScheduler when computing headroom.

YARN-3216: Max-AM-Resource-Percentage should respect node labels.

YARN-3225: New parameter of CLI for decommissioning node gracefully in RMAdmin CLI.

YARN-3243: CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits.

YARN-3248: Display count of nodes blacklisted by apps in the web UI.

YARN-3266: RMContext#inactiveNodes should have NodeId as map key.

YARN-3269: Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path.

YARN-3293: Track and display capacity scheduler health metrics in web UI.

YARN-3294: Allow dumping of Capacity Scheduler debug logs via web UI for a fixed time period.

YARN-3305: Normalize AM resource request on app submission.

YARN-3318: Create Initial OrderingPolicy Framework and FifoOrderingPolicy.

YARN-3319: Implement a FairOrderingPolicy.

YARN-3326: Support RESTful API for getLabelsToNodes.

YARN-3343: Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate timeout.

YARN-3345: Add non-exclusive node label API.

YARN-3347: Improve YARN log command to get AMContainer logs as well as running containers logs.

YARN-3348: Add a 'yarn top' tool to help understand cluster usage.

YARN-3354: Add node label expression in ContainerTokenIdentifier to support RM recovery.

YARN-3356: Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label.

YARN-3357: Move TestFifoScheduler to FIFO package.

YARN-3360: Add JMX metrics to TimelineDataManager.

YARN-3361: CapacityScheduler side changes to support non-exclusive node labels.

YARN-3362: Add node label usage in RM CapacityScheduler web UI.

YARN-3365: Enhanced NodeManager to support using the 'tc' tool via container-executor for outbound network traffic control.

YARN-3366: Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers.

YARN-3383: AdminService should use "warn" instead of "info" to log exception when operation fails.

YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.

YARN-3394: Enrich WebApplication proxy documentation.

YARN-3397: YARN rmadmin should skip -failover.

YARN-3404: Display queue name on application page.

YARN-3406: Display count of running containers in the RM's Web UI.

YARN-3410: YARN admin should be able to remove individual application records from RMStateStore.

YARN-3413: Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime.

YARN-3425: NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed.

YARN-3433: Jersey tests failing with Port in Use -again. .

YARN-3435: AM container to be allocated Appattempt AM container shown as null.

YARN-3443: Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM.

YARN-3448: Added a rolling time-to-live LevelDB timeline store implementation.

YARN-3451: Display attempt start time and elapsed time on the web UI.

YARN-3459: Fix failure of TestLog4jWarningErrorMetricsAppender.

YARN-3463: Integrate OrderingPolicy Framework with CapacityScheduler.

YARN-3480: Remove attempts that are beyond max-attempt limit from state store.

YARN-3494: Expose AM resource limit and usage in CS QueueMetrics.

YARN-3503: Expose disk utilization percentage and bad local and log dir counts in NM metrics.

YARN-3505: Node's Log Aggregation Report with SUCCEED should not cached in RMApps.

YARN-3511: Add errors and warnings page to ATS.

YARN-3517: RM web UI for dumping scheduler logs should be for admins only.

YARN-3521: Support return structured NodeLabel objects in REST API.

YARN-3530: ATS throws exception on trying to filter results without otherinfo.

YARN-3541: Add version info on timeline service / generic history web UI and REST API.

YARN-3543: ApplicationReport should be able to tell whether the Application is AM managed or not.

YARN-3552: RM Web UI shows -1 running containers for completed apps.

YARN-3565: NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String.

YARN-3579: CommonNodeLabelsManager should support NodeLabel instead of string label name when getting node-to-label/label-to-label mappings.

YARN-3580: [JDK8] TestClientRMService.testGetLabelsToNodes fails.

YARN-3581: Deprecate -directlyAccessNodeLabelStore in RMAdminCLI.

YARN-3583: Support of NodeLabel object instead of plain String in YarnClient side.

YARN-3593: Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page.

YARN-3600: AM container link is broken (on a killed application, at least).

YARN-3604: removeApplication in ZKRMStateStore should also disable watch.

YARN-3632: Ordering policy should be allowed to reorder an application when demand changes.

YARN-3635: Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework.

YARN-3654: ContainerLogsPage web UI should not have meta-refresh.

YARN-3695: ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.

YARN-3700: Made generic history service load a number of latest applications according to the parameter or the configuration.

YARN-3707: RM Web UI queue filter doesn't work.

YARN-3717: Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API.

YARN-3727: For better error recovery, check if the directory exists before using it for localization.

YARN-3740: Fixed the typo in the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS.

YARN-3751: Fixed AppInfo to check if used resources are null.

YARN-3766: Fixed the apps table column error of generic history web UI.

YARN-3769: Consider user limit when calculating total pending resource for preemption policy in Capacity Scheduler.

YARN-3785: Support for Resource as an argument during submitApp call in MockRM test class.

YARN-3846: RM Web UI queue filter is not working for sub queue.

YARN-3849: Too much of preemption activity causing continuous killing of containers across queues.

YARN-3873: PendingApplications in LeafQueue should also use OrderingPolicy.

YARN-3885: ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level.

YARN-3894: RM startup should fail for wrong CS xml NodeLabel capacity configuration.

YARN-3896: RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously.

YARN-3932: SchedulerApplicationAttempt#getResourceUsageReport and UserInfo should based on total-used-resources.

YARN-3941: Proportional Preemption policy should try to avoid sending duplicate PREEMPT_CONTAINER event to scheduler.

YARN-3946: Update exact reason as to why a submitted app is in ACCEPTED state to app's diagnostic message.

YARN-3948: Display Application Priority in RM Web UI.

YARN-3961: Expose pending, running and reserved containers of a queue in REST API and YARN top.

YARN-3966: Fix excessive loggings in CapacityScheduler.

YARN-3971: Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery.

YARN-3978: Configurably turn off the saving of container info in Generic AHS.

YARN-3983: Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic.

YARN-3986: getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface.

YARN-3987: AM container complete msg ack to NM once RM receive it.

YARN-4000: RM crashes with NPE if leaf queue becomes parent queue during restart.

YARN-4023: Publish Application Priority to TimelineServer.

YARN-4026: Refactored ContainerAllocator to accept a list of priories rather than a single priority.

YARN-4082: Container shouldn't be killed when node's label updated.

YARN-4087: Followup fixes after YARN-2019 regarding RM behavior when state-store error occurs.

YARN-4092: Fixed UI redirection to print useful messages when both RMs are in standby mode.

YARN-4101: RM should print alert messages if ZooKeeper and Resourcemanager gets connection issue.

YARN-4108: CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request.

YARN-4140: RM container allocation delayed in case of app submitted to Nodelabel partition.

YARN-4149: YARN logs -am should provide an option to fetch all the log files.

YARN-4155: TestLogAggregationService.testLogAggregationServiceWithInterval failing.

YARN-4162: CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API.

YARN-4215: RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request.

YARN-4225: Add preemption status to yarn queue -status for capacity scheduler.

YARN-4233: YARN Timeline Service plugin: ATS v1.5.

YARN-4285: Display resource usage as percentage of queue and cluster in the RM UI.

YARN-4287: Capacity Scheduler: Rack Locality improvement.

YARN-4288: Fixed RMProxy to retry on IOException from local host.

YARN-4290: Add -showDetails option to YARN Nodes CLI to print all nodes reports information.

YARN-4304: AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics.

YARN-4309: Add container launch related debug information to container logs when a container fails.

YARN-4313: Race condition in MiniMRYarnCluster when getting history server address.

YARN-4315: NaN in Queue percentage for cluster apps page.

YARN-4345: YARN rmadmin -updateNodeResource doesn't work.

YARN-4347: Resource manager fails with Null pointer exception.

YARN-4349: YARN_APPLICATION call to ATS does not have YARN_APPLICATION_CALLER_CONTEXT.

YARN-4365: FileSystemNodeLabelStore should check for root dir existence on startup.

YARN-4384: updateNodeResource CLI should not accept negative values for resource.

YARN-4390: Do surgical preemption based on reserved container in CapacityScheduler.

YARN-4405: Support node label store in non-appendable file system.

YARN-4414: Nodemanager connection errors are retried at multiple levels.

YARN-4416: Deadlock due to synchronised get Methods in AbstractCSQueue.

YARN-4417: Make RM and Timeline-server REST APIs more consistent.

YARN-4418: AM Resource Limit per partition can be updated to ResourceUsage as well.

YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.

YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.

YARN-4439: Clarify NMContainerStatus#toString method.

YARN-4454: NM to nodelabel mapping going wrong after RM restart.

YARN-4479: Change CS LeafQueue pendingOrderingPolicy to honor recovered apps.

YARN-4497: RM might fail to restart when recovering apps whose attempts are missing.

YARN-4502: Fix two AM containers get allocated when AM restart.

YARN-4522: Queue acl can be checked at app submission.

YARN-4535: Fix checkstyle error in CapacityScheduler.java.

YARN-4546: ResourceManager crash due to scheduling opportunity overflow.

YARN-4557: Fix improper Queues sorting in PartitionedQueueComparator when accessible-node-labels=*.

YARN-4565: Sometimes when sizeBasedWeight FairOrderingPolicy is enabled, under stress appears that cluster is virtually in deadlock.

YARN-4573: Fix test failure in TestRMAppTransitions#testAppRunningKill and testAppKilledKilled.

YARN-4584: RM startup failure when AM attempts greater than max-attempts.

YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.

YARN-4607: Pagination support for AppAttempt page TotalOutstandingResource Requests table.

YARN-4610: Reservations continue looking for one app causes other apps to starve.

YARN-4617: LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy.

YARN-4623: TestSystemMetricsPublisher#testPublishAppAttemptMetricsForUnmanagedAM fails with NPE. 7.

YARN-4625: Make ApplicationSubmissionContext and ApplicationSubmissionContextInfo more consistent.

YARN-4633: TestRMRestart.testRMRestartAfterPreemption fails intermittently.

YARN-4634: Scheduler UI/Metrics need to consider cases like non-queue label mappings.

YARN-4667: RM Admin CLI for refreshNodesResources throws NPE when nothing is configured.

YARN-4696: TimelineClient to add flush operation for deterministic writes (including testing) and Changes to EntityGroupFSTimelineStore for testability.

YARN-4699: Scheduler UI and REST o/p is not in sync when -replaceLabelsOnNode is used to change label of a node.

YARN-4709: NMWebServices produces incorrect JSON for containers.

YARN-4723: NodesListManager$UnknownNodeId ClassCastException.

YARN-4737: Add CSRF filter support in YARN.

YARN-4746: YARN web services should convert parse failures of appId, appAttemptId and containerId to 400.

YARN-4769: Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI.

YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.

YARN-4811: Generate histograms in ContainerMetrics for actual container resource usage.

YARN-4814: ATS 1.5 timelineclient impl call flush after every event write.

YARN-4815: ATS 1.5 timelineclinet impl try to create attempt directory for every event call.

YARN-4822: Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates.

YARN-4832: NM side resource value should get updated if change applied in RM side.

YARN-4844: Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource.

YARN-4846: Fix random failures for TestCapacitySchedulerPreemption #testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers.

YARN-4865: Track Reserved resources in ResourceUsage and QueueCapacities.

YARN-4890: Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate.

YARN-4893: Fix intermittent failures in TestRMAdminService.

YARN-4916: TestNMProxy.tesNMProxyRPCRetry fails.

YARN-4934: Reserved Resource for QueueMetrics needs to be handled correctly in few cases.

YARN-4955: Add retry for SocketTimeoutException in TimelineClient.

YARN-4984: LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak.

YARN-5023: TestAMRestart#testShouldNotCountFailureToMaxAttemptRetry random failure.

YARN-5048: DelegationTokenRenewer#skipTokenRenewal may throw NPE.

YARN-5076: YARN web interfaces lack XFS (Cross-Frame Script) protection.

YARN-5112: Excessive log warnings for directory permission issue on NM recovery.

YARN-5131: Distributed shell AM fails when extra container arrives during finishing.

YARN-5190: Registering/unregistering container metrics triggered by ContainerEvent and ContainersMonitorEvent are conflict which cause uncaught exception in ContainerMonitorImpl.

YARN-5214: Fixed locking in DirectoryCollection to avoid hanging NMs when various code-paths hit slow disks.

YARN-5270: Solve miscellaneous issues caused by YARN-4844.

YARN-5294: Pass remote ip address down to YarnAuthorizationProvider.

YARN-5296: NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl.

YARN-5297: Avoid printing a stack trace when recovering an app after the RM restarts.

YARN-5337: Fix Dshell OOM issues.