Issues Fixed in CDH 5.4.x

The following upstream issues are fixed in CDH 5.4.10:

• FLUME-2712 - Optional channel errors slows down the Source to Main channel event rate
• FLUME-2886 - Optional Channels can cause OOMs
• HADOOP-10406 - TestIPC.testIpcWithReaderQueuing may fail
• HADOOP-10668 - TestZKFailoverControllerStress#testExpireBackAndForth occasionally fails
• HADOOP-11218 - Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory
• HADOOP-12200 - TestCryptoStreamsWithOpensslAesCtrCryptoCodec should be skipped in non-native profile
• HADOOP-12240 - Fix tests requiring native library to be skipped in non-native profile
• HADOOP-12280 - Skip unit tests based on Maven profile rather than NativeCodeLoader.isNativeCodeLoaded
• HADOOP-12417 - TestWebDelegationToken failing with port in use
• HADOOP-12418 - TestRPC.testRPCInterruptedSimple fails intermittently
• HADOOP-12464 - Interrupted client may try to fail-over and retry
• HADOOP-12468 - Partial group resolution failure should not result in user lockout.
• HADOOP-12474 - MiniKMS should use random ports for Jetty server by default
• HADOOP-12559 - KMS connection failures should trigger TGT renewal
• HADOOP-12604 - Exception may be swallowed in KMSClientProvider.
• HADOOP-12605 - Fix intermittent failure of TestIPC.testIpcWithReaderQueuing
• HADOOP-12668 - Support excluding weak Ciphers in HttpServer2 through ssl-server.conf
• HADOOP-12682 - Fix TestKMS#testKMSRestart* failure
• HADOOP-12699 - TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
• HADOOP-12715 - TestValueQueue#testgetAtMostPolicyALL fails intermittently
• HADOOP-12736 - TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out
• HADOOP-12788 - OpensslAesCtrCryptoCodec should log which random number generator is used
• HDFS-6533 - TestBPOfferService#testBasicFunctionalitytest fails intermittently
• HDFS-7553 - fix the TestDFSUpgradeWithHA due to BindException
• HDFS-8647 - Abstract BlockManager's rack policy into BlockPlacementPolicy
• HDFS-9083 - Replication violates block placement policy
• HDFS-9092 - Nfs silently drops overlapping write requests and causes data copying to fail
• HDFS-9289 - Make DataStreamer#block thread safe and verify genStamp in commitBlock
• HDFS-9313 - Possible NullPointerException in BlockManager if no excess replica can be chosen
• HDFS-9347 - Invariant assumption in TestQuorumJournalManager.shutdown() is wrong
• HDFS-9358 - TestNodeCount#testNodeCount timed out
• HDFS-9406 - FSImage may get corrupted after deleting snapshot
• HDFS-9445 - Datanode may deadlock while handling a bad volume
• HDFS-9688 - Test the effect of nested encryption zones in HDFS downgrade
• HDFS-9721 - Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
• MAPREDUCE-6302 - Incorrect headroom can lead to a deadlock between map and reduce allocations
• MAPREDUCE-6460 - TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails
• YARN-2902 - Killing a container that is localizing can orphan resources in the DOWNLOADING state
• YARN-4155 - TestLogAggregationService.testLogAggregationServiceWithInterval failing
• YARN-4204 - ConcurrentModificationException in FairSchedulerQueueInfo
• YARN-4347 - Resource manager fails with Null pointer exception
• YARN-4380 - TestResourceLocalizationService.testDownloadingResourcesOnContainerKill fails intermittently
• YARN-4393 - Fix intermittent test failure for TestResourceLocalizationService#testFailedDirsResourceRelease
• YARN-4573 - Fix test failure in TestRMAppTransitions#testAppRunningKill and testAppKilledKilled
• YARN-4613 - Fix test failure in TestClientRMService#testGetClusterNodes
• HBASE-14205 - RegionCoprocessorHost System.nanoTime() performance bottleneck
• HBASE-14621 - ReplicationLogCleaner stuck on RS crash
• HBASE-14923 - VerifyReplication should not mask the exception during result comparison
• HBASE-14926 - Hung ThriftServer; no timeout on read from client; if client crashes, worker thread gets stuck reading
• HBASE-15019 - Replication stuck when HDFS is restarted
• HBASE-15031 - Fix merge of MVCC and SequenceID performance regression in branch-1.0
• HBASE-15032 - hbase shell scan filter string assumes UTF-8 encoding
• HBASE-15035 - bulkloading hfiles with tags that require splits do not preserve tags
• HBASE-15052 - Use EnvironmentEdgeManager in ReplicationSource
• HBASE-15104 - Occasional failures due to NotServingRegionException in IT tests
• HBASE-15157 - Add *PerformanceTest for Append, CheckAnd*
• HBASE-15213 - Fix increment performance regression caused by HBASE-8763 on branch-1.0
• HIVE-7575 - GetTables thrift call is very slow
• HIVE-10213 - MapReduce jobs using dynamic-partitioning fail on commit
• HIVE-10514 - Fix MiniCliDriver tests failure
• HIVE-11826 - 'hadoop.proxyuser.hive.groups' configuration does not prevent unauthorized user to access metastore
• HIVE-11828 - beeline -f fails on scripts with tabs between column type and comment
• HIVE-11977 - Hive should handle an external avro table with zero length files present
• HIVE-12008 - Hive queries failing when using count(*) on column in view
• HIVE-12388 - GetTables cannot get external tables when TABLE type argument is given
• HIVE-12505 - Insert overwrite in same encrypted zone silently fails to remove some existing files
• HIVE-12566 - Incorrect result returns when using COALESCE in WHERE condition with LEFT JOIN
• HIVE-12713 - Miscellaneous improvements in driver compile and execute logging
• HIVE-12784 - Group by SemanticException: Invalid column reference
• HIVE-12790 - Metastore connection leaks in HiveServer2
• HIVE-12795 - Vectorized execution causes ClassCastException
• HIVE-12946 - alter table should also add default scheme and authority for the location similar to create table
• HIVE-13039 - BETWEEN predicate is not functioning correctly with predicate pushdown on Parquet table
• HIVE-13065 - Hive throws NPE when writing map type data to a HBase backed table
• HUE-3106 - [filebrowser] Add support for full paths in zip file uploads
• HUE-3110 - [oozie] Fix bundle submission when coordinator points to multiple bundles
• HUE-3180 - [useradmin] Override duplicate username validation message
• IMPALA-1702 - Check for duplicate table IDs at the end of analysis (issue not entirely fixed, but now fails gracefully)
• IMPALA-2264 - Implicit casts to integers from decimals with higher precision sometimes allowed
• IMPALA-2473 - Excessive memory usage by scan nodes
• IMPALA-2621 - Fix flaky UNIX_TIMESTAMP() test
• IMPALA-2643 - Nested inline view produces incorrect result when referencing the same column implicitly
• IMPALA-2765 - AnalysisException: operands of type BOOLEAN and TIMESTAMP are not comparable when OUTER JOIN with CASE statement
• IMPALA-2798 - After adding a column to avro table, Impala returns weird result if codegen is enabled.
• IMPALA-2861 - Fix flaky scanner test added via IMPALA-2473 backport
• IMPALA-2914 - Hit DCHECK Check failed: HasDateOrTime()
• IMPALA-3034 - MemTracker leak on PHJ failure to spill
• IMPALA-3085 - DataSinks' MemTrackers need to unregister themselves from parent
• IMPALA-3093 - ReopenClient() could NULL out 'client_key' causing a crash
• IMPALA-3095 - Allow additional Kerberos users to be authorized to access internal APIs
• KITE-1114 - fix test
• KITE-1114 - Fix missing license header
• KITE-1114 - Kite CLI json-import HDFS temp file path not multiuser safe
• OOZIE-2413 - Kerberos credentials can expire if the KDC is slow to respond
• OOZIE-2428 - TestSLAService, TestSLAEventGeneration flaky tests
• OOZIE-2432 - TestPurgeXCommand fails
• OOZIE-2435 - TestCoordChangeXCommand is flaky
• SENTRY-835 - Drop table leaves a connection open when using MetastoreListener
• SENTRY-885 - DB name should be case insensitive in HDFS sync plugin
• SENTRY-944 - Setting HDFS rules on Sentry managed hdfs paths should not affect original hdfs rules
• SENTRY-953 - External Partitions which are referenced by more than one table can cause some unexpected behavior with Sentry HDFS sync
• SENTRY-957 - Exceptions in MetastoreCacheInitializer should probably not prevent HMS from starting up
• SENTRY-988 - It's better to let SentryAuthorization setter path always fall through and update HDFS
• SENTRY-991 - backportRoles of Sentry Permission needs to be case insensitive
• SENTRY-994 - SentryAuthorizationInfoX should override isSentryManaged
• SENTRY-1002 - PathsUpdate.parsePath(path) will throw an NPE when parsing relative paths
• SENTRY-1003 - backportSupport "reload" by updating the classpath of Sentry function aux jar path during run time
• SENTRY-1008 - Path should be not be updated if the create/drop table/partition event fails
• SENTRY-1044 - Tables with non-HDFS locations breaks HMS startup
• SOLR-7281 - Add an overseer action to publish an entire node as 'down'
• SOLR-8367 - Fix the LeaderInitiatedRecovery 'all replicas participate' fail-safe
• SOLR-8371 - Try and prevent too many recovery requests from stacking up and clean up some faulty cancel recovery logic
• SOLR-8372 - backportCanceled recovery can lead to data loss
• SOLR-8575 - Addendum to Fix HDFSLogReader replay
• SOLR-8575 - Fix HDFSLogReader replay status numbers and a performance bug where we can reopen FSDataInputStream too often
• SOLR-8615 - Just like creating cores, we should use multiple threads when closing cores
• SOLR-8720 - ZkController#publishAndWaitForDownStates should use #publishNodeAsDown
• SOLR-8771 - Multithreaded core shutdown creates executor per core
• SQOOP-2847 - Sqoop --incremental + missing parent --target-dir reports success with no data
• SQOOP-2422 - Sqoop2: Test TestJSONIntermediateDataFormat is failing on JDK8
• ZOOKEEPER-442 - Need a way to remove watches that are no longer of interest"

The following upstream issues are fixed in CDH 5.4.9:

• FLUME-2841 - Upgrade commons-collections to 3.2.2
• HADOOP-7713 - dfs -count -q should label output column
• HADOOP-11171 - Enable using a proxy server to connect to S3a
• HADOOP-12568 - Update core-default.xml to describe posixGroups support
• HADOOP-12577 - Bumped up commons-collections version to 3.2.2 to address a security flaw
• HDFS-7785 - Improve diagnostics information for HttpPutFailedException
• HDFS-7798 - Checkpointing failure caused by shared KerberosAuthenticator
• HDFS-7871 - NameNodeEditLogRoller can keep printing 'Swallowing exception' message
• HDFS-7990 - IBR delete ack should not be delayed
• HDFS-8646 - Prune cached replicas from DatanodeDescriptor state on replica invalidation
• HDFS-9123 - Copying from the root to a subdirectory should be forbidden
• HDFS-9250 - Add Precondition check to LocatedBlock#addCachedLoc
• HDFS-9273 - ACLs on root directory may be lost after NN restart
• HDFS-9332 - Fix Precondition failures from NameNodeEditLogRoller while saving namespace
• HDFS-9364 - Unnecessary DNS resolution attempts when creating NameNodeProxies
• HDFS-9470 - Encryption zone on root not loaded from fsimage after NN restart
• MAPREDUCE-6191 - Improve clearing stale state of Java serialization
• MAPREDUCE-6549 - Multibyte delimiters with LineRecordReader cause duplicate records
• YARN-4235 - FairScheduler PrimaryGroup does not handle empty groups returned for a user
• HBASE-6617 - ReplicationSourceManager should be able to track multiple WAL paths
• HBASE-12865 - WALs may be deleted before they are replicated to peers
• HBASE-13134 - mutateRow and checkAndMutate APIs don't throw region level exceptions
• HBASE-13618 - ReplicationSource is too eager to remove sinks
• HBASE-13703 - ReplicateContext should not be a member of ReplicationSource
• HBASE-14003 - Work around JDK-8044053
• HBASE-14283 - Reverse scan doesn’t work with HFile inline index/bloom blocks
• HBASE-14374 - Backport parent 'HBASE-14317 Stuck FSHLog' issue to 1.1
• HBASE-14501 - NPE in replication with TDE
• HBASE-14533 - Connection Idle time 1 second is too short and the connection is closed too quickly by the ChoreService
• HBASE-14547 - Add more debug/trace to zk-procedure
• HBASE-14799 - Commons-collections object deserialization remote command execution vulnerability
• HBASE-14809 - Grant / revoke Namespace admin permission to group
• HIVE-10265 - Hive CLI crashes on != inequality
• HIVE-11149 - Sometimes HashMap in PerfLogger.java hangs
• HIVE-11616 - DelegationTokenSecretManager reuses the same objectstore, which has concurrency issues
• HIVE-12058 - Change hive script to record errors when calling hbase fails
• HIVE-12188 - DoAs does not work properly in non-Kerberos secured HS2
• HIVE-12189 - The list in pushdownPreds of ppd.ExprWalkerInfo should not be allowed to grow very large
• HIVE-12250 - ZooKeeper connection leaks in Hive's HBaseHandler
• HIVE-12365 - Added resource path is sent to cluster as an empty string when externally removed
• HIVE-12378 - Exception on HBaseSerDe.serialize binary field
• HIVE-12406 - HIVE-9500 introduced incompatible change to LazySimpleSerDe public interface
• HIVE-12418 - HiveHBaseTableInputFormat.getRecordReader() causes ZooKeeper connection leak
• HUE-2941 - [hadoop] Cache the active RM HA
• HUE-3035 - [beeswax] Optimize sample data query for partitioned tables
• IMPALA-1459 - Fix migration/assignment of On-clause predicates inside inline views
• IMPALA-1675 - Avoid overflow when adding large intervals to TIMESTAMPs
• IMPALA-1746 - QueryExecState does not check for query cancellation or errors
• IMPALA-1917 - Do not register aux equivalence predicates with NULL on either side
• IMPALA-1949 - Analysis exception when a binary operator contain an IN operator with
• IMPALA-2086/IMPALA-2090 - Avoid boost year/month interval logic
• IMPALA-2141 - UnionNode::GetNext() does not check for query errors
• IMPALA-2252 - Crash (likely race) tearing down BufferedBlockMgr on query failure
• IMPALA-2260 - Adding a large hour interval caused an interval overflow
• IMPALA-2265 - Sorter was not checking the returned Status of PrepareRead
• IMPALA-2273 - Make MAX_PAGE_HEADER_SIZE configurable
• IMPALA-2286 - Fix race between ~BufferedBlockMgr() and BufferedBlockMgr::Create()
• IMPALA-2344 - Work-around IMPALA-2344 Fail query with OOM in case block->Pin() fails
• IMPALA-2357 - Fix spilling sorts with var-len slots that are NULL or empty
• IMPALA-2446 - Fix wrong predicate assignment in outer joins
• IMPALA-2533 - Impala throws IllegalStateException when inserting data into a partition
• IMPALA-2559 - Fix check failed: sorter_runs_.back()->is_pinned_
• IMPALA-2664 - Avoid sending large partition stats objects over thrift
• IMPALA-2731 - Refactor MemPool usage in HBase scan node
• KITE-1089 - readAvroContainer morphline command should work even if the Avro writer schema of each input file is different
• PIG-3641 - Split "otherwise" producing incorrect output when combined with ColumnPruning
• SENTRY-565 - Improve performance of filtering Hive SHOW commands
• SENTRY-702 - Hive binding should support RELOAD command
• SENTRY-936 - getGroup and getUser should always return orginal hdfs values for paths in prefixes which are not Sentry managed
• SENTRY-960 - Blacklist reflect, java_method using hive.server2.builtin.udf.blacklist
• SOLR-6443 - backportDisable test that fails on Jenkins with SolrCore.getOpenCount()==2
• SOLR-7049 - LIST Collections API call should be processed directly by the CollectionsHandler instead of the OverseerCollectionProcessor
• SOLR-7552 - Support using ZkCredentialsProvider/ZkACLProvider in custom filter
• SOLR-7989 - After a new leader is elected, it should ensure it's state is ACTIVE if it has already registered with ZK
• SOLR-8075 - Leader Initiated Recovery should not stop a leader that participated in an election with all of it's replicas from becoming a valid leader
• SOLR-8223 - Avoid accidentally swallowing OutOfMemoryError
• SOLR-8288 - DistributedUpdateProcessor#doFinish should explicitly check and ensure it does not try to put itself into LIR
• SPARK-11484 - [WEBUI] Using proxyBase set by spark AM
• SPARK-11652 - [CORE] Remote code execution with InvokerTransformer

The following upstream issues are fixed in CDH 5.4.8:

• FLUME-2095 - JMS source with TIBCO
• HADOOP-11261 - Set custom endpoint for S3A
• HADOOP-12404 - Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class
• HADOOP-12413 - AccessControlList should avoid calling getGroupNames in isUserInList with empty groups
• HDFS-7916 - 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop
• HDFS-7978 - Add LOG.isDebugEnabled() guard for some LOG.debug()
• HDFS-8384 - Allow NN to startup if there are files having a lease but are not under construction
• HDFS-8735 - Inotify: All events classes should implement toString() API
• HDFS-8860 - Remove unused Replica copyOnWrite code
• HDFS-8964 - When validating the edit log, do not read at or beyond the file offset that is being written
• HDFS-8965 - Harden edit log reading code against out of memory errors
• MAPREDUCE-5918 - LineRecordReader can return the same decompressor to CodecPool multiple times
• MAPREDUCE-5948 - org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well
• MAPREDUCE-6273 - HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
• MAPREDUCE-6481 - LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes
• MAPREDUCE-6484 - YARN Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled
• YARN-2666 - TestFairScheduler.testContinuousScheduling fails intermittently
• YARN-3385 - Fixed a race-condition in ResourceManager's ZooKeeper based state-store to avoid crashing on duplicate deletes
• YARN-3469 - ZKRMStateStore: Avoid setting watches that are not required.
• YARN-3943 - Use separate threshold configurations for disk-full detection and disk-not-full detection
• HBASE-13217 - Procedure fails due to ZK issue
• HBASE-13331 - Exceptions from DFS client can cause CatalogJanitor to delete referenced files
• HBASE-13388 - Handling NullPointer in ZKProcedureMemberRpcs while getting ZNode data
• HBASE-13933 - DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
• HBASE-14196 - Thrift server idle connection timeout issue
• HBASE-14302 - TableSnapshotInputFormat should not create back references when restoring snapshot
• HBASE-14347 - Add a switch to DynamicClassLoader to disable it
• HBASE-14385 - Close the sockets that is missing in connection closure
• HBASE-14394 - Properly close the connection after reading records from table
• HBASE-14471 - Thrift - HTTP Error 413 full HEAD if using Kerberos authentication
• HBASE-14492 - Increase REST server header buffer size from 8k to 64k
• HIVE-5545 - HCatRecord getInteger method returns String when used on Partition columns of type INT
• HIVE-8529 - HiveSessionImpl#fetchResults should not try to fetch operation log when hive.server2.logging.operation.enabled is false
• HIVE-9867 - Migrate usage of deprecated Calcite methods
• HIVE-9984 - JoinReorder's getOutputSize is exponential
• HIVE-10021 - "Alter index rebuild" statements submitted through HiveServer2 fail when Sentry is enabled
• HIVE-10122 - Hive metastore filter-by-expression is broken for non-partition expressions
• HIVE-10421 - DROP TABLE with qualified table name ignores database name when checking partitions
• HIVE-10451 - PTF deserializer fails if values are not used in reducer
• HIVE-10658 - Insert with values clause may expose data that should be encrypted
• HIVE-10980 - Merge of dynamic partitions loads all data to default partition
• HIVE-11077 - Part of Exchange partition does not properly populate fields for post/pre execute hooks.
• HIVE-11440 - Create Parquet predicate push down (PPD) unit tests and q-tests
• HIVE-11504 - Predicate pushing down does not work for float type for Parquet
• HIVE-11590 - AvroDeserializer is very chatty
• HIVE-11618 - BackportCorrect the SARG api to reunify the PredicateLeaf.Type INTEGER and LONG
• HIVE-11657 - HIVE-2573 introduces some issues during metastore init (and CLI init)
• HIVE-11695 - If user has no permission to create LOCAL DIRECTORY, the Hql does not throw any exception and fails silently
• HIVE-11696 - Exception when table-level serde is Parquet while partition-level serde is JSON
• HIVE-11712 - Duplicate groupby keys cause ClassCastException
• HIVE-11737 - IndexOutOfBounds compiling query with duplicated groupby keys
• HIVE-11745 - Alter table Exchange partition with multiple partition_spec is not working
• HIVE-11816 - Upgrade groovy to 2.4.4
• HIVE-11824 - Insert to local directory causes staging directory to be copied
• HIVE-11843 - Add 'sort by c' to Parquet PPD q-tests to avoid different output issues with hadoop-1
• HIVE-11891 - Add basic performance logging to metastore calls
• HIVE-11926 - Backport:Stats annotation might not extract stats for varchar/decimal columns
• HIVE-11982 - Some test cases for union all fail with recent changes
• HIVE-11995 - Remove repetitively setting permissions in insert/load overwrite partition
• HUE-2881 - [oozie] A fork can point to a deleted node
• IMPALA-1136 - Support loading Avro tables without an explicit Avro schema
• IMPALA-1899 - Cleanup handling of Hive's field schema3e0fee5 IMPALA-2369, IMPALA-2435: Impala crashes when the sorter hits an OOM error
• IMPALA-2130 - Wrong verification of Parquet file version
• IMPALA-2161 - Skip \u0000 characters when dealing Avro schemas
• IMPALA-2165 - Avoid cardinality 0 in scan nodes of small tables and low selectivity
• IMPALA-2168 - Do not try to access streams of repartitioned spilled partition in right-joins
• IMPALA-2213 - Make Parquet scanner fail query if the file size metadata is stale
• IMPALA-2249 - Avoid allocating StringBuffer > 1GB in ScannerContext::Stream::GetBytesInternal()
• IMPALA-2256 - Handle joins with right side of high cardinality and zero materialized slots
• IMPALA-2270 - Avoid FnvHash64to32 with empty inputs
• IMPALA-2284 - Disallow long (1<<30) strings in group_concat()
• IMPALA-2292 - Change the type of timestamp_col to string in the table no_avro_schema.
• IMPALA-2314 - LargestSpilledPartition was not checking if partition is closed
• IMPALA-2348 - The catalog does not close the connection to HMS during table invalidation
• IMPALA-2364 - Wrong DCHECK in PHJ::ProcessProbeBatch
• IMPALA-2366 - Check fread return code correctly
• IMPALA-2440 - Fix old HJ full outer join with no rows
• IMPALA-2477 - Parquet metadata randomly 'appears stale'
• IMPALA-2514 - DCHECK on destroying an ExprContext
• KITE-1069 - Make zkClientSessionTimeout and zkClientConnectTimeout configurable in SolrLocator
• KITE-1074 - Partial updates aka Atomic updates with loadSolr aren't recognized with SolrCloud
• MAHOUT-1771 - Cluster dumper omits indices and 0 elements for dense vector or sparse containing 0s
• OOZIE-2376 - Default action configs not honored if no <configuration> section in workflow
• SENTRY-878 - collect_list missing from HIVE_UDF_WHITE_LIST
• SENTRY-884 - Give execute permission by default to paths managed by sentry
• SENTRY-893 - Synchronize calls in SentryClient and create sentry client once per request in SimpleDBProvider
• SOLR-5776 - Use less TLS/SSL in a test run
• SOLR-7109 - Indexing threads stuck during network partition can put leader into down state
• SOLR-7844 - Zookeeper session expiry during shard leader election can cause multiple leaders
• SOLR-7956 - There are interrupts on shutdown in places that can cause ChannelAlreadyClosed exceptions which prevents proper closing of transaction logs and can poison the IndexWriter and interfere with the HDFS client
• SOLR-8046 - HdfsCollectionsAPIDistributedZkTest checks that no transaction logs failed to be opened during the test but does not isolate this to the test and could fail due to other tests
• SOLR-8069 - Ensure that only the valid ZooKeeper registered leader can put a replica into Leader Initiated Recovery
• SOLR-8075 - Leader Initiated Recovery should not stop a leader that participated in an election with all of it's replicas from becoming a valid leader
• SOLR-8077 - Replication can still cause index corruption
• SOLR-8085 - Fix a variety of issues that can result in replicas getting out of sync
• SOLR-8094 - HdfsUpdateLog should not replay buffered documents as a replacement to dropping them
• SOLR-8095 - Add enable prop for HDFS Locality Metrics
• SOLR-8121 - It looks like ChaosMonkeySafeLeader test can fail with replica inconsistency because waitForThingsToLevelOut can pass while state is still changing.
• SPARK-6880 - Spark Shutdowns with NoSuchElementException when running parallel collect on cachedRDD
• SQOOP-2597 - Missing method AvroSchemaGenerator.generate()

As a result of the above fixes, the following issues, previously published as Known Issues in CDH 5, are also fixed.

The following upstream issues are fixed in CDH 5.4.7:

• CRUNCH-525 - Correct (more) accurate default scale factors for built-in MapFn implementations
• CRUNCH-527 - Use hash smearing for partitioning
• CRUNCH-528 - Improve Pair comparison
• CRUNCH-531 - Fix split graph rendering typo.
• CRUNCH-535 - call initCredentials on the job
• CRUNCH-536 - Refactor CrunchControlledJob.Hook interface and make it client-accessible
• CRUNCH-539 - Fix reading WritableComparables bimap
• CRUNCH-540 - Make AvroReflectDeepCopier serializable
• CRUNCH-543 - Have AvroPathPerKeyTarget handle child directories properly
• CRUNCH-544 - Improve performance/serializability of materialized toMap.
• CRUNCH-546 - Remove calls to CellUtil.cloneXXX
• CRUNCH-547 - Properly handle nullability for Avro union types
• CRUNCH-548 - Have the AvroReflectDeepCopier use the class of the source object when constructing new instances instead of the target class
• CRUNCH-551 - Make the use of Configuration objects consistent in CrunchInputSplit and CrunchRecordReader
• CRUNCH-553 - Fix record drop issue that can occur w/From.formattedFile TableSources
• FLUME-1934 - Spooling Directory Source dies on encountering zero-byte files.
• FLUME-2753 - Error when specifying empty replace string in Search and Replace Interceptor
• HADOOP-12317 - Applications fail on NM restart on some linux distro because NM container recovery declares AM container as LOST
• HDFS-8806 - Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared
• HDFS-8850 - VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks.
• MAPREDUCE-5817 - Mappers get rescheduled on node transition even after all reducers are completed.
• MAPREDUCE-6277 - Job can post multiple history files if attempt loses connection to the RM
• MAPREDUCE-6439 - AM may fail instead of retrying if RM shuts down during the allocate call.
• YARN-2921 - Fix MockRM/MockAM#waitForState sleep too long.
• YARN-3823 - Fix mismatch in default values for yarn.scheduler.maximum-allocation-vcores property
• YARN-3990 - AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected
• HBASE-13329 - ArrayIndexOutOfBoundsException in CellComparator#getMinimumMidpointArray.
• HBASE-13437 - ThriftServer leaks ZooKeeper connections
• HBASE-13471 - Fix a possible infinite loop in doMiniBatchMutation
• HBASE-13684 - Allow mlockagent to be used when not starting as root
• HBASE-14162 - Fixing maven target for regenerating thrift classes fails against 0.9.2
• HBASE-14354 - Minor improvements for usage of the mlock agent
• HIVE-7476 - CTAS does not work properly for s3
• HIVE-9327 - CBO (Calcite Return Path): Removing Row Resolvers from ParseContext
• HIVE-9512 - HIVE-9327 causing regression in stats annotation
• HIVE-9580 - Server returns incorrect result from JOIN ON VARCHAR columns
• HIVE-9613 - Left join query plan outputs wrong column when using subquery
• HIVE-10085 - Lateral view on top of a view throws RuntimeException
• HIVE-10140 - Window boundary is not compared correctly
• HIVE-10288 - Cannot call permanent UDFs
• HIVE-10319 - Hive CLI startup takes a long time with a large number of databases
• HIVE-10719 - Hive metastore failure when alter table rename is attempted.
• HIVE-10875 - Select query with view in subquery adds underlying table as direct input
• HIVE-10906 - Value based UDAF function without orderby expression throws NPE
• HIVE-10911 - Add support for date datatype in the value based windowing function
• HIVE-10972 - DummyTxnManager always locks the current database in shared mode, which is incorrect
• HIVE-10985 - Value based windowing on timestamp and double can't handle NULL value
• HIVE-10996 - Aggregation / Projection over Multi-Join Inner Query producing incorrect results
• HIVE-11139 - PROPOSEDQTest combine2_hadoop20.q fails when using -Phadoop-1 profile due to
• HIVE-11172 - Vectorization wrong results for aggregate query with where clause without group by
• HIVE-11203 - Beeline force option does not force execution when errors occurred in a script.
• HIVE-11250 - Change in spark.executor.instances (and others) does not take effect after RSC is launched for HS2
• HIVE-11255 - get_table_objects_by_name() in HiveMetaStore.java needs to retrieve table objects in multiple batches
• HIVE-11258 - The function drop_database_core() of HiveMetaStore.java may not drop all the tables
• HIVE-11271 - java.lang.IndexOutOfBoundsException when union all with if function
• HIVE-11288 - Avro SerDe InstanceCache returns incorrect schema
• HIVE-11333 - ColumnPruner prunes columns of UnionOperator that should be kept
• HIVE-11502 - Map side aggregation is extremely slow
• HIVE-11604 - HIVE return wrong results in some queries with PTF function
• HIVE-11620 - Fix several qtest output order
• HUE-2873 - [oozie] Handle TransactionManagementError on workflow dashboard
• HUE-2877 - [desktop] Add pyasn1 and ndg_httpsclient to support SSL Server Name Indication
• HUE-2880 - [hadoop] Fix uploading large files to a kerberized HTTPFS
• HUE-2882 - [oozie] Fix parsing error when workflow job uses Australian timezone
• HUE-2883 - [impala] Canceling a query shows an error message
• HUE-2885 - [oozie] Java options java-opts not generated correctly in XML
• HUE-2893 - [desktop] Backport CherryPy SSL file upload fix
• HUE-2903 - [oozie] Fix error with Workflow parameter on rerun
• IMPALA-1737 - Substitute an InsertStmt's partition key exprs with the root node's smap.
• IMPALA-1756 - Constant filter expressions are not checked for errors and state cleanup is not done before throwing exception.
• IMPALA-1898 - Explicit aliases + ordinals analysis bug
• IMPALA-1983 - Warn if table stats are potentially corrupt.
• IMPALA-1987 - Fix TupleIsNullPredicate to return false if no tuples are nullable.
• IMPALA-2088 - Fix planning of empty union operands with analytics.
• IMPALA-2089 - Retain eq predicates bound by grouping slots with complex grouping exprs.
• IMPALA-2178 - fix Expr::ComputeResultsLayout() logic.
• IMPALA-2199 - Row count not set for empty partition when spec is used with compute incremental stats
• IMPALA-2201 - Unconditionally update the partition stats and row count.
• IMPALA-2203 - Set an InsertStmt's result exprs from the source statement's result exprs.
• IMPALA-2216 - Set the output smap of an EmptySetNode produced from an empty inline view.
• IMPALA-2239 - update misc.test to match the new .test file format.
• IMPALA-2266 - Pass correct child node in 2nd phase merge aggregation.
• KITE-1053 - Fix int overflow bug in FS writer.
• SENTRY-810 - CTAS without location is not verified properly
• SOLR-7135 - Allow the server build.xml 'sync-hack' target to by skipped by specifying a system property.
• SOLR-7999 - SolrRequetParserTest#testStreamURL started failing.
• SPARK-8606 - Prevent exceptions in RDD.getPreferredLocations() from crashing DAGScheduler
• ZOOKEEPER-442 - need a way to remove watches that are no longer of interest

The following upstream issues are fixed in CDH 5.4.5:

• CRUNCH-508 - Improve performance of Scala Enumeration counters in Scrunch
• CRUNCH-511 - Scrunch product type support should use derived() instead of derivedImmutable()
• CRUNCH-514 - AvroDerivedDeepCopier should initialize delegate MapFns
• CRUNCH-516 - Scrunch needs some additional null checks
• CRUNCH-530 - Fix object reuse bug in GenericRecordToTuple
• CRUNCH-542 - Wider tolerance for flaky scrunch PCollectionTest
• FLUME-2215 - ResettableFileInputStream can't support ucs-4 character
• FLUME-2732 - Make maximum tolerated failures before shutting down and recreating client in AsyncHbaseSink configurable
• FLUME-2738 - Async HBase sink FD leak on client shutdown
• FLUME-2749 - Kerberos configuration error when using short names in multiple HDFS Sinks
• HADOOP-12017 - Hadoop archives command should use configurable replication factor when closing
• HADOOP-12103 - Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing
• HADOOP-8151 - Error handling in snappy decompressor throws invalid exceptions
• HDFS-7501 - TransactionsSinceLastCheckpoint can be negative on SBNs
• HDFS-7546 - Document, and set an accepting default for dfs.namenode.kerberos.principal.pattern
• HDFS-7890 - Improve information on Top users for metrics in RollingWindowsManager and lower log level
• HDFS-7894 - Rolling upgrade readiness is not updated in jmx until query command is issued.
• HDFS-8072 - Reserved RBW space is not released if client terminates while writing block
• HDFS-8337 - Accessing httpfs via webhdfs doesn't work from a jar with kerberos
• HDFS-8656 - Preserve compatibility of ClientProtocol#rollingUpgrade after finalization
• HDFS-8681 - BlockScanner is incorrectly disabled by default
• MAPREDUCE-5965 - Hadoop streaming throws error if list of input files is high.
• YARN-3143 - RM Apps REST API can return NPE or entries missing id and other fields
• YARN-3453 - Fair Scheduler: Parts of preemption logic uses DefaultResourceCalculator even in DRF mode causing thrashing
• YARN-3535 - Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED
• YARN-3793 - Several NPEs when deleting local files on NM recovery
• YARN-3842 - NMProxy should retry on NMNotYetReadyException
• HBASE-13342 - Fix incorrect interface annotations
• HBASE-13419 - Thrift gateway should propagate text from exception causes.
• HBASE-13491 - Fix bug in FuzzyRowFilter#getNextForFuzzyRule
• HBASE-13851 - RpcClientImpl.close() can hang with cancelled replica RPCs
• HBASE-13885 - ZK watches leaks during snapshots
• HBASE-13958 - RESTApiClusterManager calls kill() instead of suspend() and resume()
• HBASE-13995 - ServerName is not fully case insensitive
• HBASE-14027 - Clean up netty dependencies
• HBASE-14045 - Bumping thrift version to 0.9.2.
• HBASE-14076 - ResultSerialization and MutationSerialization can throw InvalidProtocolBufferException when serializing a cell larger than 64MB
• HIVE-10252 - Make PPD work for Parquet in row group level
• HIVE-10270 - Cannot use Decimal constants less than 0.1BD
• HIVE-10553 - Remove hardcoded Parquet references from SearchArgumentImpl SearchArgumentImpl
• HIVE-10706 - Make vectorized_timestamp_funcs test more stable
• HIVE-10801 - 'drop view' fails throwing java.lang.NullPointerException
• HIVE-10808 - Inner join on Null throwing Cast Exception
• HIVE-11150 - Remove wrong warning message related to chgrp
• HIVE-11174 - Hive does not treat floating point signed zeros as equal
• HIVE-11216 - UDF GenericUDFMapKeys throws NPE when a null map value is passed in
• HIVE-11401 - Predicate push down does not work with Parquet when partitions are in the expression expression
• HIVE-6099 - Multi insert does not work properly with distinct count
• HIVE-9500 - Support nested structs over 24 levels
• HIVE-9665 - Parallel move task optimization causes race condition
• HIVE-10427 - collect_list() and collect_set() should accept struct types as argument
• HIVE-10437 - NullPointerException on queries where map/reduce is not involved on tables with partitions
• HIVE-10895 - ObjectStore does not close Query objects in some calls, causing a potential leak in some metastore db resources
• HIVE-10976 - Redundant HiveMetaStore connect check in HS2 CLIService start
• HIVE-10977 - No need to instantiate MetaStoreDirectSql when HMS DirectSql is disabled
• HIVE-11095 - Fix SerDeUtils bug when Text is reused
• HIVE-11100 - Beeline should escape semi-colon in queries
• HIVE-11112 - ISO-8859-1 text output has fragments of previous longer rows appended
• HIVE-11157 - Hive.get(HiveConf) returns same Hive object to different user sessions
• HIVE-11194 - Exchange partition on external tables should fail with error message when target folder already exists
• HIVE-11433 - NPE for a multiple inner join query
• HIVE-9767 - Fixes in Hive UDF to be usable in Pig
• HIVE-10629 - Dropping table in an encrypted zone does not drop warehouse directory
• HIVE-10630 - Renaming tables across encryption zones renames table even though the operation throws error
• HIVE-10659 - Beeline command which contains semi-colon as a non-command terminator will fail
• HIVE-10788 - Change sort_array to support non-primitive types
• HIVE-10895 - ObjectStore does not close Query objects in some calls causing potential leak.
• HIVE-11109 - Replication factor is not properly set in SparkHashTableSinkOperator [Spark Branch]
• HIVE-10594 - Remote Spark client doesn't use Kerberos keytab to authenticate [Spark Branch]
• HUE-2618 - [hive] Recent query results show character encoding in view
• HUE-2767 - [impala] Issue showing sample data for a table
• HUE-2796 - sync_groups_on_login doesn't work with posixGroups
• HUE-2807 - [useradmin] Support deleting numeric groups
• HUE-2808 - [dbquery] Add row numbers to support default order by
• HUE-2813 - [hive] Report when Hue server is down when trying to execute a query
• HUE-2814 - Revert pyopenssl 0.13.1
• HUE-2835 - Fixed issue with DN's that have weird comma location
• HUE-2840 - [useradmin] Fix create home directories for Add/Sync LDAP group
• HUE-2849 - [useradmin] Fix exception in Add/Sync LDAP group for undefined group name
• IMPALA-1929 - Avoiding a DCHECK of NULL hash table in spilled right joins
• IMPALA-2136 - Bug in PrintTColumnValue caused wrong stats for TINYINT partition cols
• IMPALA-2133 - Properly unescape string value for HBase filters
• IMPALA-2018 - Where clause does not propagate to joins inside nested views
• IMPALA-2064 - Add effective_user() builtin
• IMPALA-2125 - Make UTC to local TimestampValue conversion faster.
• IMPALA-2048 - Impala DML/DDL operations corrupt table metadata leading to Hive query failures
• KITE-1014 - Fix support for Hive datasets on Kerberos enabled clusters.
• KITE-1015 - Add "replaceValues" morphline command that replaces all matching record field values with a given replacement string
• KITE-462 - Oozie jobs do not pass credentials
• KITE-976 - DatasetKeyInputFormat/DatasetKeyOutputFormat not setting job configuration before loading dataset
• KITE-1030 - readCSV WARN log msg on overly long lines where quoteChar is non-empty should print the whole record seen so far
• OOZIE-2268 - Update ActiveMQ version for security and other fixes
• OOZIE-2286 - Update Log4j and Log4j-extras to latest 1.2.x release
• PIG-4053 - PIG-4053: TestMRCompiler succeeded with sun jdk 1.6 while failed with sun jdk 1.7
• PIG-4338 - PIG-4338: Fix test failures with JDK8
• PIG-4326 - PIG-4326: AvroStorageSchemaConversionUtilities does not properly convert schema for maps of arrays of records
• SENTRY-695 - Sentry service should read the hadoop group mapping properties from core-site
• SENTRY-721 - HDFS Cascading permissions not applied to child file ACLs if a direct grant exists
• SENTRY-752 - Sentry service audit log file name format should be consistent
• SOLR-7457 - Make DirectoryFactory publishing MBeanInfo extensible
• SOLR-7458 - Expose HDFS Block Locality Metrics
• SPARK-6480 - histogram() bucket function is wrong in some simple edge cases
• SPARK-6954 - ExecutorAllocationManager can end up requesting a negative number of executors
• SPARK-7503 - Resources in .sparkStaging directory can't be cleaned up on error
• SPARK-7705 - Cleanup of .sparkStaging directory fails if application is killed
• SQOOP-2103 - Not able define Decimal(n,p) data type in map-column-hive option
• SQOOP-2149 - Update Kite dependency to 1.0.0
• SQOOP-2252 - Add default to Avro Schema
• SQOOP-2294 - Change to Avro schema name breaks some use cases
• SQOOP-2295 - Hive import with Parquet should append automatically
• SQOOP-2327 - Sqoop2: Change package name from Authorization to authorization
• SQOOP-2339 - Move sub-directory might fail in append mode
• SQOOP-2362 - Add oracle direct mode in list of supported databases
• SQOOP-2400 - hive.metastore.sasl.enabled should be set to true for Oozie integration
• SQOOP-2406 - Add support for secure mode when importing Parquet files into Hive
• SQOOP-2437 - Use hive configuration to connect to secure metastore

The following upstream issues are fixed in CDH 5.4.4:

• HIVE-10572 - Improve Hive service test to check empty string
• HIVE-9934 - Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to "none", allowing authentication without password
• HIVE-10006 - RSC has memory leak while execute multi queries.
• HUE-2814 - Revert pyopenssl 0.13.1

As a result of the above fixes, the following issues, previously published as Known Issues in CDH 5, are also fixed.

Problem: Because of a change in the implementation of the NameNode metadata upgrade mechanism, upgrading to CDH 5.4.1 from a version lower than 5.4.0 can take an inordinately long time. In a cluster with NameNode high availability (HA) configured and a large number of edit logs, the upgrade can fail, with errors indicating a timeout in the pre-upgrade step on JournalNodes.

What to do:

To avoid the problem: Do not upgrade to CDH 5.4.1; upgrade to CDH 5.4.2 instead.

If you experience the problem: If you have already started an upgrade and seen it fail, contact Cloudera Support. This problem involves no risk of data loss, and manual recovery is possible.

If you have already completed an upgrade to CDH 5.4.1, or are installing a new cluster: In this case you are not affected and can continue to run CDH 5.4.1.

The following upstream issues are fixed in CDH 5.4.3:

• HADOOP-12043 - Display warning if defaultFs is not set when running fs commands.
• HADOOP-11969 - ThreadLocal initialization in several classes is not thread safe
• HADOOP-11402 - Negative user-to-group cache entries are never cleared for never-again-accessed users
• HADOOP-11238 - Update the NameNode's Group Cache in the background when possible
• HDFS-8535 - Clarify that dfs usage in dfsadmin -report output includes all block replicas.
• HDFS-8486 - DN startup may cause severe data loss
• HDFS-7917 - Use file to replace data dirs in test to simulate a disk failure.
• HDFS-7833 - DataNode reconfiguration does not recalculate valid volumes required, based on configured failed volumes tolerated.
• HDFS-7604 - Track and display failed DataNode storage locations in NameNode.
• HDFS-8380 - Always call addStoredBlock on blocks which have been shifted from one storage to another
• HDFS-7980 - Incremental BlockReport will dramatically slow down the startup of a namenode
• HDFS-8305 - HDFS INotify: the destination field of RenameOp should always end with the file name
• YARN-3842 - NMProxy should retry on NMNotYetReadyException
• YARN-3467 - Expose allocatedMB, allocatedVCores, and runningContainers metrics on running Applications in RM Web UI
• YARN-3762 - FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
• YARN-3675 - FairScheduler: RM quits when node removal races with continousscheduling on the same node
• YARN-3491 - PublicLocalizer#addResource is too slow.
• MAPREDUCE-6387 - Serialize the recently added Task#encryptedSpillKey field at the end
• HBASE-13481 - Master should respect master (old) DNS/bind related configurations
• HBASE-13826 - Unable to create table when group acls are appropriately set.
• HBASE-13729 - Old hbase.regionserver.global.memstore.upperLimit and hbase.regioniserver.global.memstore.lowerLimit properties are ignored if present
• HBASE-13789 - ForeignException should not be sent to the client
• HBASE-13779 - Calling table.exists() before table.get() end up with an empty Result
• HBASE-13780 - Default to 700 for HDFS root dir permissions for secure deployments
• HBASE-13768 - ZooKeeper znodes are bootstrapped with insecure ACLs in a secure configuration
• HBASE-13767 - Allow ZKAclReset to set and not just clear ZK ACLs
• HBASE-13086 - Show ZK root node on Master WebUI
• HBASE-13413 - Create an integration test for Replication
• HBASE-13611 - update clover to work for current versions
• HIVE-10841 - [WHERE col is not null] does not work sometimes for queries with many JOIN statements
• HIVE-10956 - HS2 leaks HMS connections
• HIVE-10571 - HiveMetaStoreClient should close existing thrift connection before its reconnect
• HIVE-10835 - Concurrency issues in JDBC driver
• HIVE-10802 - Table join query with some constant field in select fails
• HIVE-10538 - Fix NPE in FileSinkOperator from hashcode mismatch
• HIVE-10771 - "separatorChar" has no effect in "CREATE TABLE AS SELECT" statement
• HIVE-10732 - Hive JDBC driver does not close operation for metadata queries
• HIVE-10151 - insert into A select from B is broken when both A and B are Acid tables and bucketed the same way
• HIVE-10483 - insert overwrite partition deadlocks on itself with DbTxnManager
• HIVE-10050 - Support overriding memory configuration for AM launched for TempletonControllerJob
• HIVE-10242 - ACID: insert overwrite prevents create table command
• HIVE-10481 - ACID table update finishes but values not really updated if column names are not all lower case
• HIVE-10150 - delete from acidTbl where a in(select a from nonAcidOrcTbl) fails
• HIVE-10721 - SparkSessionManagerImpl leaks SparkSessions [Spark Branch]
• HIVE-10671 - yarn-cluster mode offers a degraded performance from yarn-client [Spark Branch]
• HIVE-10453 - HS2 leaking open file descriptors when using UDFs
• HIVE-2573 - Create per-session function registry
• HIVE-9520 - Create NEXT_DAY UDF
• HIVE-9143 - select user(), current_user()
• HIVE-5472 - support a simple scalar which returns the current timestamp
• HIVE-10646 - ColumnValue does not handle NULL_TYPE
• HUE-2784 - [oozie] Coordinator editor generate wrong Monday cron expression
• HUE-2793 - [JB] Fix Mapper & Reducer counts in job page
• HUE-2776 - [jb] Fix "View All Tasks" pagination
• HUE-2778 - [jobbrowser] Fix "Text Filter" search box in JB "View All Tasks" page
• HUE-2767 - [impala] Issue showing sample data for a table
• HUE-2754 - [oozie] Sqoop action with variable adds an empty argument
• HUE-2743 - [search] Error HTML style leaks in the UI
• HUE-2587 - [jb] Kill jobs in accepted state
• HUE-2731 - [core] Validate that Hue is running in collect data script
• HUE-2687 - [core] Create script to gather hue process info for troubleshooting
• HUE-2656 - [tools] Add cron scripts for restart when mem usage is high
• HUE-2701 - [oozie] Java action relative jar path results in error on submit
• HUE-2703 - [sentry] Make more obvious why a user is not a Sentry admin
• HUE-2741 - [home] Hide the document move dialog
• HUE-2739 - [metastore] Autocomplete with databases/tables with built in names fails
• HUE-2732 - Hue isn't correctly doing add_column migrations with non-blank defaults
• IMPALA-1963: Impala Timestamp ISO-8601 Support.
• IMPALA-2043: skip metadata/testddl.py#test_create_alter_bulk_partition on S3
• IMPALA-1968: Part 1: Improve planner numNodes estimate for remote scans
• IMPALA-1730: reduce scanner thread spinning windows
• IMPALA-2002: Provide way to cache ext data source classes
• IMPALA-2008: Fix wrong warning when insert overwrite to empty table
• IMPALA-1381: Expand set of supported timezones.
• IMPALA-1952: Expand parsing of decimals to include scientific notation
• SENTRY-227 - Fix for "Unsupported entity type DUMMYPARTITION"
• SOLR-7503 - Recovery after ZK session expiration happens in a single thread for all cores in a node
• SPARK-6299 - ClassNotFoundException in standalone mode when running groupByKey with class defined in REPL.
• SPARK-5522 - Accelerate the History Server start

Problem: Because of a change in the implementation of the NameNode metadata upgrade mechanism, upgrading to CDH 5.4.1 from a version lower than 5.4.0 can take an inordinately long time. In a cluster with NameNode high availability (HA) configured and a large number of edit logs, the upgrade can fail, with errors indicating a timeout in the pre-upgrade step on JournalNodes.

What to do:

To avoid the problem: Do not upgrade to CDH 5.4.1; upgrade to CDH 5.4.2 instead.

If you experience the problem: If you have already started an upgrade and seen it fail, contact Cloudera Support. This problem involves no risk of data loss, and manual recovery is possible.

If you have already completed an upgrade to CDH 5.4.1, or are installing a new cluster: In this case you are not affected and can continue to run CDH 5.4.1.

The following upstream issues are fixed in CDH 5.4.1:

• HADOOP-11891 - OsSecureRandom should lazily fill its reservoir to avoid open too many file descriptors.
• HADOOP-11802 - DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm
• HADOOP-11724 - DistCp throws NPE when the target directory is root.
• HDFS-7645 - Rolling upgrade is restoring blocks from trash multiple times which could cause significant and unnecessary block churn.
• HDFS-7869 - Inconsistency in the return information while performing rolling upgrade
• HDFS-8127 - NameNode Failover during HA upgrade can cause DataNode to finalize upgrade
• HDFS-3443 - Fix NPE when NameNode transition to active during startup.
• HDFS-7312 - Update DistCp v1 to optionally not use tmp location
• HDFS-8292 - Move conditional in fmt_time from dfs-dust.js to status.html
• HDFS-6673 - Add delimited format support to PB OIV tool
• HDFS-8214 - Secondary NN Web UI shows wrong date for Last Checkpoint
• HDFS-7884 - Fix NullPointerException in BlockSender when the generation stamp provided by the client is larger than the one stored in the DataNode
• HDFS-4448 - Allow HA NN to start in secure mode with wildcard address configured
• HDFS-8070 - Fix issue that Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode
• HDFS-7915 - The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error
• HDFS-7931 - DistributedFileSystem should not look for keyProvider in cache if Encryption is disabled
• HDFS-7916 - 'reportBadBlocks' from DataNodes to standby Node BPServiceActor goes for infinite loop
• HDFS-8099 - Change "DFSInputStream has been closed already" message to debug log level
• HDFS-7996 - After swapping a volume, BlockReceiver reports ReplicaNotFoundException
• HDFS-7587 - Edit log corruption can happen if append fails with a quota violation
• HDFS-7881 - TestHftpFileSystem#testSeek fails
• HDFS-7929 - inotify is unable to fetch pre-upgrade edit log segments once upgrade starts
• YARN-3363 - Add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container.
• YARN-3485 - FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies
• YARN-3464 - Race condition in LocalizerRunner kills localizer before localizing all resources
• YARN-3516 - Killing ContainerLocalizer action doesn't take effect when private localizer receives FETCH_FAILURE status.
• YARN-3021 - YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp
• YARN-3241 - FairScheduler handles invalid queue names inconsistently.
• YARN-2868 - FairScheduler: Add a metric for measuring latency of allocating first container for an application
• YARN-3428 - Add debug logs to capture the resources being localized for a container.
• MAPREDUCE-6339 - Job history file is not flushed correctly because isTimerActive flag is not set true when flushTimerTask is scheduled.
• MAPREDUCE-5710 - Running distcp with -delete incurs avoidable penalties
• MAPREDUCE-6343 - JobConf.parseMaximumHeapSizeMB() fails to parse value greater than 2GB expressed in bytes
• MAPREDUCE-6238 - MR2 can't run local jobs with -libjars command options which is a regression from MR1
• MAPREDUCE-6076 - Zero map split input length combined with none zero map split input length may cause MR1 job hung sometimes.
• HBASE-13374 - Small scanners (with particular configurations) do not return all rows
• HBASE-13269 - Limit result array pre-allocation to avoid OOME with large scan caching values
• HBASE-13335 - Update ClientSmallScanner and ClientSmallReversedScanner to use serverHasMoreResults context
• HBASE-13534 - Change HBase master WebUI to explicitly mention if it is a backup master
• HBASE-13111 - truncate_preserve command is failing with undefined method error
• HBASE-13430 - HFiles that are in use by a table cloned from a snapshot may be deleted when that snapshot is deleted
• HBASE-13546 - NPE on RegionServer status page if all masters are down
• HBASE-13350 - Add a debug-warning if we fail HTD checks even if table.sanity.checks is disabled
• HBASE-13262 - ResultScanner doesn't return all rows in Scan
• HIVE-10452 - Avoid sending Beeline prompt+query to the standard output/error only when in script mode.
• HIVE-10541 - Beeline requires newline at the end of each query in a file
• HIVE-9625 - Delegation tokens for HMS are not renewed
• HIVE-10499 - Ensure Session/ZooKeeperClient instances are closed
• HIVE-10312 - SASL.QOP in JDBC URL is ignored for Delegation token Authentication
• HIVE-10324 - Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
• HIVE-10202 - Beeline outputs prompt+query on standard output when used in non-interactive mode
• HIVE-10087 - Beeline's --silent option should suppress query from being echoed when running with -f option
• HIVE-10098 - HS2 local task for map join fails in KMS encrypted cluster
• HIVE-10146 - Add option to not count session as idle if query is running
• HIVE-10108 - Index#getIndexTableName() should return db.index_table_name instead of qualified table name
• HIVE-10093 - Unnecessary HMSHandler initialization for default MemoryTokenStore on HS2
• HIVE-10085 - Lateral view on top of a view throws RuntimeException
• HIVE-10086 - Parquet file using column index access throws error in Hive
• HIVE-9839 - HiveServer2 leaks OperationHandle on async queries which fail at compile phase
• HIVE-9920 - DROP DATABASE IF EXISTS throws exception if database does not exist
• HIVE-10476 - Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism
• HIVE-10434 - Cancel connection when remote Spark driver process has failed
• HIVE-10473 - Spark client is recreated even spark configuration is not changed
• HIVE-10291 - Hive on Spark job configuration needs to be logged
• HIVE-10143 - HS2 fails to clean up Spark client state on timeout
• HIVE-10073 - Runtime exception when querying HBase with Spark
• HUE-2723 - [hive] Listing table information in non default DB fails
• HUE-2722 - [hive] Query returns wrong number of rows when HiveServer2 returns data not encoded properly
• HUE-2713 - [oozie] Deleting a Fork of Fork can break the workflow
• HUE-2717 - [oozie] Coordinator editor does not save non-default schedules
• HUE-2716 - [pig] Scripts fail on hcat auth with org.apache.hive.hcatalog.pig.HCatLoader()
• HUE-2707 - [hive] Allow sample of data on partitioned tables in strict mode
• HUE-2720 - [oozie] Intermittent 500s when trying to view oozie workflow history v1
• HUE-2712 - [oozie] Creating a fork can error
• HUE-2710 - [search] Heatmap select on yelp example errors
• HUE-2686 - [impala] Explain button is erroring
• HUE-2671 - [core] sync_groups_on_login doesn't work with NT Domain
• IMPALA-1519/IMPALA-1946 - Fix wrapping of exprs via a TupleIsNullPredicate with analytics.
• IMPALA-1900 - Assign predicates below analytic functions with a compatible partition by clause for partition pruning.
• IMPALA-1919 - When out_batch->AtCapacity(), avoid calling ProcessBatch in right joins.
• IMPALA-1960 - Illegal reference to non-materialized tuple when query has an empty select-project-join block.
• IMPALA-1969 - OpenSSL init must not be called concurrently.
• IMPALA-1973 - Fixing crash when uninitialized, empty row is added in HdfsTextScanner due to missing newline at the end of file.
• OOZIE-2218 - META-INF directories in the war file have 777 permissions
• OOZIE-2170 - Oozie should automatically set configs to make Spark jobs show up in the Spark History Server
• SENTRY-699 - Memory leak when running Sentry with HiveServer2
• SENTRY-703 - Calls to add_partition fail when passed a Partition object with a null location
• SENTRY-696 - Improve Metastoreplugin Cache Initialization time
• SENTRY-683 - HDFS service client should ensure the kerberos ticket is valid before new service connection
• SOLR-7478 - UpdateLog#close shuts down it's executor with interrupts before running close, possibly preventing a clean close.
• SOLR-7437 - Make HDFS transaction log replication factor configurable.
• SOLR-7338/SOLR-6583 - A reloaded core will never register itself as active after a ZK session expiration.
• SPARK-7281 - No option for AM native library path in yarn-client mode.
• SPARK-6087 - Provide actionable exception if Kryo buffer is not large enough
• SPARK-6868 - Container link broken on Spark UI Executors page when YARN is set to HTTPS_ONLY
• SPARK-6506 - python support in yarn cluster mode requires SPARK_HOME to be set
• SPARK-6650 - ExecutorAllocationManager never stops
• SPARK-6578 - Outbound channel in network library is not thread-safe, can lead to fetch failures
• SQOOP-2343 - AsyncSqlRecordWriter stuck if any exception is thrown out in its close method
• SQOOP-2286 - Ensure Sqoop generates valid avro column names
• SQOOP-2283 - Support usage of --exec and --password-alias
• SQOOP-2281 - Set overwrite on kite dataset
• SQOOP-2282 - Add validation check for --hive-import and --append
• SQOOP-2257 - Import Parquet data into a hive table with --hive-overwrite option does not work
• ZOOKEEPER-2146 - BinaryInputArchive readString should check length before allocating memory
• ZOOKEEPER-2149 - Log client address when socket connection established

As a result of the above fixes, the following issues, previously published as Known Issues in CDH 5, are also fixed.