Issues Fixed in CDH 5.11.x

The following upstream issues are fixed in CDH 5.11.2:

• FLUME-2752 - Fix AvroSource startup resource leaks
• FLUME-2905 - Fix NetcatSource file descriptor leak if startup fails
• HADOOP-12751 - While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple .
• HADOOP-14141 - Store KMS SSL keystore password in catalina.properties
• HADOOP-14242 - Make KMS Tomcat SSL property sslEnabledProtocols and clientAuth configurable
• HADOOP-14511 - WritableRpcEngine.Invocation#toString NPE on null parameters
• HDFS-6757 - Simplify lease manager with INodeID
• HDFS-8856 - Make LeaseManager#countPath O(1).
• HDFS-10220 - A large number of expired leases can make namenode unresponsive and cause failover
• HDFS-10506 - OIV's ReverseXML processor cannot reconstruct some snapshot details
• HDFS-11579 - Make HttpFS Tomcat SSL property sslEnabledProtocols and clientAuth configurable
• HDFS-11708 - Positional read will fail if replicas moved to different DNs after stream is opened
• HDFS-11741 - Long running balancer may fail due to expired DataEncryptionKey
• HDFS-11861 - ipc.Client.Connection#sendRpcRequest should log request name
• HDFS-11881 - NameNode consumes a lot of memory for snapshot diff report generation
• HDFS-11960 - Successfully closed files can stay under-replicated
• HDFS-12042 - Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumption
• HDFS-12139 - HTTPFS liststatus returns incorrect pathSuffix for path of file
• YARN-2780 - Log aggregated resource allocation in rm-appsummary.log
• YARN-6368 - Decommissioning an NM results in a -1 exit code
• YARN-6615 - AmIpFilter drops query parameters on redirect
• HBASE-15720 - Print row locks at the debug dump page
• HBASE-15837 - Memstore size accounting is wrong if postBatchMutate() throws exception
• HBASE-16033 - Add more details in logging of responseTooSlow/TooLarge
• HBASE-16630 - Fragmentation in long running Bucket Cache
• HBASE-16739 - Timed out exception message should include encoded region name
• HBASE-16977 - VerifyReplication should log a printable representation of the row keys
• HBASE-17131 - Avoid livelock caused by HRegion#processRowsWithLocks
• HBASE-17501 - guard against NPE while reading FileTrailer and HFileBlock
• HBASE-17587 - Do not Rethrow DoNotRetryIOException as UnknownScannerException
• HBASE-17673 - Monitored RPC Handler not shown in the WebUI
• HBASE-17688 - MultiRowRangeFilter not working correctly if given same start and stop RowKey
• HBASE-17710 - HBase in standalone mode creates directories with 777 permission
• HBASE-17731 - Fractional latency reporting in MultiThreadedAction
• HBASE-17798 - RpcServer.Listener.Reader can abort due to CancelledKeyException
• HBASE-17970 - Set yarn.app.mapreduce.am.staging-dir when starting MiniMRCluster
• HBASE-18096 - Limit HFileUtil visibility and add missing annotations
• HIVE-9567 - JSON SerDe not escaping special chars when writing char/varchar data
• HIVE-10209 - FetchTask with VC may fail because ExecMapper.done is true
• HIVE-11418 - Dropping a database in an encryption zone with CASCADE and trash enabled fails
• HIVE-11592 - ORC metadata section can sometimes exceed protobuf message size limit
• HIVE-11878 - ClassNotFoundException can possibly occur if multiple jars are registered one at a time in Hive
• HIVE-12274 - Increase width of columns used for general configuration in the metastore.
• HIVE-12551 - Fix several kryo exceptions in branch-1
• HIVE-12762 - Common join on parquet tables returns incorrect result when hive.optimize.index.filter set to true
• HIVE-13330 - ORC vectorized string dictionary reader does not differentiate null vs empty string dictionary
• HIVE-13947 - HoS prints wrong number for hash table size in map join scenario
• HIVE-14178 - Hive::needsToCopy should reuse FileUtils::equalsFileSystem
• HIVE-14564 - Column Pruning generates out of order columns in SelectOperator which cause ArrayIndexOutOfBoundsException.
• HIVE-15122 - Hive: Upcasting types should not obscure stats (min/max/ndv)
• HIVE-16004 - OutOfMemory in SparkReduceRecordHandler with vectorization mode
• HIVE-16060 - GenericUDTFJSONTuple's json cache could overgrow beyond its limit
• HIVE-16291 - Hive fails when unions a parquet table with itself
• HIVE-16413 - Create table as select does not check ownership of the location
• HIVE-16559 - Parquet schema evolution for partitioned tables may break if table and partition serdes differ
• HIVE-16593 - SparkClientFactory.stop may prevent JVM from exiting
• HIVE-16647 - Improve the validation output to make the output to stderr and stdout more consistent
• HIVE-16660 - Not able to add partition for views in hive when sentry is enabled
• HIVE-16665 - Race condition in Utilities.GetInputPathsCallable --> createDummyFileForEmptyPartition
• HIVE-16693 - beeline "source" command freezes if you have a comment in it.
• HIVE-16697 - Schema table validator should return a sorted list of missing tables
• HIVE-16869 - Hive returns wrong result when predicates on non-existing columns are pushed down to Parquet reader
• HIVE-16930 - HoS should verify the value of Kerberos principal and keytab file before adding them to spark-submit command parameters
• HIVE-16935 - Hive should strip comments from input before choosing which CommandProcessor to run.
• HIVE-17050 - Multiline queries that have comment in middle fail when executed via "beeline -e"
• HIVE-17052 - Remove logging of predicate filters
• HIVE-17149 - Hdfs directory is not cleared if partition creation failed on HMS
• HUE-5504 - [oozie] Only use JDBC URL from hive2 action when hardcoded
• HUE-6370 - [jobsub] Attempting to create and save any JobSub fails
• HUE-6398 - [jobsub] Fix loading edit design page
• HUE-6407 - [pig] Play button doesn't come back after killing the running pig job
• HUE-6446 - [oozie] User cant edit shared coordinator or bundle
• HUE-6604 - [oozie] Fix timestamp conversion to server timezone
• HUE-6694 - [aws] Gracefully handle bucket creation error for non-DNS compliant names
• HUE-6695 - [aws] Provide user-friendly message when accessing forbidden paths
• HUE-6696 - [aws] Display user friendly message when accessing non-DNS compliant bucket
• HUE-6702 - [about] About page is accessible without authentication
• HUE-6710 - [notebook] Application reachable directly by users without granted access
• HUE-6791 - [search] Protect against pivot facets conflicting with nested facets
• HUE-6813 - [metastore] Fix database comment last character truncation
• HUE-6814 - [search] Only return distinct usernames in security impersonate dropdown
• HUE-6819 - [oozie] Set generic widget for generic actions in oozie graph
• HUE-6856 - [search] Protect against reflected XSS in search query parameters
• HUE-6950 - [saml] Create home directory for new user login
• HUE-6995 - [oozie] Set minimum width of a workflow node
• IMPALA-4276 - Profile displays non-default query options set by planner
• IMPALA-4546 - Fix Moscow timezone conversion after 2014
• IMPALA-4631 - don't use floating point operations for time unit conversions
• IMPALA-4716 - Expr rewrite causes IllegalStateException
• IMPALA-4738 - STDDEV_SAMP should return NULL for single record input
• IMPALA-4962 - Fix SHOW COLUMN STATS for HS2
• IMPALA-5021 - Fix count(*) remaining rows overflow in Parquet.
• IMPALA-5056 - Ensure analysis uses 'fresh' catalog after metadata loading
• IMPALA-5154 - Handle 'unpartitioned' Kudu tables
• IMPALA-5172 - Buffer overrun for Snappy decompression
• IMPALA-5187 - Bump breakpad version to include the fix for Breakpad #681, re-enable the strict check that was disabled in IMPALA-3794.
• IMPALA-5189 - Pin version of setuptools-scm
• IMPALA-5197 - Erroneous corrupted Parquet file message
• IMPALA-5198 - Error messages are sometimes dropped before reaching client
• IMPALA-5217 - KuduTableSink checks null constraints incorrectly
• IMPALA-5223 - Add waiting for HBase Zookeeper nodes to retry loop
• IMPALA-5301 - Set Kudu minicluster memory limit
• IMPALA-5318 - Generate access events with fully qualified table names
• IMPALA-5355 - Fix the order of Sentry roles and privileges
• IMPALA-5363 - Reset probe_batch_ after reaching limit
• IMPALA-5419 - Check for cancellation when building hash tables
• IMPALA-5469 - Fix exception when processing catalog update
• IMPALA-5487 - Fix race in RuntimeProfile::toThrift()
• IMPALA-5524 - Fixes NPE during planning with DISABLE_UNSAFE_SPILLS=1
• IMPALA-5554 - sorter DCHECK on null column
• IMPALA-5580 - fix Java UDFs that return NULL strings
• IMPALA-5615 - Fix compute incremental stats for general partition exprs
• IMPALA-5623 - Fix lag() on STRING cols to release UDF mem
• IMPALA-5638 - Fix Kudu table set tblproperties inconsistencies
• IMPALA-5657 - Fix a couple of bugs with FunctionCallExpr and IGNORE NULLS
• IMPALA-5172 - fix incorrect cast in call to LZO decompress
• KITE-1155 - Deleting an already deleted empty path should not fail the job
• OOZIE-2816 - Strip out the first command word from Sqoop action if its "sqoop"
• OOZIE-2844 - Increase stability of Oozie actions when log4j.properties is missing or not readable
• OOZIE-2872 - Address backward compatibility issue introduced by OOZIE-2748
• OOZIE-2908 - Fix typo in oozie.actions.null.args.allowed property in oozie-default.xml
• OOZIE-2923 - Improve Spark options parsing
• OOZIE-2984 - Parse spark-defaults.conf values with spaces without needing the quotes
• PARQUET-389 - Support predicate push down on missing columns.
• PIG-3567 - LogicalPlanPrinter throws OOM for large scripts
• PIG-3655 - BinStorage and InterStorage approach to record markers is broken
• SENTRY-1644 - Partition ACLs disappear after renaming Hive table with partitions.
• SENTRY-1646 - Unable to truncate table <database>.<tablename>; from "default" databases
• SENTRY-1759 - UpdatableCache leaks connections
• SENTRY-1811 - Optimize data structures used in HDFS sync
• SENTRY-1827 - Minimize TPathsDump thrift message used in HDFS sync
• SOLR-6673 - MDC-based logging of collection, shard, etc.
• SOLR-8836 - /update should return BAD REQUEST when invalid JSON provided.
• SOLR-9153 - Update beanutils version to 1.9.2
• SOLR-9527 - Solr RESTORE API doesn't distribute the replicas uniformly
• SOLR-10076 - Hiding keystore and truststore passwords from /admin/info/* outputs.
• SOLR-10889 - Stale zookeeper information is used during failover check.
• SPARK-13278 - Launcher fails to start with JDK 9 EA
• SPARK-15067 - YARN executors are launched with fixed perm gen size
• SPARK-16845 - org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering `grows beyond 64 KB
• SPARK-19019 - PySpark does not work with Python 3.6.0
• SPARK-19688 - Spark on YARN Credentials File set to different application directory.
• SPARK-20393 - Strengthen Spark to prevent XSS vulnerabilities
• SPARK-20904 - Task failures during shutdown cause problems with preempted executors.
• SPARK-20922 - Unsafe deserialization in Spark LauncherConnection.
• ZOOKEEPER-1653 - zookeeper fails to start because of inconsistent epoch.
• ZOOKEEPER-2040 - Server to log underlying cause of SASL connection problems.

The following upstream issues are fixed in CDH 5.11.1:

• FLUME-3080 - Close failure in HDFS Sink might cause data loss
• FLUME-3085 - HDFS Sink can skip flushing some BucketWriters, might lead to data loss
• HADOOP-11400 - GraphiteSink does not reconnect to Graphite after 'broken pipe'
• HADOOP-11599 - Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when IPC_CLIENT_PING_KEY is not configured
• HADOOP-12672 - RPC timeout should not override IPC ping interval
• HADOOP-13503 - Improve SaslRpcClient failure logging
• HADOOP-13926 - S3Guard: S3AFileSystem::listLocatedStatus() to employ MetadataStore
• HADOOP-14019 - Fix some typos in the s3a docs
• HADOOP-14028 - S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures
• HADOOP-14051 - S3Guard: link docs from index, fix typos
• HADOOP-14059 - typo in s3a rename(self, subdir) error message
• HADOOP-14092 - Typo in hadoop-aws index.md
• HADOOP-14104 - Client should always ask namenode for kms provider path
• HADOOP-14144 - s3guard: CLI diff non-empty after import on new table
• HADOOP-14172 - S3Guard: import does not import empty directory
• HADOOP-14195 - CredentialProviderFactory$getProviders is not thread-safe
• HADOOP-14204 - S3A multipart commit failing, "UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".
• HADOOP-14215 - DynamoDB client should waitForActive on existing tables
• HADOOP-14236 - S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries
• HADOOP-14255 - S3A to delete unnecessary fake directory objects in mkdirs()
• HADOOP-14256 - [S3A DOC] Correct the format for "Seoul" example
• HADOOP-14268 - Fix markdown itemization in hadoop-aws documents
• HADOOP-14282 - S3Guard: DynamoDBMetadata::prune() should self interrupt correctly
• HADOOP-14417 - Update default SSL cipher list for KMS
• HDFS-10715 - NPE when applying AvailableSpaceBlockPlacementPolicy
• HDFS-10797 - Revert "Disk usage summary of snapshots causes renamed blocks to get counted twice
• HDFS-11499 - Decommissioning stuck because of failing recovery
• HDFS-11515 - -du throws ConcurrentModificationException
• HDFS-11689 - New exception thrown by DFSClient%isHDFSEncryptionEnabled broke hacky hive code
• HDFS-11816 - Update default SSL cipher list for HttpFS
• YARN-3251 - Fixed a deadlock in CapacityScheduler when computing absoluteMaxAvailableCapacity in LeafQueue
• YARN-6042 - Dump scheduler and queue state information into FairScheduler DEBUG log.
• YARN-6360 - Prevent FS state dump logger from cramming other log files
• YARN-6432 - FairScheduler: Reserve preempted resources for corresponding applications.
• YARN-6433 - Only accessible cgroup mount directories should be selected for a controller.
• YARN-6453 - fairscheduler-statedump.log gets generated regardless of service
• YARN-6500 - Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler.
• HBASE-15125 - BackportHBaseFsck's adoptHdfsOrphan function creates region with wrong end key boundary
• HBASE-15941 - HBCK repair should not unsplit healthy splitted region
• HBASE-15955 - Disable action in CatalogJanitor#setEnabled should wait for active cleanup scan to finish
• HBASE-16032 - Possible memory leak in StoreScanner
• HBASE-16238 - It's useless to catch SESSIONEXPIRED exception and retry in RecoverableZooKeeper
• HBASE-16350 - Undo server abort from HBASE-14968
• HBASE-16429 - FSHLog: deadlock if rollWriter called when ring buffer filled with appends
• HBASE-16663 - JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
• HBASE-16721 - Concurrency issue in WAL unflushed seqId tracking
• HBASE-17265 - Region left unassigned in master failover when region failed to open
• HBASE-17328 - Properly dispose of looped replication peers
• HBASE-17460 - enable_table_replication can not perform cyclic replication of a table
• HBASE-17717 - Explicitly use "sasl" ACL scheme for hbase superuser
• HBASE-17779 - disable_table_replication returns misleading message and does not turn off replication
• HBASE-17792 - Use a shared thread pool for AtomicityWriter, AtomicGetReader, AtomicScanReader's connections in TestAcidGuarantees
• HIVE-11141 - Improve RuleRegExp when the Expression node stack gets huge
• HIVE-11428 - Performance: Struct IN() clauses are extremely slow
• HIVE-11671 - Optimize RuleRegExp in DPP codepath
• HIVE-11842 - Improve RuleRegExp by caching some internal data structures
• HIVE-12179 - Add option to not add spark-assembly.jar to Hive classpath
• HIVE-12768 - Thread safety: binary sortable serde decimal deserialization
• HIVE-14210 - ExecDriver should call jobclient.close() to trigger cleanup
• HIVE-14380 - Queries on tables with remote HDFS paths fail in "encryption" checks.
• HIVE-15282 - Different modification times are used when an index is built and when its staleness is checked
• HIVE-15782 - query on parquet table returns incorrect result when hive.optimize.index.filter is set to true
• HIVE-15879 - Fix HiveMetaStoreChecker.checkPartitionDirs method
• HIVE-15997 - Resource leaks when query is cancelled
• HIVE-16024 - MSCK Repair Requires nonstrict hive.mapred.mode
• HIVE-16047 - Shouldn't try to get KeyProvider unless encryption is enabled
• HIVE-16156 - FileSinkOperator should delete existing output target when renaming
• HIVE-16175 - Possible race condition in InstanceCache
• HIVE-16205 - Improving type safety in Objectstore
• HIVE-16297 - Improving hive logging configuration variables
• HIVE-16394 - HoS does not support queue name change in middle of session
• HIVE-16459 - Forward channelInactive to RpcDispatcher
• HIVE-16646 - Alias in transform ... as clause shouldn't be case sensitive
• HUE-6109 - [core] Remove the restriction on Document2 invalid chars
• HUE-6144 - [oozie] Add generic XSL template to workflow graph parser
• HUE-6154 - [core] Ace paste event shouldn't reset the cursor position
• HUE-6155 - [core] Remove table extender header throttling for Firefox
• HUE-6158 - [autocomplete] The autocompleter eats characters to the right of the cursor on insert
• HUE-6193 - [converter] Retain last_executed time when creating doc2 object
• HUE-6197 - [impala] Fix XSS Vulnerability in the old editors' error messages
• HUE-6207 - [editor] Avoid to always show the horizontal scroll bar when there's no scrolling needed
• HUE-6208 - [core] The scroll left anchor should reset the horizontal scrollbar position
• HUE-6212 - [oozie] Prevent XSS injection in coordinator cron frequency field
• HUE-6223 - [autocomplete] Fix issue where tables from subsequent statements appear in the autocomplete results
• HUE-6228 - [core] Disable touchscreen detection on Nicescroll
• HUE-6251 - [editor] Log warnings and continue on failed bulk delete and copy actions
• IMPALA-3641 - Fix catalogd RPC responses to DROP IF EXISTS.
• IMPALA-4088 - Assign fix values to the minicluster server ports
• IMPALA-4293 - query profile should include error log
• IMPALA-4544 - ASAN should ignore SEGV and leaks
• IMPALA-4615 - Fix create_table.sql command order
• IMPALA-4733 - Change HBase ports to non-ephemeral
• IMPALA-4787 - Optimize APPX_MEDIAN() memory usage
• IMPALA-4822 - Implement dynamic log level changes
• IMPALA-4899 - Fix parquet table writer dictionary leak
• IMPALA-4902 - Copy parameters map in HdfsPartition.toThrift().
• IMPALA-4998 - Fix missing table lock acquisition.
• IMPALA-5028 - Lock table in /catalog_objects endpoint.
• IMPALA-5055 - Fix DCHECK in parquet-column-readers.cc ReadPageHeader()
• IMPALA-5088 - Fix heap buffer overflow
• IMPALA-5115 - Handle status from HdfsTableSink::WriteClusteredRowBatch
• IMPALA-5145 - Do not constant fold null in CastExprs
• IMPALA-5156 - Drop VLOG level passed into Kudu client
• IMPALA-5186 - Handle failed CreateAndOpenScanner() in MT scan.
• IMPALA-5193 - Initialize decompressor before finding first tuple
• IMPALA-5251 - Fix propagation of input exprs' types in 2-phase agg
• IMPALA-5252 - Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded
• IMPALA-5253 - Use appropriate transport for StatestoreSubscriber
• IMPALA-5322 - Fix a potential crash in Frontend & Catalog JNI startup
• OOZIE-2739 - Remove property expansion pattern from ShellMain's log4j properties content
• OOZIE-2818 - Can't overwrite oozie.action.max.output.data on a per-workflow basis
• OOZIE-2819 - Make Oozie REST API accept multibyte characters for script Actions
• SENTRY-1508 - Revert ""REVERT: MetastorePlugin.java does not handle properly initialization failure
• SENTRY-1605 - SENTRY-1508 need to be fixed because of Kerberos initialization issue
• SENTRY-1683 - MetastoreCacheInitializer has a race condition in handling results list
• SENTRY-1714 - MetastorePlugin.java should quetly return from renameAuthzObject() when both paths are null
• SOLR-9836 - Add ability to recover from leader when index corruption is detected on SolrCore creation.
• SOLR-9848 - Lower solr.cloud.wait-for-updates-with-stale-state-pause back down from 7 seconds.
• SOLR-10360 - Remove an extra space from Hadoop distcp cmd used by Solr backup/restore
• SOLR-10430 - Add ls command to ZkCLI for listing sub-dirs
• SPARK-14930 - [SPARK-13693] Fix race condition in CheckpointWriter.stop()
• SPARK-19178 - [SQL][Backport-to-1.6] convert string of large numbers to int should return null
• SPARK-19263 - DAGScheduler should avoid sending conflicting task set.
• SPARK-19537 - Move pendingPartitions to ShuffleMapStage.
• SPARK-20435 - [CORE] More thorough redaction of sensitive information
• SQOOP-3123 - Introduce escaping logic for column mapping parameters (same what Sqoop already uses for the DB column names), thus special column names (e.g. containing '#' character) and mappings realted to those columns can be in the same format (thus not confusing the end users), and also eliminates the related AVRO format clashing issues.
• SQOOP-3140 - Removing deprecated mapred.map.max.attempts, mapred.reduce.max.attempts entries and using the new constants directly from Hadoop instead
• SQOOP-3159 - Sqoop (export + --table) with Oracle table_name having '$' fails with error

The following upstream issues are fixed in CDH 5.11.0:

• AVRO-1973 - Upgrade netty to the latest 3.x release
• FLUME-2318 - Make SpoolingDirectorySource able to handle empty files
• FLUME-2514 - Speed up TestFileChannelRestart tests
• FLUME-2567 - Remove external Maven repositories
• FLUME-2593 - ResettableFileInputStream returns negate values from read() method
• FLUME-2619 - Spooldir source should log channel exceptions
• FLUME-2704 - Configurable poll delay for spooling directory source
• FLUME-2718 - HTTP Source to support generic Stream Handler
• FLUME-2725 - HDFS Sink does not use configured timezone for rounding
• FLUME-2761 - Move Hive sink out of preview mode
• FLUME-2798 - Malformed Syslog messages can lead to OutOfMemoryException
• FLUME-2801 - Performance improvement on TailDir source
• FLUME-2806 - flume-ng.ps1 Error running script to start an agent on Windows
• FLUME-2810 - Add static Schema URL to AvroEventSerializer configuration
• FLUME-2812 - Fix semaphore leak causing java.lang.Error: Maximum permit count exceeded in MemoryChannel
• FLUME-2835 - Hive Sink tests need to create table with transactional property set
• FLUME-2857 - Make Kafka Source/Channel/Sink restore default values when live updating config
• FLUME-2875 - Allow RollingFileSink to specify a file prefix and a file extension.
• FLUME-2881 - Windows Launch Script fails in plugins dir code
• FLUME-2889 - Fixes to DateTime computations
• FLUME-2890 - Typo in Twitter source warning
• FLUME-2911 - Add include pattern option in SpoolDir source
• FLUME-2936 - KafkaSource tests: Use loopback instead of host address
• FLUME-2937 - Integrate checkstyle for non-test classes
• FLUME-2939 - Update recursive SpoolDir source to use Java 7 APIs
• FLUME-2941 - Integrate checkstyle for test classes
• FLUME-2949 - checkstyle: Make suppressions compatible with Windows
• FLUME-2950 - ReliableSpoolingFileEventReader.rollCurrentFile is broken
• FLUME-2952 - SyslogAgent: Avoid NPE on stop()
• FLUME-2955 - Add file path to the header in TaildirSource
• FLUME-2959 - Fix issues with flume-checkstyle module
• FLUME-2966 - Fix NPE in JMS Source
• FLUME-2971 - Add secure Kafka Sink/Source/Channel setup to the User Guide
• FLUME-2974 - Fix some broken SpoolDirSource unit tests
• FLUME-2998 - Add a missing configuration parameter to SequenceSource docs
• FLUME-2999 - Kafka channel and sink should enable statically assigned partition per event via header
• FLUME-3027 - Change Kafka Channel to clear offsets map after commit
• FLUME-3049 - Make HDFS sink rotate more reliably in secure mode
• HADOOP-10823 - TestReloadingX509TrustManager is flaky
• HADOOP-11447 - Add a more meaningful toString method to SampleStat and MutableStat.
• HADOOP-11619 - FTPFileSystem should override getDefaultPort.
• HADOOP-11859 - PseudoAuthenticationHandler fails with httpcomponents v4.4
• HADOOP-11878 - FileContext#fixRelativePart should check for not null for a more informative exception.
• HADOOP-12009 - Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus.
• HADOOP-12655 - TestHttpServer.testBindAddress bind port range is wider than expected.
• HADOOP-12667 - s3a to support createNonRecursive API
• HADOOP-12774 - s3a should use UGI.getCurrentUser.getShortname() for username
• HADOOP-12977 - s3a to handle delete("/", true) robustly
• HADOOP-13037 - Refactor Azure Data Lake Store as an independent FileSystem
• HADOOP-13119 - Add ability to secure log servlet using proxy users. Contribute by Yuanbo Liu.
• HADOOP-13164 - Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories
• HADOOP-13204 - Add support for SSE-KMS and SSE-C in s3a filesystem.
• HADOOP-13207 - Specify FileSystem listStatus, listFiles and RemoteIterator
• HADOOP-13257 - Improve Azure Data Lake contract tests
• HADOOP-13309 - Document S3A known limitations in file ownership and permission model
• HADOOP-13336 - S3A to support per-bucket configuration (S3Guard specific changes)
• HADOOP-13345 - S3Guard: Ensure GenericOptionsParser is used for S3Guard CLI
• HADOOP-13375 - o.a.h.security.TestGroupsCaching.testBackgroundRefreshCounters seems flaky.
• HADOOP-13433 - Race in UGI.reloginFromKeytab
• HADOOP-13446 - Support running isolated unit tests separate from AWS integration tests
• HADOOP-13448 - S3Guard: Define MetadataStore interface.
• HADOOP-13449 - S3Guard: Implement DynamoDBMetadataStore
• HADOOP-13452 - S3Guard: Implement access policy for intra-client consistency with in-memory metadata store
• HADOOP-13455 - S3Guard: Write end user docs, change table autocreate default.
• HADOOP-13496 - Include file lengths in Mismatch in length error for distcp
• HADOOP-13502 - Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests
• HADOOP-13508 - FsPermission string constructor does not recognize sticky bit
• HADOOP-13540 - improve section on troubleshooting s3a auth problems
• HADOOP-13560 - S3ABlockOutputStream to support huge (many GB) file writes
• HADOOP-13573 - S3Guard: create basic contract tests for MetadataStore implementations
• HADOOP-13589 - S3Guard: Allow execution of all S3A integration tests with S3Guard enabled
• HADOOP-13599 - s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown
• HADOOP-13614 - Purge some superfluous/obsolete S3 FS tests that are slowing test runs down
• HADOOP-13627 - Have an explicit KerberosAuthException for UGI to throw, text from public constants
• HADOOP-13631 - S3Guard: implement move() for LocalMetadataStore, add unit tests
• HADOOP-13650 - S3Guard: Provide command line tools to manipulate metadata store.
• HADOOP-13651 - S3Guard: S3AFileSystem Integration with MetadataStore
• HADOOP-13674 - S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint
• HADOOP-13680 - fs.s3a.readahead.range to use getLongBytes
• HADOOP-13692 - hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts
• HADOOP-13727 - S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider
• HADOOP-13735 - ITestS3AFileContextStatistics.testStatistics() failing
• HADOOP-13742 - Expose NumOpenConnectionsPerUser as a metric.
• HADOOP-13749 - KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
• HADOOP-13782 - Make MutableRates metrics thread-local write, aggregate-on-read
• HADOOP-13793 - S3guard: add inconsistency injection, integration tests
• HADOOP-13801 - regression: ITestS3AMiniYarnCluster failing
• HADOOP-13804 - MutableStat mean loses accuracy if add(long, long) is used
• HADOOP-13805 - UGI.getCurrentUser() fails if user does not have a keytab associated
• HADOOP-13823 - s3a rename: fail if dest file exists
• HADOOP-13826 - S3A Deadlock in multipart copy due to thread pool limits
• HADOOP-13830 - Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string"
• HADOOP-13847 - KMSWebApp should close KeyProviderCryptoExtension
• HADOOP-13850 - s3guard to log choice of metadata store at debug
• HADOOP-13855 - Fix a couple of the s3a statistic names to be consistent with the rest
• HADOOP-13857 - S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs
• HADOOP-13876 - S3Guard: better support for multi-bucket access
• HADOOP-13877 - S3Guard: fix TestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is set
• HADOOP-13886 - s3guard: ITestS3AFileOperationCost.testFakeDirectoryDeletion failure
• HADOOP-13893 - dynamodb dependency -> compile
• HADOOP-13899 - S3Guard: tune dynamodb client & tests
• HADOOP-13900 - Remove snapshot version of SDK dependency from Azure Data Lake Store File System
• HADOOP-13903 - Improvements to KMS logging to help debug authorization errors.
• HADOOP-13904 - DynamoDBMetadataStore to handle DDB throttling
• HADOOP-13908 - S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
• HADOOP-13911 - Remove TRUSTSTORE_PASSWORD related scripts from KMS
• HADOOP-13914 - S3guard: Improve S3AFileStatus#isEmptyDirectory handling
• HADOOP-13928 - TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime error.
• HADOOP-13929 - ADLS connector should not check in contract-test-options.xml.
• HADOOP-13931 - S3AGuard: Use BatchWriteItem in DynamoDBMetadataStore#put()
• HADOOP-13934 - S3Guard: DynamoDBMetadataStore#move() could be throwing exception due to BatchWriteItem limits
• HADOOP-13937 - S3Guard: Mock bucket locations in MockS3ClientFactory
• HADOOP-13953 - Make FTPFileSystem's data connection mode and transfer mode configurable
• HADOOP-13956 - Read ADLS credentials from Credential Provider.
• HADOOP-13958 - Bump up release year to 2017
• HADOOP-13960 - Initialize DynamoDBMetadataStore without associated S3AFileSystem.
• HADOOP-13962 - Update ADLS SDK to 2.1.4.
• HADOOP-13976 - Path globbing does not match newlines
• HADOOP-13985 - s3guard: add a version marker to every table
• HADOOP-13988 - KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
• HADOOP-13995 - s3guard cli: make tests easier to run and address failure
• HADOOP-14001 - Improve delegation token validity checking.
• HADOOP-14003 - Make additional KMS tomcat settings configurable
• HADOOP-14013 - S3Guard: fix multi-bucket integration tests
• HADOOP-14017 - User friendly name for ADLS user and group
• HADOOP-14020 - Optimize dirListingUnion
• HADOOP-14027 - S3Guard: Implicitly creating DynamoDB table ignores endpoint config
• HADOOP-14029 - Fix KMSClientProvider for non-secure proxyuser use case
• HADOOP-14040 - Use shaded aws-sdk uber-JAR 1.11.86
• HADOOP-14041 - CLI command to prune old metadata
• HADOOP-14044 - Synchronization issue in delegation token cancel functionality
• HADOOP-14046 - Metastore destruction test creates table w/o version
• HADOOP-14049 - Honour AclBit flag associated to file/folder permission for Azure datalake account
• HADOOP-14050 - Add process name to kms process
• HADOOP-14079 - s3guard: fix breaking link in s3guard.md
• HADOOP-14083 - KMS should support old SSL clients.
• HADOOP-14085 - s3guard: drop unnecessary type assertion and cast.
• HADOOP-14094 - Rethink S3GuardTool options
• HADOOP-14096 - s3guard: regression in dirListingUnion
• HADOOP-14102 - Relax error message assertion in S3A test ITestS3AEncryptionSSEC
• HADOOP-14110 - In S3AFileSystem, make getAmazonClient() package private; export getBucketLocation()
• HADOOP-14113 - Review ADL Docs
• HADOOP-14114 - S3A can no longer handle unencoded + in URIs
• HADOOP-14125 - s3guard tool tests aren't isolated; can't run in parallel
• HADOOP-14129 - ITestS3ACredentialsInURL sometimes fails
• HADOOP-14130 - Simplify DynamoDBClientFactory for creating Amazon DynamoDB clients
• HADOOP-14131 - kms.sh creates bogus dir for tomcat logs
• HADOOP-14168 - S3GuardTool tests should not run if S3Guard is not set up
• HADOOP-14181 - S3Guard: Add validation of DynamoDB region
• HDFS-3716 - Purger should remove stale fsimage ckpt files
• HDFS-4366 - Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks.
• HDFS-8307 - Spurious DNS Queries from hdfs shell
• HDFS-8640 - Make reserved RBW space visible through JMX.
• HDFS-8883 - NameNode Metrics : Add FSNameSystem lock Queue Length
• HDFS-9145 - Tracking methods that hold FSNamesytemLock for too long
• HDFS-9205 - Do not schedule corrupt blocks for replication.
• HDFS-9390 - Block management for maintenance states.
• HDFS-9391 - Update webUI/JMX to display maintenance state info.
• HDFS-9467 - Fix data race accessing writeLockHeldTimeStamp in FSNamesystem
• HDFS-10292 - Add block id when client got Unable to close file exception
• HDFS-10534 - NameNode WebUI should display DataNode usage histogram
• HDFS-10559 - DiskBalancer: Use SHA1 for Plan ID
• HDFS-10713 - Throttle FsNameSystem lock warnings
• HDFS-10733 - NameNode terminated after full GC thinking QJM is unresponsive
• HDFS-10798 - Make the threshold of reporting FSNamesystem lock contention configurable
• HDFS-10808 - DiskBalancer does not execute multi-steps plan-redux
• HDFS-10817 - Add Logging for Long-held NN Read Locks
• HDFS-10896 - Move lock logging logic from FSNamesystem into FSNamesystemLock
• HDFS-10941 - Improve BlockManager#processMisReplicatesAsync log
• HDFS-10966 - Enhance Dispatcher logic on deciding when to give up a source DataNode
• HDFS-11087 - NamenodeFsck should check if the output writer is still writable
• HDFS-11132 - Allow AccessControlException in contract tests when getFileStatus on subdirectory of existing files
• HDFS-11160 - VolumeScanner reports write-in-progress replicas as corrupt incorrectly
• HDFS-11180 - Intermittent deadlock in NameNode when failover happens.
• HDFS-11197 - Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
• HDFS-11259 - Update fsck to display maintenance state info.
• HDFS-11263 - ClassCastException when we use Bzipcodec for Fsimage compression
• HDFS-11265 - Extend visualization for Maintenance Mode under Datanode tab in the NameNode UI.
• HDFS-11271 - Typo in NameNode UI
• HDFS-11275 - Check groupEntryIndex and throw a helpful exception on failures when removing ACL.
• HDFS-11280 - Allow WebHDFS to reuse HTTP connections to NN
• HDFS-11292 - log lastWrittenTxId etc info in logSyncAll
• HDFS-11296 - Maintenance state expiry should be an epoch time and not jvm monotonic.
• HDFS-11306 - Print remaining edit logs from buffer if edit log can't be rolled
• HDFS-11363 - Need more diagnosis info when seeing Slow waitForAckedSeqno.
• HDFS-11378 - Verify multiple DataNodes can be decommissioned/maintenance at the same time.
• HDFS-11379 - DFSInputStream may infinite loop requesting block locations
• HDFS-11390 - Add process name to httpfs process
• HDFS-11411 - Avoid OutOfMemoryError in TestMaintenanceState test runs.
• HDFS-11412 - Maintenance minimum replication config value allowable range should be [0, DefaultReplication].
• HDFS-11418 - HttpFS should support old SSL clients
• HDFS-11441 - Add escaping to error message in KMS web UI
• HDFS-11466 - Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms
• MAPREDUCE-5155 - Race condition in test case TestFetchFailure cause it to fail
• MAPREDUCE-5335 - Rename Job Tracker terminology in ShuffleSchedulerImpl
• MAPREDUCE-5883 - "Total megabyte-seconds" in job counters is slightly misleading
• MAPREDUCE-5981 - Log levels of certain MR logs can be changed to DEBUG
• MAPREDUCE-6172 - TestDbClasses timeouts are too aggressive
• MAPREDUCE-6571 - JobEndNotification info logs are missing in AM container syslog
• MAPREDUCE-6639 - Process hangs in LocatedFileStatusFetcher if FileSystem.get throws.
• MAPREDUCE-6715 - Fix Several Unsafe Practices
• MAPREDUCE-6817 - The format of job start time in JHS is different from those of submit and finish time.
• MAPREDUCE-6829 - Add peak memory usage counter for each task.
• YARN-1047 - Expose # of pre-emptions as a queue counter
• YARN-2301 - Improved yarn container command
• YARN-2306 - Add test for leakage of reservation metrics in fair scheduler.
• YARN-2336 - Fair scheduler's REST API returns a missing '[' bracket JSON for deep queue tree
• YARN-2902 - Addendum. Killing a container that is localizing can orphan resources in the DOWNLOADING state
• YARN-2988 - Graph#save() may leak file descriptors.
• YARN-3269 - Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path
• YARN-3396 - Handle URISyntaxException in ResourceLocalizationService.
• YARN-3933 - FairScheduler: Multiple calls to completedContainer are not safe.
• YARN-3957 - FairScheduler NPE In FairSchedulerQueueInfo causing scheduler page to return 500.
• YARN-4232 - TopCLI console support for HA mode
• YARN-4363 - In TestFairScheduler, testcase should not create FairScheduler redundantly.
• YARN-4544 - All the log messages about rolling monitoring interval are shown with WARN level
• YARN-4555 - TestDefaultContainerExecutor#testContainerLaunchError fails on non-english locale environment
• YARN-4719 - Add a helper library to maintain node state and allows common queries.
• YARN-4752 - Improved preemption in FairScheduler.
• YARN-4805 - Don't go through all schedulers in ParameterizedTestBase.
• YARN-4882 - Change the log level to DEBUG for recovering completed applications
• YARN-5136 - Error in handling event type APP_ATTEMPT_REMOVED to the scheduler
• YARN-5181 - ClusterNodeTracker: add method to get list of nodes matching a specific resourceName.
• YARN-5182 - MockNodes.newNodes creates one more node per rack than requested.
• YARN-5257 - Fix unreleased resources and null dereferences
• YARN-5308 - FairScheduler: Move continuous scheduling related tests to TestContinuousScheduling
• YARN-5752 - TestLocalResourcesTrackerImpl#testLocalResourceCache times out
• YARN-5798 - Set UncaughtExceptionHandler for all FairScheduler threads.
• YARN-5830 - FairScheduler: Avoid preempting AM containers.
• YARN-5859 - TestResourceLocalizationService#testParallelDownloadAttemptsForPublicResource sometimes fails
• YARN-5890 - FairScheduler should log information about AM-resource-usage and max-AM-share for queues
• YARN-5920 - Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang
• YARN-6061 - Add an UncaughtExceptionHandler for critical threads in RM.
• YARN-6112 - UpdateCallDuration is calculated only when debug logging is enabled.
• YARN-6125 - The application attempt's diagnostic message should have a maximum size
• YARN-6131 - FairScheduler: Lower update interval for faster tests.
• YARN-6144 - FairScheduler: preempted resources can become negative.
• YARN-6163 - FS Preemption is a trickle for severely starved applications.
• YARN-6171 - ConcurrentModificationException on FSAppAttempt.containersToPreempt.
• YARN-6172 - FSLeafQueue demand update needs to be atomic.
• YARN-6193 - FairScheduler might not trigger preemption when using DRF.
• YARN-6194 - Cluster capacity in SchedulingPolicy is updated only on allocation file reload.
• YARN-6210 - FairScheduler: Node reservations can interfere with preemption.
• YARN-6215 - Custom Cloudera version. FairScheduler preemption and update should not run concurrently.
• YARN-6218 - Fix TestAMRMClient when using FairScheduler.
• YARN-6264 - AM not launched when a single vcore is available on the cluster.
• YARN-6448 - Continuous scheduling thread crashes while sorting nodes.
• HBASE-6721 - RegionServer Group based Assignment. Adds new rsgroups feature which allows isol
• HBASE-12949 - Scanner can be stuck in infinite loop if the HFile is corrupted
• HBASE-15295 - MutateTableAccess.multiMutate() does not get high priority causing a deadlock
• HBASE-15315 - Remove always set super user call as high priority
• HBASE-15322 - Operations using Unsafe path broken for platforms not having sun.misc.Unsafe.
• HBASE-15325 - ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
• HBASE-15328 - sanity check the redirect used to send master info requests to the embedded regionserver.
• HBASE-15378 - Scanner cannot handle heartbeat message with no results
• HBASE-15435 - Add WAL (in bytes) written metric
• HBASE-15441 - Fix WAL splitting when region has moved multiple times
• HBASE-15587 - FSTableDescriptors.getDescriptor() logs stack trace erronously
• HBASE-15645 - hbase.rpc.timeout is not used in operations of HTable
• HBASE-15931 - Add log for long-running tasks in AsyncProcess
• HBASE-16062 - Improper error handling in WAL Reader/Writer creation
• HBASE-16165 - Decrease RpcServer.callQueueSize before writeResponse causes OOM
• HBASE-16237 - Blocks for hbase:meta table are not cached in L1 cache
• HBASE-16266 - Do not throw ScannerTimeoutException when catch UnknownScannerException
• HBASE-16304 - HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
• HBASE-16460 - Can't rebuild the BucketAllocator's data structures when BucketCache uses FileIOEngine
• HBASE-16527 - IOExceptions from DFS client still can cause CatalogJanitor to delete referenced files
• HBASE-16604 - Scanner retries on IOException can cause the scans to miss data
• HBASE-16649 - Truncate table with splits preserved can cause both data loss and truncated data appeared again
• HBASE-16662 - Fix open POODLE vulnerabilities
• HBASE-16754 - All WALSplitter OutputSinks should process compaction events
• HBASE-16765 - backportNew SteppingRegionSplitPolicy, avoid too aggressive spread of regions for small tables. Includesand the amendmentAmend: Make SteppingSplitPolicy the default.
• HBASE-16807 - , RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover.
• HBASE-16841 - Data loss in MOB files after cloning a snapshot and deleting that snapshot
• HBASE-16931 - Setting cell's seqId to zero in compaction flow might cause RS down.
• HBASE-16960 - RegionServer hang when aborting
• HBASE-16981 - Expand Mob Compaction Partition policy from daily to weekly, monthly
• HBASE-17020 - keylen in midkey() dont computed correctly
• HBASE-17023 - Region left unassigned due to AM and SSH each thinking others would do the assignment work
• HBASE-17044 - Fix merge failed before creating merged region leaves meta inconsistent
• HBASE-17069 - RegionServer writes invalid META entries for split daughters in some circumstances
• HBASE-17206 - FSHLog may roll a new writer successfully with unflushed entries
• HBASE-17238 - Wrong in-memory hbase:meta location causing SSH failure
• HBASE-17241 - Avoid compacting already compacted mob files with _del files
• HBASE-17243 - Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects
• HBASE-17243 - Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects
• HBASE-17275 - Assign timeout may cause region to be unassigned forever
• HBASE-17309 - Fix connection leaks in TestAcidGuarantees
• HBASE-17381 - ReplicationSourceWorkerThread can die due to unhandled exceptions
• HBASE-17409 - Limit jsonp callback name to prevent xss
• HBASE-17437 - Support specifying a WAL directory outside of the root directory
• HBASE-17452 - Failed taking snapshot - region Manifest proto-message too large
• HBASE-17522 - Handle JVM throwing runtime exceptions when we ask for details on heap usage the same as a correctly returned 'undefined'.
• HBASE-17558 - ZK dumping jsp should escape HTML.
• HBASE-17561 - table status page should escape values that may contain arbitrary characters.
• HBASE-17624 - ADDENDUMincluding missing parts of TestWALObserver changes from
• HBASE-17640 - Unittest error in TestMobCompactor with different timezone
• HBASE-17653 - HBASE-17624 rsgroup synchronizations will (distributed) deadlock
• HBASE-17654 - RSGroup refactoring.
• HBASE-17675 - ReplicationEndpoint should choose new sinks if a SaslException occurs
• HBASE-17677 - ServerName parsing from directory name should be more robust to errors from guava's HostAndPort
• HIVE-7723 - Explain plan for complex query with lots of partitions is slow due to in-efficient collection used to find a matching ReadEntity
• HIVE-9138 - Add some explain to PTF operator
• HIVE-9481 - allow column list specification in INSERT statement
• HIVE-10641 - create CRC32 UDF
• HIVE-11147 - MetaTool doesn't update FS root location for partitions with space in name
• HIVE-11293 - HiveConnection.setAutoCommit(true) throws exception
• HIVE-11376 - Removes legacy code wrt skipping files with codec to HiveInputFormat from CombineHiveInputFormat
• HIVE-11594 - Analyze Table for column names with embedded spaces
• HIVE-11849 - NPE in HiveHBaseTableShapshotInputFormat in query with just count(*)
• HIVE-11882 - Fetch optimizer should stop source files traversal once it exceeds the hive.fetch.task.conversion.threshold
• HIVE-11920 - ADD JAR failing with URL schemes other than file/ivy/hdfs
• HIVE-12222 - Define port range in property for RPCServer
• HIVE-12236 - Enable SimpleFetchOptimizer for more query types
• HIVE-12261 - schematool version info exit status should depend on compatibility, not equality
• HIVE-12349 - NPE in ORC SARG for IS NULL queries on Timestamp and Date columns
• HIVE-12465 - Hive might produce wrong results when (outer) joins are merged
• HIVE-12780 - Fix the output of the history command in Beeline
• HIVE-12789 - Fix output twice in the history command of Beeline
• HIVE-12897 - Improve dynamic partition loading
• HIVE-12976 - MetaStoreDirectSql doesn't batch IN lists in all cases
• HIVE-13149 - Remove some unnecessary HMS connections from HS2
• HIVE-13179 - Allow custom HiveConf to be passed to Authentication Providers
• HIVE-13240 - GroupByOperator: Drop the hash aggregates when closing operator
• HIVE-13463 - Fix ImportSemanticAnalyzer to allow for different src/dst filesystems
• HIVE-13510 - Dynamic partitioning doesn’t work when remote metastore is used
• HIVE-13726 - Improve dynamic partition loading VI
• HIVE-13864 - Beeline ignores the command that follows a semicolon and comment
• HIVE-13866 - flatten callstack for directSQL errors
• HIVE-13895 - HoS start-up overhead in yarn-client mode
• HIVE-13901 - Hivemetastore add partitions can be slow depending on filesystems
• HIVE-13902 - [Refactor] Minimize metastore jar dependencies on task nodes
• HIVE-13904 - Ignore case when retrieving ColumnInfo from RowResolver
• HIVE-13966 - DbNotificationListener: can loose DDL operation notifications
• HIVE-14204 - Optimize loading dynamic partitions
• HIVE-14423 - S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore
• HIVE-14511 - Improve MSCK for partitioned table to deal with special cases Updated changed q.out
• HIVE-14693 - Some paritions will be left out when partition number is the multiple of the option hive.msck.repair.batch.size
• HIVE-14713 - LDAP Authentication Provider should be covered with unit tests
• HIVE-14764 - Enabling "hive.metastore.metrics.enabled" throws OOM in HiveMetastore
• HIVE-14803 - S3: Stats gathering for insert queries can be expensive for partitioned dataset
• HIVE-14819 - FunctionInfo for permanent functions shows TEMPORARY FunctionType
• HIVE-14820 - RPC server for spark inside HS2 is not getting server address properly
• HIVE-14873 - Add UDF for extraction of 'day of week'
• HIVE-14920 - S3: Optimize SimpleFetchOptimizer::checkThreshold()
• HIVE-15025 - Secure-Socket-Layer (SSL) support for HMS
• HIVE-15060 - Remove the autoCommit warning from beeline
• HIVE-15065 - SimpleFetchOptimizer should decide based on metastore stats when available
• HIVE-15137 - metastore add partitions background thread should use current username
• HIVE-15207 - Implement a capability to detect incorrect sequence numbers
• HIVE-15072 - Schematool should recognize missing tables in metastore
• HIVE-15073 - Schematool should detect malformed URIs
• HIVE-12261 - Schematool version info exit status should depend on compatibility, not equality
• HIVE-15074 - Schematool provides a way to detect invalid entries in VERSION table
• HIVE-15263 - Detect the values for incorrect NULL values
• HIVE-15391 - Location validation for table should ignore the values for view
• HIVE-15392 - Refactoring the validate function of HiveSchemaTool to make the output consistent
• HIVE-15232 - Add notification events for functions and indexes
• HIVE-15338 - Wrong result from non-vectorized DATEDIFF with scalar parameter of type DATE/TIMESTAMP
• HIVE-15346 - "values temp table" should not be an input
• HIVE-15359 - CDH=48042skip.footer.line.count doesnt work properly for certain situations
• HIVE-15410 - WebHCat supports get/set table property with its name containing period and hyphen
• HIVE-15430 - Change SchemaTool table validator to test based on the dbType
• HIVE-15485 - Investigate the DoAs failure in HoS
• HIVE-15517 - NOT (x <=> y) returns NULL if x or y is NULL
• HIVE-15530 - Optimize the column stats update logic in table alteration
• HIVE-15546 - Optimize Utilities.getInputPaths() so each listStatus of a partition is done in parallel
• HIVE-15550 - fix arglist logging in schematool
• HIVE-15551 - memory leak in directsql for mysql+bonecp specific initialization
• HIVE-15572 - Improve the response time for query canceling when it happens during acquiring locks
• HIVE-15735 - In some cases, view objects inside a view do not have parents.
• HIVE-15754 - exchange partition is not generating notifications
• HIVE-15766 - DBNotificationlistener leaks JDOPersistenceManager : Use same PersistenceManager for metadata and notification
• HIVE-15778 - DROP INDEX (non-existent) throws NPE when using DbNotificationListener
• HIVE-15803 - msck can hang when nested partitions are present
• HIVE-15805 - Some minor improvement on the validation tool
• HIVE-15815 - Allow to pass some Oozie properties to Spark in HoS
• HIVE-15872 - The PERCENTILE_APPROX UDAF does not work with empty set
• HIVE-15879 - Fix HiveMetaStoreChecker.checkPartitionDirs method
• HIVE-15881 - Use hive.exec.input.listing.max.threads variable name instead of mapred.dfsclient.parallelism.max
• HIVE-15895 - Use HDFS for stats collection temp dir on blob storage
• HIVE-16014 - HiveMetastoreChecker should use hive.metastore.fshandler.threads instead of hive.mv.files.thread for pool size
• HIVE-16019 - Query fails when group by/order by on same column with uppercase name
• HIVE-16086 - Fix HiveMetaStoreChecker.checkPartitionDirsSingleThreaded method
• HIVE-16176 - Fixing the non-zero exit code issue with HiveSchemaTool
• HUE-5167 - Improve SASL error reporting in HUE
• HUE-5482 - [home] Handle multiple home/trash directories by merging them into one
• HUE-5594 - [editor] The new autocomplete dropdown should include popular group by and order by clauses
• HUE-5659 - [home] Ignore history dependencies when importing document from different cluster
• HUE-5701 - [security] Remove last references to $.cookie and migrate to $.totalStorage from FB
• IMPALA-1430 - IMPALA-4878,IMPALA-4879: codegen native UDAs
• IMPALA-2020 - 4915, 4936: Add rounding for decimal casts
• IMPALA-2020 - IMPALA-4809: Codegen support for DECIMAL_V2
• IMPALA-2605 - Omit the sort and mini stress tests
• IMPALA-4055 - Speed up to_date() with custom implementation.
• IMPALA-4263 - Fix wrong ommission of agg/analytic hash exchanges.
• IMPALA-4282 - Remove max length check for type strings.
• IMPALA-4370 - Divide and modulo result types for DECIMAL version V2
• IMPALA-4449 - Revisit table locking pattern in the catalog
• IMPALA-4675 - Case-insensitive matching of Parquet fields.
• IMPALA-4702 - Fix command line help for webserver_private_key_file
• IMPALA-4705 - IMPALA-4779, IMPALA-4780: Fix some Expr bugs with codegen
• IMPALA-4725 - Query option to control Parquet array resolution.
• IMPALA-4729 - Implement REPLACE()
• IMPALA-4742 - Change "{}".format() to "{0}".format() for Py 2.6
• IMPALA-4749 - hit DCHECK in sorter with scratch limit
• IMPALA-4767 - Workaround for HIVE-15653 to preserve table stats.
• IMPALA-4808 - old hash join can reference invalid memory
• IMPALA-4809 - Enable support for DECIMAL_V2 in decimal_casting.py
• IMPALA-4810 - fix incorrect expr-test decimal types
• IMPALA-4810 - Make DECIMAL expr-test cases table driven
• IMPALA-4810 - Add DECIMAL_V2 query option
• IMPALA-4813 - Round on divide and multiply
• IMPALA-4821 - Update AVG() for DECIMAL_V2
• IMPALA-4828 - Alter Kudu schema outside Impala may crash on read
• IMPALA-4854 - Fix incremental stats with complex types.
• IMPALA-4916 - Fix maintenance of set of item sets in DisjointSet.
• IMPALA-4929 - Safe concurrent access to IR function call graph
• IMPALA-4933 - IMPALA-4931: Simplify SSL initialization on startup
• IMPALA-4934 - Disable Kudu OpenSSL initialization
• IMPALA-4981 - Re-enable spilling with MT_DOP.
• IMPALA-4995 - Fix integer overflow in TopNNode::PrepareForOutput
• IMPALA-4997 - Fix overflows in Sorter::TupleIterator
• IMPALA-5005 - Don't allow server to send SASL COMPLETE msg out of order
• IMPALA-5025 - Update binutils to 2.26.1
• IMPALA-5027 - make udf headers buildable externally
• OOZIE-2243 - Kill Command does not kill the child job for java action
• OOZIE-2519 - Oozie HA with SSL info is slightly incorrect
• OOZIE-2584 - Eliminate Thread.sleep() calls in TestMemoryLocks
• OOZIE-2659 - TestPauseTransitService is flaky
• OOZIE-2720 - Test failure - TestCoordMaterializeTriggerService#testMaxMatThrottleNotPicked
• OOZIE-2742 - Unable to kill applications based on tag
• OOZIE-2748 - NPE in LauncherMapper.printArgs()
• OOZIE-2757 - Malformed xml is Spark action doc page
• OOZIE-2777 - Config-default.xml longer than 64k results in java.io.UTFDataFormatException
• OOZIE-2786 - Pass Oozie workflow ID and settings to Spark application configuration
• OOZIE-2787 - Oozie distributes application jar twice making the spark job fail
• OOZIE-2792 - Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
• OOZIE-2802 - Spark action failure on Spark 2.1.0 due to duplicate sharelibs
• OOZIE-2803 - Mask passwords when printing out configs/args in MapReduceMain and SparkMain
• OOZIE-2811 - Add support for filtering out properties from SparkConfigurationService
• SENTRY-1476 - SentryStore is subject to JDQL injection. Sentry is subject to JDOQL injection
• SENTRY-1504 - NPE in log4j.properties parsingSentryMain trips over missing log4j property
• SENTRY-1505 - CommitContext isn't used by anything and should be removed
• SENTRY-1507 - Sentry should use Datanucleus version of javax.jdo
• SENTRY-1508 - MetastorePlugin.java does not handle properly initialization failure
• SENTRY-1515 - Cleanup exception handling in SentryStore
• SENTRY-1517 - SentryStore should actually use function getMSentryRole to get roles
• SENTRY-1518 - Add metrics for SentryStore transactions
• SENTRY-1520 - Provide mechanism for triggering HMS full snapshot
• SENTRY-1533 - Sentry console metrics reporting interval should be configurable
• SENTRY-1534 - Oracle supports serializable instead of repeatable-read
• SENTRY-1557 - getRolesForGroups(),getRoleNamesForGroups() does too many trips to the DB
• SENTRY-1564 - Improve error detection and reporting in MetastoreCacheInitializer.java
• SENTRY-1577 - Support "create function using jar" for hive when Sentry is enabled
• SENTRY-1581 - Provide Log4J metrics reporter
• SENTRY-1582 - Additional comments to clarify the intent of string manipulation methods in SentryStore.java
• SENTRY-1594 - TransactionBlock should become generic
• SENTRY-1599 - CloseablePersistenceManager is no longer needed
• SENTRY-1609 - DelegateSentryStore is subject to JDQL injection
• SENTRY-1615 - SentryStore should not allocate empty objects that are immediately returned
• SENTRY-1625 - PrivilegeOperatePersistence can use QueryParamBuilder Fix [SQL Injection: JDO] in [sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/generic/service/persistent/PrivilegeOperatePersistence.java]
• SENTRY-1636 - Remove thrift dependency on fb303
• SOLR-9284 - The HDFS BlockDirectoryCache should not let it's keysToRelease or names maps grow indefinitely.
• SOLR-9330 - Fix AlreadyClosedException on admin/mbeans?stats=true
• SOLR-9389 - HDFS Transaction logs stay open for writes which leaks Xceivers.
• SOLR-9506 - cache IndexFingerprint for each segment
• SOLR-9699, SOLR-4668 - fix exception from core status in parallel with core reload
• SOLR-9819 - Upgrade Apache commons-fileupload to 1.3.2, fixing a security vulnerability
• SOLR-9859 - backport of replication.properties cannot be updated after being written and neither replication.properties nor index.properties are durable after a crash. Don't log error on NoSuchFileException
• SOLR-9901 - backport of SOLR-9899 Implement move in HdfsDirectoryFactory. SOLR-9899: StandardDirectoryFactory should use optimizations for all FilterDirectorys not just NRTCachingDirectory.
• SOLR-10031 - Validation of filename params in ReplicationHandler
• SOLR-10114, SOLR-9941 - Reordered delete-by-query can delete or omit child documents
• SOLR-10119 - TestReplicationHandler assertion fixes part of
• SOLR-10121, SOLR-10116 - BlockCache corruption with high concurrency
• SPARK-4105 - retry the fetch or stage if shuffle block is corrupt
• SPARK-12241 - [YARN] Improve failure reporting in Yarn client obtainTokenForHBase()
• SPARK-12523 - [YARN] Support long-running of the Spark On HBase and hive meta store.
• SPARK-16654 - [CORE] Add UI coverage for Application Level Blacklisting. Automatically Kill Executors and Nodes when they are Blacklisted
• SPARK-17874 - [CORE] Add SSL port configuration.
• SPARK-18372 - [SQL][BRANCH-1.6] Staging directory fail to be removed
• SPARK-18750 - [YARN] Avoid using "mapValues" when allocating containers.
• SPARK-19033 - [CORE] Add admin acls for history server
• SPARK-19220 - [UI] Make redirection to HTTPS apply to all URIs.
• SPARK-19554 - [UI,YARN] Allow SHS URL to be used for tracking in YARN RM.
• SPARK-19652 - [UI] Do auth checks for REST API access.
• SPARK-19720 - [CORE] Redact sensitive information from SparkSubmit console
• SQOOP-2349 - Add command line option for setting transaction isolation levels for metadata queries
• SQOOP-2896 - Sqoop exec job fails with SQLException Access denied for user
• SQOOP-2909 - Oracle related ImportTest fails after SQOOP-2737
• SQOOP-2911 - Fix failing HCatalogExportTest caused by SQOOP-2863
• SQOOP-2950 - Sqoop trunk has consistent UT failures - need fixing
• SQOOP-3053 - Create a cmd line argument for sqoop.throwOnError and use it through SqoopOptions
• SQOOP-3055 - Fixing MySQL tests failing due to ignored test inputs/configuration
• SQOOP-3057 - Fixing 3rd party Oracle tests failing due to invalid case of column names
• SQOOP-3066 - Introduce an option + env variable to enable/disable SQOOP-2737 feature
• SQOOP-3068 - Enhance error (tool.ImportTool: Encountered IOException running import job: java.io.IOException: Expected schema) to suggest workaround
• SQOOP-3069 - Get OracleExportTest#testUpsertTestExport in line with SQOOP-3066
• SQOOP-3071 - Fix OracleManager to apply localTimeZone correctly in case of Date objects too
• SQOOP-3072 - Reenable escaping in ImportTest#testProductWithWhiteSpaceImport for proper execution
• SQOOP-3081 - use OracleEscapeUtils.escapeIdentifier in OracleUpsertOutputFormat instead of inline appending quotes
• SQOOP-3124 - Fix ordering in column list query of PostgreSQL connector to reflect the logical order instead of adhoc ordering
• ZOOKEEPER-1045 - Backport — Support Quorum Peer mutual authentication via SASL
• ZOOKEEPER-2044 - Backport — Fix CancelledKeyException on ZooKeeper server (zookeeper branch-3.4)

Security vulnerability resolved

An attacker can download any file accessible to Solr superuser using the path manipulation vulnerability in Solr (CVE-2017-3163; see Cloudera Security Bulletins more information). This release provides the fix for it. In addition, attackers can leverage this vulnerability to manipulate files unrelated to Solr index, by launching a “man in the middle” attack between individual Solr servers. To avoid this issue, enable TLS and Kerberos authentication.