Fixed Issues in CDH 6.3.1

The Idempotent and Transactional Capabilities of Kafka are Incompatible with Sentry

The idempotent and transactional capabilities of Kafka are not compatible with Sentry. The issue is due to Sentry being unable to handle authorization policies for Kafka transactions. As a result, users cannot use Kafka transaction in combination with Sentry.

Workaround: Use the Sentry super user in applications where idempotent producing is a requirement or disable Sentry.

Affected Versions: CDK 4.0 and later, CDH 6.0.0, 6.0.1, 6.1.0, 6.1.1, 6.2.0, 6.3.0

Fixed Versions: CDH 6.2.1, 6.3.1

Apache Issue: N/A

Cloudera Issue: CDH-80606

Upstream Issues Fixed

The following upstream issues are fixed in CDH 6.3.1:

Apache Accumulo

There are no notable fixed issues in this release.

Apache Avro

The following issue is fixed in CDH 6.3.1:

  • HIVE-17829 - Fixed ArrayIndexOutOfBoundsException that occurred when using HBASE-backed tables with Avro schema in Hive2

Apache Crunch

There are no notable fixed issues in this release.

Apache Flume

There are no notable fixed issues in this release.

Apache Hadoop

The following issue is fixed in CDH 6.3.1:

  • HADOOP-16018 - DistCp does not reassemble chunks when the value of blocks per chunk is greater than zero.

HDFS

The following issues are fixed in CDH 6.3.1:

  • HDFS-12828 - OIV ReverseXML Processor fails with escaped characters.
  • HDFS-13101 - An fsimage corruption related to snapshots.
  • HDFS-13709 - Report bad block to NameNode when transfer block encounters EIO exception
  • HDFS-14148 - HDFS OIV ReverseXML SnapshotSection parser throws exception when there is more than one snapshottable directory.
  • HDFS-14687 - Standby Namenode does not come out of safemode when EC files are being written.
  • HDFS-14706 - Checksums are not checked if the block meta file size is less than 7 bytes.

MapReduce 2

The following issue is fixed in CDH 6.3.1:

  • MAPREDUCE-7225 - Fix broken current folder expansion during MR job start

YARN

The following issues are fixed in CDH 6.3.1:

  • YARN-9667 - Container-executor.c duplicates messages to stdout
  • YARN-9833 - Race condition when DirectoryCollection.checkDirs() runs during container launch

Apache HBase

The following issues are fixed in CDH 6.3.1:

  • HBASE-19893 - restore_snapshot is broken in master branch when region splits
  • HBASE-20305 - adding options to skip deletes/puts on target when running SyncTable
  • HBASE-22169 - Open region failed cause memory leak
  • HBASE-22539 - WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used
  • HBASE-22617 - Recovered WAL directories not getting cleaned up
  • HBASE-22690 - Deprecate / Remove OfflineMetaRepair in hbase-2+
  • HBASE-22759 - Extended grant and revoke audit events with caller info

Apache Hive

The following issues are fixed in CDH 6.3.1:

  • HIVE-17829 - Fixed ArrayIndexOutOfBoundsException that occurred when using HBASE-backed tables with Avro schema in Hive2

Hue

The following issues are fixed in CDH 6.3.1:

  • HUE-8922 - [frontend] Show dates and times in local format with timezone offset details
  • HUE-8933 - [editor] Results are not properly cleared in multi-statement execution
  • HUE-8950 - [core] Saving newly copied Oozie workflow throws an exception
  • HUE-8979 - [jb] Oozie spark jobs display a NoneType object that is not iterable

Apache Impala

The following issues are fixed in CDH 6.3.1:

  • IMPALA-8549 - Added support for scanning DEFLATE text files.

  • IMPALA-8820 - Fixed an issue where the catalogd process was not found when Impala starts in a cluster.
  • IMPALA-8847 - The event based automatic metadata invalidation can now correctly ignore empty partition lists generated for certain Hive queries.

Apache Kafka

There are no notable fixed issues in this release.

Apache Kite

There are no notable fixed issues in this release.

Apache Kudu

There are no notable fixed issues in this release.

Apache Oozie

The following issues are fixed in CDH 6.3.1:

  • OOZIE-3397 - Improve logging in NotificationXCommand.
  • OOZIE-3542 - Handle better HDFS implementations in ECPolicyDisabler.

Apache Parquet

There are no notable fixed issues in this release.

Apache Pig

There are no notable fixed issues in this release.

Cloudera Search

There are no notable fixed issues in this release.

Apache Sentry

The following issues are fixed in CDH 6.3.1:

  • SENTRY-2276 - Sentry-Kafka integration does not support Kafka's Alter/DescribeConfigs and IdempotentWrite operations
  • SENTRY-2528 - Format exception when fetching a full snapshot

Apache Spark

The following issues are fixed in CDH 6.3.1:

  • SPARK-18364 - [YARN] Expose metrics for YarnShuffleService
  • SPARK-24352 - [CORE][TESTS] De-flake StandaloneDynamicAllocationSuite blacklist test
  • SPARK-24355 - Spark external shuffle server improvement to better handle block fetch requests.
  • SPARK-25139 - [SPARK-18406][CORE][2.4] Avoid NonFatals to kill the Executor in PythonRunner
  • SPARK-25641 - Change the spark.shuffle.server.chunkFetchHandlerThreadsPercent default to 100
  • SPARK-25642 - [YARN] Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service
  • SPARK-25692 - [CORE] Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite as well
  • SPARK-26615 - [CORE] Fixing transport server/client resource leaks in the core unittests
  • SPARK-27021 - [CORE] Cleanup of Netty event loop group for shuffle chunk fetch requests
  • SPARK-28150 - [CORE][FOLLOW-UP] Don't try to log in when impersonating.
  • SPARK-28150 - [CORE] Log in user before getting delegation tokens.
  • SPARK-28261 - [CORE] Fix client reuse test
  • SPARK-28335 - [DSTREAMS][TEST] DirectKafkaStreamSuite wait for Kafka async commit
  • SPARK-28584 - [CORE] Fix thread safety issue in blacklist timer, tests

Apache Sqoop

The following issue is fixed in CDH 6.3.1:

Apache ZooKeeper

There are no notable fixed issues in this release.