Fixed Issues in CDH 6.3.1

CDH 6.3.1 fixes the following issues:

Hue Silently Disables StartTLS in LDAP Connections

There are two mechanisms to secure communication to an LDAP server. One is to use an ‘ldaps’ connection, where all traffic is encrypted inside a TLS tunnel - much like ‘https’. The other is to use ‘StartTLS’, where traffic begins unencrypted in the “ldap” protocol and then upgrades itself to a TLS connection.

If StartTLS is enabled in the Hue configuration but the ‘ldap_cert’ parameter is not configured, then Hue silently disables StartTLS.

StartTLS will not be used for synchronization or import, even if StartTLS is enabled and the ‘ldap_cert’ parameter is set.

The result is that connections that the administrator assumes to be secured, using StartTLS, are not actually secure.

CVE: CVE-2019-19146

Date/time of detection: 22nd March, 2019

Detected by: Ben Gooley, Cloudera

Severity (Low/Medium/High): 8.8 High CVSS AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H

Products affected: CDH

Releases affected:
  • CDH 5.x
  • CDH 6.1.0
  • CDH 6.1.1
  • CDH 6.2.0
  • CDH 6.2.1
  • CDH 6.3.0

Users affected: All users who are using StartTLS enabled in the Hue configuration when using LDAP as Authentication Backend to login in Hue.

Impact: Sensitive data exposure.

Immediate action required:
  • Upgrade (recommended): Update to a version of CDH containing the fix.
  • Workaround: Use “ldaps” instead of “ldap” and StartTLS.

Addressed in release/refresh/patch: CDH 6.3.1 and above

Knowledge article: For the latest update on this issue see the corresponding Knowledge article: TSB 2020-371: Hue Silently Disables StartTLS in LDAP Connections

The Idempotent and Transactional Capabilities of Kafka are Incompatible with Sentry

The idempotent and transactional capabilities of Kafka are not compatible with Sentry. The issue is due to Sentry being unable to handle authorization policies for Kafka transactions. As a result, users cannot use Kafka transaction in combination with Sentry.

Workaround: Use the Sentry super user in applications where idempotent producing is a requirement or disable Sentry.

Affected Versions: CDK 4.0 and later, CDH 6.0.0, 6.0.1, 6.1.0, 6.1.1, 6.2.0, 6.3.0

Fixed Versions: CDH 6.2.1, 6.3.1

Apache Issue: N/A

Cloudera Issue: CDH-80606

Upstream Issues Fixed

The following upstream issues are fixed in CDH 6.3.1:

Apache Accumulo

There are no notable fixed issues in this release.

Apache Avro

The following issue is fixed in CDH 6.3.1:

  • HIVE-17829 - Fixed ArrayIndexOutOfBoundsException that occurred when using HBASE-backed tables with Avro schema in Hive2

Apache Crunch

There are no notable fixed issues in this release.

Apache Flume

There are no notable fixed issues in this release.

Apache Hadoop

The following issue is fixed in CDH 6.3.1:

  • HADOOP-16018 - DistCp does not reassemble chunks when the value of blocks per chunk is greater than zero.

HDFS

The following issues are fixed in CDH 6.3.1:

  • HDFS-12828 - OIV ReverseXML Processor fails with escaped characters.
  • HDFS-13101 - An fsimage corruption related to snapshots.
  • HDFS-13709 - Report bad block to NameNode when transfer block encounters EIO exception
  • HDFS-14148 - HDFS OIV ReverseXML SnapshotSection parser throws exception when there is more than one snapshottable directory.
  • HDFS-14687 - Standby Namenode does not come out of safemode when EC files are being written.
  • HDFS-14706 - Checksums are not checked if the block meta file size is less than 7 bytes.

MapReduce 2

The following issue is fixed in CDH 6.3.1:

  • MAPREDUCE-7225 - Fix broken current folder expansion during MR job start

YARN

The following issues are fixed in CDH 6.3.1:

  • YARN-9667 - Container-executor.c duplicates messages to stdout
  • YARN-9833 - Race condition when DirectoryCollection.checkDirs() runs during container launch

Apache HBase

The following issues are fixed in CDH 6.3.1:

  • HBASE-19893 - restore_snapshot is broken in master branch when region splits
  • HBASE-20305 - adding options to skip deletes/puts on target when running SyncTable
  • HBASE-22169 - Open region failed cause memory leak
  • HBASE-22539 - WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used
  • HBASE-22617 - Recovered WAL directories not getting cleaned up
  • HBASE-22690 - Deprecate / Remove OfflineMetaRepair in hbase-2+
  • HBASE-22759 - Extended grant and revoke audit events with caller info

Apache Hive

The following issues are fixed in CDH 6.3.1:

  • HIVE-17829 - Fixed ArrayIndexOutOfBoundsException that occurred when using HBASE-backed tables with Avro schema in Hive2

Hue

The following issues are fixed in CDH 6.3.1:

  • HUE-8922 - [frontend] Show dates and times in local format with timezone offset details
  • HUE-8933 - [editor] Results are not properly cleared in multi-statement execution
  • HUE-8950 - [core] Saving newly copied Oozie workflow throws an exception
  • HUE-8979 - [jb] Oozie spark jobs display a NoneType object that is not iterable

Apache Impala

The following issues are fixed in CDH 6.3.1:

  • IMPALA-8549 - Added support for scanning DEFLATE text files.

  • IMPALA-8820 - Fixed an issue where the catalogd process was not found when Impala starts in a cluster.
  • IMPALA-8847 - The event based automatic metadata invalidation can now correctly ignore empty partition lists generated for certain Hive queries.

Apache Kafka

There are no notable fixed issues in this release.

Apache Kite

There are no notable fixed issues in this release.

Apache Kudu

There are no notable fixed issues in this release.

Apache Oozie

The following issues are fixed in CDH 6.3.1:

  • OOZIE-3397 - Improve logging in NotificationXCommand.
  • OOZIE-3542 - Handle better HDFS implementations in ECPolicyDisabler.

Apache Parquet

There are no notable fixed issues in this release.

Apache Pig

There are no notable fixed issues in this release.

Cloudera Search

There are no notable fixed issues in this release.

Apache Sentry

The following issues are fixed in CDH 6.3.1:

  • SENTRY-2276 - Sentry-Kafka integration does not support Kafka's Alter/DescribeConfigs and IdempotentWrite operations
  • SENTRY-2528 - Format exception when fetching a full snapshot

Apache Spark

The following issues are fixed in CDH 6.3.1:

  • SPARK-18364 - [YARN] Expose metrics for YarnShuffleService
  • SPARK-24352 - [CORE][TESTS] De-flake StandaloneDynamicAllocationSuite blacklist test
  • SPARK-24355 - Spark external shuffle server improvement to better handle block fetch requests.
  • SPARK-25139 - [SPARK-18406][CORE][2.4] Avoid NonFatals to kill the Executor in PythonRunner
  • SPARK-25641 - Change the spark.shuffle.server.chunkFetchHandlerThreadsPercent default to 100
  • SPARK-25642 - [YARN] Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service
  • SPARK-25692 - [CORE] Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite as well
  • SPARK-26615 - [CORE] Fixing transport server/client resource leaks in the core unittests
  • SPARK-27021 - [CORE] Cleanup of Netty event loop group for shuffle chunk fetch requests
  • SPARK-28150 - [CORE][FOLLOW-UP] Don't try to log in when impersonating.
  • SPARK-28150 - [CORE] Log in user before getting delegation tokens.
  • SPARK-28261 - [CORE] Fix client reuse test
  • SPARK-28335 - [DSTREAMS][TEST] DirectKafkaStreamSuite wait for Kafka async commit
  • SPARK-28584 - [CORE] Fix thread safety issue in blacklist timer, tests

Apache Sqoop

The following issue is fixed in CDH 6.3.1:

Apache ZooKeeper

There are no notable fixed issues in this release.