HDP-2.4.0 Release Notes
Also available as:
PDF

Known Issues

Hortonworks Bug ID

Apache JIRA

Component

Summary

BUG-23260HIVE-11421, HIVE-11981Hive

Insert after alter table on Acid table causes ArrayIndexOutOfBoundsException

Issue: Schema evolution is not supported for Acid tables. Alter Table commands on Acid table may make the table unreadable. There is not w/a after that.

BUG-55196HIVE-12937HiveDbNotificationListener unable to clean up old notification events
BUG-30022HIVE-7693HiveInvalid column ref error in order by when using column alias in select clause and using having
BUG-30556 HiveKilled job is not display correctly by ATS hook failure
BUG-38054RANGER-577Ranger

Ranger performs authorization check, even if Hive authorization is disabled

When Ranger is Enabled for authorization, it would do the authorization check even if the hive authorization is disabled (via hive.security.authorization.enabled = false).

BUG-38928HIVE-11599Hive

Alter table transactional=false on ACID table misleading user. create transactional table would be better

Issue: Once an ACID table is created (with transactional=true) and data is written to it, changing the property to transactional=false may cause wrong data to be returned by queries over this table.

BUG-39663 HiveHive compactor fails when 'hive' user does not have x permissions on the table directory
BUG-39988HIVE-11110HiveCBO : Default partition filter is from MetaStore query causing TPC-DS to regress by 3x
BUG-43128HIVE-12724HiveQuery wrong results: non-ACID to ACID, major compaction does not compact all pre-ACID data into base dir
BUG-43129 HiveACID update/delete query fails with ArrayIndexOutOfBoundsException when there is null struct
BUG-45846HIVE-11605HiveIncorrect results with bucket map join in tez
BUG-45857HIVE-5623HiveORC accessing array column that's empty will fail with java out of bound exception
BUG-46092HIVE-7193HiveWe need to filter users who can connect on our HiveServer2 to those in a particular LDAP group (with nested groups)
BUG-46128 StormKafka spout showing errors in the log streaming workflow
BUG-46771HIVE-11716HiveReading ACID table from non-acid session should raise an error
BUG-47626 HBase, Hive, PhoenixCreating Hive table on HBase fails in HDP 2.3.x with Phoenix Dependencies
BUG-48217 YARNAfter restarting Yarn services, applications stays in ACCEPTED state
BUG-48416 FalconUnable to save a feed via Falcon UI
BUG-48979 YARN

Issue: Jobs fail with javax.security.sasl.SaslException

Cause: OpenJDK 7u91 is stricter about validating ticket caches; if there is any ticket in the cache without a default realm, SASL will fail to work.

Workaround: To avoid this problem, the /etc/krb5.conf file must contain a domain_realm section.

BUG-48990HIVE-12837HiveMicroStrategy query 117 fails with out of memory error
BUG-49726HIVE-10632HiveHive compactor processing partitions of table that does not exist
BUG-49784HDFS-9445HDFSDatanode may deadlock while handling a bad volume
BUG-49949 HiveQuery with duplicate columns in GROUP BY clause fails
BUG-49955PHOENIX-2531Phoenix

Phoenix thin-client jar has incorrect META-INF/services/java.sql.Driver file

Issue: The Phoenix Thin Client Driver, org.apache.phoenix.queryserver.client.Driver, is not automatically registered in the JDBC DriverManager.

Workaround:

  1. Add the following to your Java application before trying to obtain the driver:

    Properties props = new Properties();
    String URL = "jdbc:phoenix:thin:URL=http://localhost:8765";
    try {
      Class.forName("org.apache.phoenix.queryserver.client.Driver");
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
    DriverManager.getConnection(URL, props);
  2. Manually register the driver.

BUG-50073HDFS-9557HDFSReduce object allocation in PB conversion
BUG-50089HDFS-9572HDFSPrevent DataNode log spam if a client connects on the data transfer port but sends no data.
BUG-50302HBASE-15039HBase

HMaster and RegionServers should try to refresh token keys from zk when facing InvalidToken

Issue: Many customers use importTsv to load data into HBase, but the cluster network may not always be good. Sometimes they face many map tasks inside importTsv throwing InvalidToken exception.

Workaround: Restart the region server.

BUG-50479HADOOP-11252Hadoop CommonRPC client does not time out by default
BUG-50531 Kafka

Kafka file system support

Issue: Encrypted file systems such as SafenetFS are not supported for Kafka. Index file corruption can occur.

Workaround: N/A

For more information, see: Install Kafka.

BUG-50576SQOOP-2779SqoopSqoop metastore doesn't seem to recognize --schema option
BUG-50599HDFS-9600HDFSDo not check replication if the block is under construction
BUG-50652HDFS-9574HDFSReduce client failures during datanode restart
BUG-51239 RangerNameNode shutdown hangs due to Ranger HDFS plugin
BUG-51292 Kafka, Spark

Spark Streaming Kafka direct API unit test failure in HDP release

Workaround: None.

BUG-51293HDFS-9655HDFSNN should start JVM pause monitor before loading fsimage
BUG-51423HDFS-9313HDFSPossible NullPointerException in BlockManager if no excess replica can be chosen
BUG-51424HDFS-9314HDFSImprove BlockPlacementPolicyDefault's picking of excess replicas
BUG-51426HDFS-9574HDFSReduce client failures during datanode restart
BUG-51427HDFS-8647, HDFS-9600HDFSDo not check replication if the block is under construction
BUG-51428HADOOP-11252Hadoop CommonRPC client does not time out by default
BUG-51433HDFS-9572HDFSPrevent DataNode log spam if a client connects on the data transfer port but sends no data.
BUG-51436HDFS-9557HDFSReduce object allocation in PB conversion
BUG-51439HDFS-9445HDFSDatanode may deadlock while handling a bad volume
BUG-51442HDFS-9655HDFSNN should start JVM pause monitor before loading fsimage
BUG-51463HDFS-8898HDFSCreate API and command-line argument to get quota and quota usage without detailed content summary
BUG-51657 Ranger

Cyclic symlink for ranger usersync configuration after upgrade

Issue: When manually upgrading from 2.3.4 to 2.4.0, a cyclic symlink gets created for usersync conf.

Cause: This issue arises due to wrongly created symlinks which make the conf links cyclic and invalid.

Workaround:

  • For Ranger-Admin

    1. Delete the conf links from below paths:

      /usr/hdp/current/ranger-admin/
      /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/classes/
      /etc/ranger/admin/
    2. Run the set_globals.sh script present under /usr/hdp/current/ranger-admin/ to create appropriate symlinks.

    3. Re-Install the application from Ambari.

  • For Ranger-Usersync:

    1. Delete the conf links from below paths:

      /usr/hdp/current/ranger-usersync/
      /etc/ranger/usersync/
    2. Run the set_globals.sh script present under /usr/hdp/current/ranger-usersync/ to create appropriate symlinks.

    3. Re-Install the application from Ambari.

BUG-51712 Ranger

After upgrade getting java.lang.NoClassDefFoundError

Issue: java.lang.NoClassDefFoundError error found in ranger usersync log after upgrade.

Workaround: Copy the below jar files into ranger usersync lib folder and restart the service.

cp /usr/hdp/current/hadoop-client/lib/commons-httpclient-3.1.jar /usr/hdp/current/ranger-usersync/lib/
cp /usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar /usr/hdp/current/ranger-usersync/lib/
BUG-67482RANGER-1136Ranger

Description of Problem: Ranger audit to HDFS fails with TGT errors

Workaround: Currently, there is no known workaround for this issue.

Technical Service BulletinApache JIRAApache ComponentSummary
TSB-405N/AN/A

Impact of LDAP Channel Binding and LDAP signing changes in Microsoft Active Directory

Microsoft has introduced changes in LDAP Signing and LDAP Channel Binding to increase the security for communications between LDAP clients and Active Directory domain controllers. These optional changes will have an impact on how 3rd party products integrate with Active Directory using the LDAP protocol.

Workaround

Disable LDAP Signing and LDAP Channel Binding features in Microsoft Active Directory if they are enabled

For more information on this issue, see the corresponding Knowledge article: TSB-2021 405: Impact of LDAP Channel Binding and LDAP signing changes in Microsoft Active Directory

TSB-406N/AHDFS

CVE-2020-9492 Hadoop filesystem bindings (ie: webhdfs) allows credential stealing

WebHDFS clients might send SPNEGO authorization header to remote URL without proper verification. A maliciously crafted request can trigger services to send server credentials to a webhdfs path (ie: webhdfs://…) for capturing the service principal

For more information on this issue, see the corresponding Knowledge article: TSB-2021 406: CVE-2020-9492 Hadoop filesystem bindings (ie: webhdfs) allows credential stealing

TSB-434HADOOP-17208, HADOOP-17304Hadoop

KMS Load Balancing Provider Fails to invalidate Cache on Key Delete

For more information on this issue, see the corresponding Knowledge article: TSB 2020-434: KMS Load Balancing Provider Fails to invalidate Cache on Key Delete

TSB-465N/AHBase

Corruption of HBase data stored with MOB feature

For more information on this issue, see the corresponding Knowledge article: TSB 2021-465: Corruption of HBase data stored with MOB feature on upgrade from CDH 5 and HDP 2

TSB-497N/ASolr

CVE-2021-27905: Apache Solr SSRF vulnerability with the Replication handler

The Apache Solr ReplicationHandler (normally registered at "/replication" under a Solr core) has a "masterUrl" (also "leaderUrl" alias) parameter. The “masterUrl” parameter is used to designate another ReplicationHandler on another Solr core to replicate index data into the local core. To help prevent the CVE-2021-27905 SSRF vulnerability, Solr should check these parameters against a similar configuration used for the "shards" parameter.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-497: CVE-2021-27905: Apache Solr SSRF vulnerability with the Replication handler

TSB-512N/AHBase

HBase MOB data loss

HBase tables with the MOB feature enabled may encounter problems which result in data loss.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-512: HBase MOB data loss