ACCUMULO-4389

Description of Problem: Apache Accumulo has a feature called "Replication" which automatically propagates updates to one table to a list of other Accumulo cluster instances. This feature is used for disaster-recovery scenarios allowing data-center level failover. With this replication feature, there are a number of client API methods which support developer interactions with the feature.

The ReplicationOperations#drain(String, Set) method is intended to serve as a blocking call which waits for all of the provided write-ahead log files that need to be replicated to other peers. Sometimes, the method reportedly does not actually wait for a sufficient amount of time.

Associated error message: No direct error message is generated; the primary symptom is when the configured Accumulo replication peers do not have all of the expected data from the source Accumulo cluster.

Workaround: None at this time.

Upstream fix: https://issues.apache.org/jira/browse/ACCUMULO-4389 has been opened to track this issue.

HIVE-13014

Component Affected: Modules using Remote metastore such as Hive CLI and Streaming Ingest API in HCatalog.

Description of Problem: In rare circumstances (due to network issues such as temporary partitions, lost messages, etc.), it is possible for a insert/update/delete operation on a transactional table to report a failure, when it actually committed successfully.

Workaround: Embedded metastore (used by HiveServer2) is unaffected by this behavior.

Description of Problem: SSL shuffle for LLAP is not supported

Workaround: Currently, there is no workaround.

Description of Problem: When Ranger authorization is enabled for Hive, users will be denied permission to create temporary UDFs.

Workaround: To allow users to create temporary UDFs, create a Ranger policy in the following way:

  Resource:     Database=*, udf=*
  Permissions:  Create
  Users/Groups: <as needed>

	Caution
	The above policy will allow the listed users to create both temporary and permanent UDFs. Ensure that you grant this permission only to specific/trusted users.

Hive

Description of Problem: ORC Schema Evolution does not support adding columns to a STRUCT type column unless the STRUCT column is the last column.

You can add column C to the last column last_struct:

CREATE TABLE orc_last_struct (
str STRING,
last_struct STRUCT<A:STRING,B:STRING>
) STORED AS ORC;

ALTER TABLE orc_last_struct REPLACE columns (str STRING, last_struct
STRUCT<A:STRING,B:STRING,C:BIGINT>);

You will be able to read the table.

However, in this table:

CREATE TABLE orc_inner_struct (
str STRING,
inner_struct STRUCT<A:STRING,B:STRING>,
last DATE
) STORED AS ORC;

ALTER TABLE orc_inner_struct REPLACE columns (str STRING, inner_struct
STRUCT<A:STRING,B:STRING,C:BIGINT>, last DATE);

You will not be able to read the table. You will get execution errors like: java.lang.ArrayIndexOutOfBoundsException.

Workaround: The workaround is not to use tables with Schema Evolution in inner STRUCT type columns.

Description of Problem: The search/filter functionality in the Tez View does not work correctly when looking for DAGs submitted by users with user IDs that only contain numbers.

Workaround: Currently, there is no known workaround.

Description of Problem: Unable to log in using Knox SSO even when providing correct credentials. This is because the whitelist is not correctly configured. The login page will not provide an error message to indicate a reason for the failed login.

Associated error message: Found in <log_directory_knox>/gateway.log

Workaround: In knoxsso.xml, modify the value of the knoxsso.redirect.whitelist.regex parameter in the following way. The exact value used should reflect the configuration of your environment:

<param>
           <name>knoxsso.redirect.whitelist.regex</name>
           <value>.*;^/.*$;https?://localhost*$;^http.*$</value>
</param>

Description of Problem: Storm does not support rolling upgrade from a previous HDP version to HDP-2.5.

Solution: If your cluster is managed by Ambari, the rolling upgrade process will ask you to stop Storm topologies, perform the upgrade, and redeploy your topologies.

If your cluster is not managed by Ambari, perform the following manual upgrade steps for Storm before starting the rolling upgrade process:

Next, upgrade the cluster to HDP-2.5.

To finish the Storm upgrade process: start the storm daemons, and then redeploy the topologies.

Summary: Solr bolt does not run in a Kerberos environment.

Associated error message: The following is an example: [ERROR] Request to collection hadoop_logs failed due to (401) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http:[...] Error 401 Authentication required

Workaround: None at this time.

Description of problem: When Kerberos is enabled in the cluster, Kerberos-based user authentication in the Zeppelin UI is not correctly passed to Phoenix/HBase. The user credentials will be unavailable to Phoenix, resulting in standard HBase authentication/authorization schemes working as intended.

Associated error message: Unexpected failed authentication and authorization messages from Zeppelin in talking to Phoenix/HBase.

Workaround: There is no known workaround at this time. This issue will be addressed in a future maintenance release.

Component Affected: ACID

Description of Problem: Small tables estimated to have about 300 million rows that broadcast to a Mapjoin will cause the BloomFilter to overflow. Typically, this is due to bad stats estimation.

Workaround: It is possible to avoid this issue with the following:

set hive.mapjoin.hybridgrace.hashtable=false

However, if this is caused by bad stats estimation and Hybrid grace hash join does not work, the regular mapjoin also will not work.

Component Affected: Create Policy Audit

Description of Problem: When attempting to view the details of a Audit record associated with a deleted ranger repository, the admin UI shows Page Not Found Error Page (401).

Workaround: Currently, there is no workaround for this. This will be addressed in a future release.

Description of Problem: Hive topology fails when the hive-site.xml contains an Atlas hook that tries to register any new tables/partitions created through the hcatalog streaming API.

Currently, the use case that will cause this failure is copying the hive-site.xml from the target cluster to your topology codebase and packaging/creating an uber jar.

Associated Error Message: Since the Atlas hook and its configuration is not getting packaged with Storm Topology jar, the result is NoClassDefFoundError.

Workaround: After copying the hive-site.xml to your topology code, delete the Atlas hook configuration references and than package the jar.

Description of Problem: When installing Spark manually on Debian/Ubuntu, the apt-get install spark command does not install all Spark packages.

Workaround: Use the -t option in your apt-get install command: apt-get install -t HDP spark

Component Affected: Zeppelin UI

Description of Problem: When Zeppelin SSL is enabled, the Zeppelin UI is unavailable through Safari due to a WebSocket network error:

WebSocket network error: OSStatus Error -9807: Invalid certificate chain

Workaround: This occurs due to the use of self signed certificates. Self-signed certificates require OS or Browser specific steps that you must follow prior to use in production. In production, use Certificate Authority signed certificate to prevent this error from occurring.

Description of Problem: On secure clusters that run Zeppelin, configure settings to limit interpreter editing privileges to admin roles.

Workaround: Add the following lines to the [urls] section of the Zeppelin shiro.ini configuration file. For a cluster not managed by Ambari, add the lines to /etc/zeppelin/conf/shiro.ini.

/api/interpreter/** = authc, roles[admin]

/api/configurations/** = authc, roles[admin]

/api/credential/** = authc, roles[admin]

Description of Problem: When upgrading a secure cluster from HDP-2.4.x to HDP-2.5.x, the Kafka service has incorrect security properties.

Workaround: After upgrading from HDP 2.4x to 2.5x (and after removing and then replacing Atlas, as recommended); manually update the user-defined Kerberos descriptor to use the 2.5 stack default values (if acceptable). If default values are unacceptable, copy Kerbeos descriptor from the stack, make required changes, and then replace the descriptor. Then, use Regenerate Keytabs to create the missing configurations, principals, and keytab files.

Description of Problem: LLAP containers may end up getting killed due to insufficient memory available in the system.

Assocaited Error Message: The following messages in the AM log of LLAP YARN Application.

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 194347270144 bytes for committing reserved memory.
# An error report file with more information is saved as:

Workaround: Reduce the YARN NodeManager available memory. This is defind as the Memory allocated for all YARN containers on a node under the YARN Configuration tab.

Desription of Problem: LLAP daemons can be killed by the YARN Memory Monitor

Associated Error Message: The following messages in the AM log of LLAP YARN Application.

is running beyond physical memory limits. Current usage: <USED> of <ALLOCATED> GB physical memory used

Workaround: Lower the LLAP heap size under the Advanced hive-interactive-env section of the Advanced Hive config.

	Note
	You will need to change this value each time any configs are changed under the Hive Interactive section on the main Hive Config page.

Description of Problem: Atlas web UI produces inaccessible alert after adding Atlas service on upgraded cluster

Workaround:

Description of Problem: This issue can cause a Standby NameNode to appear briefly unresponsive at times. The issue is benign and does not affect the Standby NameNode's ability to take over as Active in case of failover. The hangs will not be seen after transitioning to Active.

Workaround: Currently, there is no known workaround.

Description of Problem: HBase clusters running with "region replica" feature might run into a problem where the region flushes are blocked with exceptions similar to

org.apache.hadoop.hbase.regionserver.UnexpectedStateException: Current snapshot id is -1,passed 1469085004304

Workaround:There is no workaround possible, and the cluster should be updated to an HDP version with the patch for HBASE-16270.

Description of Problem: When more than 1000 transactions require a time out, the process for handling the time out may get stuck in an infinite loop.

Workaround: Configure Hive in the following way:

set hive.direct.sql.max.query.length=1;
set hive.direct.sql.max.elements.in.clause=1000;

	Note
	Ensure you configure the second setting value at least 1000 times that of the first setting value.

Description of Problem: Zeppelin (with or without Livy) cannot access data on encrypted (TDE) clusters when the default user settings are in effect.

Workaround:

Description of Problem: This standby NameNode can potentially fail to become active when the active NameNode process is frozen (but not actually crashed).

Workaround: Currently, there is no known workaround.

Description of Problem: Some classes in the org.apache.hadoop.hdfs.client package are in the hadoop-hdfs jar rather than the hadoop-hdfs-client jar. This may break some clients.

Workaround: Currently, there is no known workaround.

Description of Problem: The NameNode thread that reads incoming RPC requests can terminate due to an uncaught RuntimeException. This bug can result in the NameNode becoming unresponsive to RPC requests.

Workaround: Currently, there is no known workaround.

Description of Problem: Pipeline recovery failures due to an expired encryption key are not retried correctly by the HDFS client. This may result in client operations failing with InvalidEncryptionKeyException.

Workaround: Currently, there is no known workaround.

Description of Problem: Disk usage summary incorrectly counts files twice if they have been renamed since being snapshotted.

Workaround: Currently, there is no known workaround.

Description of Problem: The NameNode can incorrectly conclude that some DataNode storage directories are missing and remove them. This can lead to missing blocks. When this problem is hit you may see a number of "removing zombie storage" messages in the NameNode log files.

Workaround: Currently, there is no known workaround.

Description of Problem: Unable to access logs for a running application.

Workaround: Add the below property to the configuration for core-site.xml:

<property>
    <name>hadoop.http.staticuser.user</name>
    <value>yarn</value>
</property>

Description of Problem: By default, the Livy server times out after being idle for 60 minutes.

Associated error message: Subsequent attempts to access Livy generate an error, Exception: Session not found, Livy server would have restarted, or lost session.

Workaround: Set the timeout to a larger value through the property livy.server.session.timeout, and restart the Zeppelin Livy interpreter.

Description of Problem: When one user restarts the %livy interpreter from the Interpreters (admin) page, other users' sessions restart too.

Workaround: Restart the %livy interpreter from within a notebook.

Component Affected: Zeppelin/Livy

Description of Problem: This occurs when running applications through Zeppelin/Livy that requires some 3rd-party libraries. These libraries cannot be installed on all nodes in the cluster but they are installed on their edge nodes. Running in yarn-client mode this all works as the job is submitted on the edge node where the libraries are installed and runs there. In yarn-cluster mode, it fails because the libraries are missing.

Workaround: Set either spark.jars in spark-defaults.conf or livy.spark.jars in livy interpreters conf. Both are globally applicable. The jars need to be present on the livy machine in both cases. Updating livy conf is preferable since it affects only the zeppelin users.

Incremental backups do not capture bulk-loaded data.

Impact of LDAP Channel Binding and LDAP signing changes in Microsoft Active Directory

Microsoft has introduced changes in LDAP Signing and LDAP Channel Binding to increase the security for communications between LDAP clients and Active Directory domain controllers. These optional changes will have an impact on how 3rd party products integrate with Active Directory using the LDAP protocol.

Workaround

Disable LDAP Signing and LDAP Channel Binding features in Microsoft Active Directory if they are enabled

For more information on this issue, see the corresponding Knowledge article: TSB-2021 405: Impact of LDAP Channel Binding and LDAP signing changes in Microsoft Active Directory

CVE-2020-9492 Hadoop filesystem bindings (ie: webhdfs) allows credential stealing

WebHDFS clients might send SPNEGO authorization header to remote URL without proper verification. A maliciously crafted request can trigger services to send server credentials to a webhdfs path (ie: webhdfs://…) for capturing the service principal

For more information on this issue, see the corresponding Knowledge article: TSB-2021 406: CVE-2020-9492 Hadoop filesystem bindings (ie: webhdfs) allows credential stealing

KMS Load Balancing Provider Fails to invalidate Cache on Key Delete

For more information on this issue, see the corresponding Knowledge article: TSB 2020-434: KMS Load Balancing Provider Fails to invalidate Cache on Key Delete

Corruption of HBase data stored with MOB feature

For more information on this issue, see the corresponding Knowledge article: TSB 2021-465: Corruption of HBase data stored with MOB feature on upgrade from CDH 5 and HDP 2

CVE-2021-27905: Apache Solr SSRF vulnerability with the Replication handler

The Apache Solr ReplicationHandler (normally registered at "/replication" under a Solr core) has a "masterUrl" (also "leaderUrl" alias) parameter. The “masterUrl” parameter is used to designate another ReplicationHandler on another Solr core to replicate index data into the local core. To help prevent the CVE-2021-27905 SSRF vulnerability, Solr should check these parameters against a similar configuration used for the "shards" parameter.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-497: CVE-2021-27905: Apache Solr SSRF vulnerability with the Replication handler

HBase MOB data loss

HBase tables with the MOB feature enabled may encounter problems which result in data loss.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-512: HBase MOB data loss