ACCUMULO-4389

Description of Problem: Apache Accumulo has a feature called "Replication" which automatically propagates updates to one table to a list of other Accumulo cluster instances. This feature is used for disaster-recovery scenarios allowing data-center level failover. With this replication feature, there are a number of client API methods which support developer interactions with the feature.

The ReplicationOperations#drain(String, Set) method is intended to serve as a blocking call which waits for all of the provided write-ahead log files that need to be replicated to other peers. Sometimes, the method reportedly does not actually wait for a sufficient amount of time.

Associated error message: No direct error message is generated; the primary symptom is when the configured Accumulo replication peers do not have all of the expected data from the source Accumulo cluster.

Workaround: None at this time.

Upstream fix: https://issues.apache.org/jira/browse/ACCUMULO-4389 has been opened to track this issue.

Description of Problem: SSL shuffle for LLAP is not supported

Workaround: Currently, there is no workaround.

Description of Problem: When Ranger authorization is enabled for Hive, users will be denied permission to create temporary UDFs.

Workaround: To allow users to create temporary UDFs, create a Ranger policy in the following way:

  Resource:     Database=*, udf=*
  Permissions:  Create
  Users/Groups: <as needed>

	Caution
	The above policy will allow the listed users to create both temporary and permanent UDFs. Ensure that you grant this permission only to specific/trusted users.

Hive

Description of Problem: ORC Schema Evolution does not support adding columns to a STRUCT type column unless the STRUCT column is the last column.

You can add column C to the last column last_struct:

CREATE TABLE orc_last_struct (
str STRING,
last_struct STRUCT<A:STRING,B:STRING>
) STORED AS ORC;

ALTER TABLE orc_last_struct REPLACE columns (str STRING, last_struct
STRUCT<A:STRING,B:STRING,C:BIGINT>);

You will be able to read the table.

However, in this table:

CREATE TABLE orc_inner_struct (
str STRING,
inner_struct STRUCT<A:STRING,B:STRING>,
last DATE
) STORED AS ORC;

ALTER TABLE orc_inner_struct REPLACE columns (str STRING, inner_struct
STRUCT<A:STRING,B:STRING,C:BIGINT>, last DATE);

You will not be able to read the table. You will get execution errors like: java.lang.ArrayIndexOutOfBoundsException.

Workaround: The workaround is not to use tables with Schema Evolution in inner STRUCT type columns.

Description of Problem: The search/filter functionality in the Tez View does not work correctly when looking for DAGs submitted by users with user IDs that only contain numbers.

Workaround: Currently, there is no known workaround.

Description of Problem: Unable to log in using Knox SSO even when providing correct credentials. This is because the whitelist is not correctly configured. The login page will not provide an error message to indicate a reason for the failed login.

Associated error message: Found in <log_directory_knox>/gateway.log

Workaround: In knoxsso.xml, modify the value of the knoxsso.redirect.whitelist.regex parameter in the following way. The exact value used should reflect the configuration of your environment:

<param>
           <name>knoxsso.redirect.whitelist.regex</name>
           <value>.*;^/.*$;https?://localhost*$;^http.*$</value>
</param>

Summary: Solr bolt does not run in a Kerberos environment.

Associated error message: The following is an example: [ERROR] Request to collection hadoop_logs failed due to (401) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http:[...] Error 401 Authentication required

Workaround: None at this time.

Description of problem: When Kerberos is enabled in the cluster, Kerberos-based user authentication in the Zeppelin UI is not correctly passed to Phoenix/HBase. The user credentials will be unavailable to Phoenix, resulting in standard HBase authentication/authorization schemes working as intended.

Associated error message: Unexpected failed authentication and authorization messages from Zeppelin in talking to Phoenix/HBase.

Workaround: There is no known workaround at this time. This issue will be addressed in a future maintenance release.

Component Affected: ACID

Description of Problem: Small tables estimated to have about 300 million rows that broadcast to a Mapjoin will cause the BloomFilter to overflow. Typically, this is due to bad stats estimation.

Workaround: It is possible to avoid this issue with the following:

set hive.mapjoin.hybridgrace.hashtable=false

However, if this is caused by bad stats estimation and Hybrid grace hash join does not work, the regular mapjoin also will not work.

Component Affected: Create Policy Audit

Description of Problem: When attempting to view the details of a Audit record associated with a deleted ranger repository, the admin UI shows Page Not Found Error Page (401).

Workaround: Currently, there is no workaround for this. This will be addressed in a future release.

Description of Problem: When installing Spark manually on Debian/Ubuntu, the apt-get install spark command does not install all Spark packages.

Workaround: Use the -t option in your apt-get install command: apt-get install -t HDP spark

Component Affected: Zeppelin UI

Description of Problem: When Zeppelin SSL is enabled, the Zeppelin UI is unavailable through Safari due to a WebSocket network error:

WebSocket network error: OSStatus Error -9807: Invalid certificate chain

Workaround: This occurs due to the use of self signed certificates. Self-signed certificates require OS or Browser specific steps that you must follow prior to use in production. In production, use Certificate Authority signed certificate to prevent this error from occurring.

Description of Problem: On secure clusters that run Zeppelin, configure settings to limit interpreter editing privileges to admin roles.

Workaround: Add the following lines to the [urls] section of the Zeppelin shiro.ini configuration file. For a cluster not managed by Ambari, add the lines to /etc/zeppelin/conf/shiro.ini.

/api/interpreter/** = authc, roles[admin]

/api/configurations/** = authc, roles[admin]

/api/credential/** = authc, roles[admin]

Description of Problem: LLAP containers may end up getting killed due to insufficient memory available in the system.

Associated Error Message: The following messages in the AM log of LLAP YARN Application.

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 194347270144 bytes for committing reserved memory.
# An error report file with more information is saved as:

Workaround: Reduce the YARN NodeManager available memory. This is defined as the Memory allocated for all YARN containers on a node under the YARN Configuration tab.

Description of Problem: LLAP daemons can be killed by the YARN Memory Monitor

Associated Error Message: The following messages in the AM log of LLAP YARN Application.

is running beyond physical memory limits. Current usage: <USED> of <ALLOCATED> GB physical memory used

Workaround: Lower the LLAP heap size under the Advanced hive-interactive-env section of the Advanced Hive config.

	Note
	You will need to change this value each time any configs are changed under the Hive Interactive section on the main Hive Config page.

Description of Problem: HBase clusters running with "region replica" feature might run into a problem where the region flushes are blocked with exceptions similar to

org.apache.hadoop.hbase.regionserver.UnexpectedStateException: Current snapshot id is -1,passed 1469085004304

Workaround:There is no workaround possible, and the cluster should be updated to an HDP version with the patch for HBASE-16270.

Description of Problem: When more than 1000 transactions require a time out, the process for handling the time out may get stuck in an infinite loop.

Workaround: Configure Hive in the following way:

set hive.direct.sql.max.query.length=1;
set hive.direct.sql.max.elements.in.clause=1000;

	Note
	Ensure you configure the second setting value at least 1000 times that of the first setting value.

Description of Problem: Zeppelin (with or without Livy) cannot access data on encrypted (TDE) clusters when the default user settings are in effect.

Workaround:

Description of Problem: This standby NameNode can potentially fail to become active when the active NameNode process is frozen (but not actually crashed).

Workaround: Currently, there is no known workaround.

Description of Problem: Some classes in the org.apache.hadoop.hdfs.client package are in the hadoop-hdfs jar rather than the hadoop-hdfs-client jar. This may break some clients.

Workaround: Currently, there is no known workaround.

Description of Problem: Disk usage summary incorrectly counts files twice if they have been renamed since being snapshotted.

Workaround: Currently, there is no known workaround.

Description of Problem: The NameNode can incorrectly conclude that some DataNode storage directories are missing and remove them. This can lead to missing blocks. When this problem is hit you may see a number of "removing zombie storage" messages in the NameNode log files.

Workaround: Currently, there is no known workaround.

Description of Problem: By default, the Livy server times out after being idle for 60 minutes.

Associated error message: Subsequent attempts to access Livy generate an error, Exception: Session not found, Livy server would have restarted, or lost session.

Workaround: Set the timeout to a larger value through the property livy.server.session.timeout, and restart the Zeppelin Livy interpreter.

Description of Problem: When one user restarts the %livy interpreter from the Interpreters (admin) page, other users' sessions restart too.

Workaround: Restart the %livy interpreter from within a notebook.

Description of Problem: CBO triggers one of the rewriting rules for the plan (HivePreFiltering) indefinitely under some circumstances. In particular, it cannot recognize that the rule was triggered on current operator and the additional filters were already created.

Workaround: Currently, there is no known workaround.

Description of Problem: Apache Atlas uses reflection to introspect the StormTopology fields and gather metadata. This introspection leads to RuntimeException or NPE when a Storm topology is submitted, occuring when the reflection APIs recurse into the Jackson library.

Workaround: Currently, there is no known workaround.

Description of Problem: Phoenix indextool cannot be used on lower case data table names.

Workaround: Use upper case data table names.

Description of Problem: Hive topology fails when the hive-site.xml contains an Atlas hook that tries to register any new tables/partitions created through the hcatalog streaming API.

Currently, the use case that will cause this failure is copying the hive-site.xml from the target cluster to your topology codebase and packaging/creating an uber jar.

Associated Error Message: Since the Atlas hook and its configuration is not getting packaged with Storm Topology jar, the result is NoClassDefFoundError.

Workaround: After copying the hive-site.xml to your topology code, delete the Atlas hook configuration references and than package the jar.

Description of Problem: Multi-threaded access to CredentialProviderFactory is not thread-safe because java.util.ServiceLoader is not thread-safe (as noted in its Java doc). More and more applications start to access cloud filesystem directly using URI s3a:// or adl://. Configuration.getPassword is called during construction of each fs instance to get credentials, thus more likely in multiple threads. Configuration.getPassword uses service loader to load credential providers. Somewhere along the stack the call needs to be serialized.

Workaround: Currently, there is no known workaround.

Description of Problem: Since the package of ActiveDirectoryGroupRealm and LdapGroupRealm is changed, with Ambari 2.4.x after upgrade to this version will throw an error.

Associated error message: Class not found exception org.apache.zeppelin.server.LdapGroupRealm or org.apache.zeppelin.server.ActiveDirectoryGroupRealm

Workaround: In the shiro Zeppelin's config in Ambari change:

org.apache.zeppelin.server.LdapGroupRealm

to:

org.apache.zeppelin.realm.LdapRealm

and change:

org.apache.zeppelin.server.ActiveDirectoryGroupRealm

to:

org.apache.zeppelin.realm.ActiveDirectoryGroupRealm

Description of Problem: ZooKeeper fails to remove references to already deleted nodes in ACL tree. This can result in the creation of large snapshot files if a client program creates and deletes a lot of nodes in ZooKeeper.

Workaround: Currently, there is no known workaround.

Description of Problem: When phoenix.schema.isNamespaceMappingEnabled is set to true in hbase-site.xml, Phoenix will create/move the Phoenix SYSTEM tables into an HBase namespace called SYSTEM. The logic to check if this creation/move needs to happen requires the global ADMIN permission for the current user. Even if the tables already exist in the proper namespace, privileges lacking the global ADMIN permission cannot establish a connection to Phoenix as the implementation always performs this check and fails the client connection process.

Workaround: The only workaround is to provide the ADMIN permission to the users trying to connect to Phoenix, which introduces obvious security concerns. To avoid this, the recommended path is to upgrade to a version of Phoenix which addresses the issue. You can also avoid this issue by disabling the namespace mapping feature.

Description of Problem: Falcon is not starting after HDP upgrade (2.5.0|2.5.3 to 2.5.5) using Ambari.

Workaround: See the workaround here for more information.

Description of Problem: For a partitioned table, the class StatsNoJobTask is supposed to launch threads for all partitions and compute their stats. However, it could terminate inappropriately early due to insufficient timeout limits. The hardcoded limit is set to 100 seconds.

Workaround: Currently, there is no known workaround.

Component Affected: Zeppelin/Livy

Description of Problem: This occurs when running applications through Zeppelin/Livy that requires some 3rd-party libraries. These libraries cannot be installed on all nodes in the cluster but they are installed on their edge nodes. Running in yarn-client mode this all works as the job is submitted on the edge node where the libraries are installed and runs there. In yarn-cluster mode, it fails because the libraries are missing.

Workaround: Set either spark.jars in spark-defaults.conf or livy.spark.jars in livy interpreters conf. Both are globally applicable. The jars need to be present on the livy machine in both cases. Updating livy conf is preferable since it affects only the zeppelin users.

Impact of LDAP Channel Binding and LDAP signing changes in Microsoft Active Directory

Microsoft has introduced changes in LDAP Signing and LDAP Channel Binding to increase the security for communications between LDAP clients and Active Directory domain controllers. These optional changes will have an impact on how 3rd party products integrate with Active Directory using the LDAP protocol.

Workaround

Disable LDAP Signing and LDAP Channel Binding features in Microsoft Active Directory if they are enabled

For more information on this issue, see the corresponding Knowledge article: TSB-2021 405: Impact of LDAP Channel Binding and LDAP signing changes in Microsoft Active Directory

CVE-2020-9492 Hadoop filesystem bindings (ie: webhdfs) allows credential stealing

WebHDFS clients might send SPNEGO authorization header to remote URL without proper verification. A maliciously crafted request can trigger services to send server credentials to a webhdfs path (ie: webhdfs://…) for capturing the service principal

For more information on this issue, see the corresponding Knowledge article: TSB-2021 406: CVE-2020-9492 Hadoop filesystem bindings (ie: webhdfs) allows credential stealing

KMS Load Balancing Provider Fails to invalidate Cache on Key Delete

For more information on this issue, see the corresponding Knowledge article: TSB 2020-434: KMS Load Balancing Provider Fails to invalidate Cache on Key Delete

Corruption of HBase data stored with MOB feature

For more information on this issue, see the corresponding Knowledge article: TSB 2021-465: Corruption of HBase data stored with MOB feature on upgrade from CDH 5 and HDP 2

CVE-2021-27905: Apache Solr SSRF vulnerability with the Replication handler

The Apache Solr ReplicationHandler (normally registered at "/replication" under a Solr core) has a "masterUrl" (also "leaderUrl" alias) parameter. The “masterUrl” parameter is used to designate another ReplicationHandler on another Solr core to replicate index data into the local core. To help prevent the CVE-2021-27905 SSRF vulnerability, Solr should check these parameters against a similar configuration used for the "shards" parameter.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-497: CVE-2021-27905: Apache Solr SSRF vulnerability with the Replication handler

HBase MOB data loss

HBase tables with the MOB feature enabled may encounter problems which result in data loss.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-512: HBase MOB data loss