Apache Kudu Known Issues

In a multi master Kudu environment, if a master is restarted or goes offline for a few minutes, it can occasionally have trouble joining the cluster on startup. For example, if this happens in case of three kudu masters, and one of the other two masters is stopped or dies during this time, then the overall Kudu cluster is down because the majority of the masters are not running.

This issue is resolved by the KUDU-2748 upstream JIRA.

Products affected: Apache Kudu

For the latest update on this issue see the corresponding Knowledge article:TSB 2020-442: Kudu Masters unable to join back after a restart

Due to KUDU-2463, upon restarting Kudu, inconsistent rows may be returned from tables that have not recently been written to, resulting in any of the following:

• multiple rows for the same key being returned
• deleted data being returned
• inconsistent results consistently being returned for the same query

If this happens, you have two options to resolve the conflicts: write to the affected Kudu partitions by:

• re-deleting the known and deleted data
• upserting the most up-to-date version of affected rows.

Products affected: Apache Kudu

For the latest update on this issue see the corresponding Knowledge article:TSB 2019-353: Inconsistent rows returned from queries in Kudu

Unable to advance iterator: Corruption: checksum error on CFile block

Workaround: Remove the corrupted replica from the tablet's Raft configuration. See Kudu Troubleshooting Guide for the detailed steps.

Affected Versions: CDH 6.0.x and lower

Apache Issue: KUDU-2469

Affected Versions: CDH 5.16.0, CDH 5.16.1

Workaround: Specify an additional run-time flag --move_single_replicas=disabled when running the kudu cluster rebalance command. Specifying that flag explicitly disables balancing of non-replicated tables.

NOTE: As a good practice, non-replicated tables should not be used in production Kudu clusters.

Addressed in releases: CDH 5.16.2

Due to KUDU-2233, tablet servers might crash on restart with a fatal log message like the following:

F1201 14:55:37.052140 10508 compaction.cc:756] Check failed: pv_delete_redo != nullptr
 *
 **
 *** Check failure stack trace: ***
 Wrote minidump to /var/log/kudu/minidumps/kudu-tserver/215cde39-7795-0885-0b51038d-771d875e.dmp
 *** Aborted at 1512161737 (unix time) try "date -d @1512161737" if you are using GNU date ***
 PC: @ 0x3ec3632625 (unknown)
 *** SIGABRT (@0x3b98eec0000028e3) received by PID 10467 (TID 0x7f8b02c58700) from PID 10467; stack trace: ***
@ 0x3ec3a0f7e0 (unknown)
@ 0x3ec3632625 (unknown)
@ 0x3ec3633e05 (unknown)
0x1b53f59 (unknown)
@ 0x8b9f6d google::LogMessage::Fail()
@ 0x8bbe2d google::LogMessage::SendToLog()
@ 0x8b9aa9 google::LogMessage::Flush()
@ 0x8bc8cf google::LogMessageFatal::~LogMessageFatal()
@ 0x9db0fe kudu::tablet::FlushCompactionInput()
@ 0x9a056a kudu::tablet::Tablet::DoMergeCompactionOrFlush()
@ 0x9a372d kudu::tablet::Tablet::Compact()
@ 0x9bd8d1 kudu::tablet::CompactRowSetsOp::Perform()
@ 0x1b4145f kudu::MaintenanceManager::LaunchOp()
@ 0x1b8da06 kudu::ThreadPool::DispatchThread()
@ 0x1b888ea kudu::Thread::SuperviseThread()
@ 0x3ec3a07aa1 (unknown)
@ 0x3ec36e893d (unknown)
@ 0x0 (unknown)}}

After this crash occurs, the affected tablet server requires a patch in order to start again.

This crash is much more likely to occur on workloads involving deleting and re-inserting the same rows repeatedly.

sudo -u kudu kudu cluster ksck -checksum_scan <master addresses>

Products affected: Kudu

Users affected: All users running Kudu, but the risk is much greater for workloads that involve deleting and re-inserting rows repeatedly.

Severity: High

Impact: The affected tablet server will not be able to start until a patch is applied. Data loss or corruption is possible.

Immediate action required: Upgrade to a version containing a fix.

If this is not possible, raising the minimum number of log segments retained to 2 will make the problem very unlikely to occur. To do this, go to the configuration page for Kudu and add --log_min_segments_to_retain=2 to the Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile option.

If the issue still occurs, it’s necessary to patch the affected tablet server or upgrade to a version containing a fix.

Once upgraded to a version containing a fix, the --log_min_segments_to_retain setting should be removed.

Addressed in release: CDH5.13.3, CDH5.14.2

Without the fix from KUDU-2173, the Kudu C++ client’s partition pruner may incorrectly prune a partition when evaluating certain queries containing predicates on both the range partition columns and other columns that are part of the primary key. This may cause queries to fail to return some matching rows.

Specifically, the pruner mistakenly treats a range partition which is a proper prefix of a primary key as an exclusive bound, when in fact it is an inclusive bound if the remaining primary key column constraints are greater than the minimum value.

CREATE TABLE t ( 
a INT NOT NULL,
b INT NOT NULL,
PRIMARY KEY (a,b)
) PARTITION BY RANGE(a) PARTITIONS (
...
)

RANGE (a) PARTITION VALUES >= 10

SELECT * FROM t WHERE a < 11 AND b < 11

The partition pruner would exclude the above partition, even though it contains rows like (10, 1). Instead, the partition pruner treats >= 10 as > 10. This wouldn't be a problem if the range partition was on column (b) instead, since that is not a prefix of the primary key. The same applies for (a, b), since that is the whole primary key and not a prefix.

Products affected: Kudu

Releases affected:

• CDH 5.10.0, 5.10.1, 5.10.2
• CDH 5.11.0, 5.11.1, 5.11.2
• CDH 5.12.0, 5.12.1
• CDH 5.13.0

Users affected: Direct Kudu users via its C++ or Python client. Indirect users via Impala.

Severity (Low/Medium/High): High

Impact: Certain queries won’t return all the available matching rows and no error message will be displayed.

A security-related issue can cause Impala queries to start failing on busy clusters in the following scenario:

• The cluster runs with the --rpc_authentication set as optional or required. The default is optional. Secure clusters use required.
• The cluster is using multiple masters.
• Impala queries happen frequently enough that the leader master connection to some impalad isn't idle-closed (more than 1 query per 65 seconds).
• The connection stays alive for longer than the authentication token timeout (1 week by default).
• A master leadership change occurs after the authentication token expiration.

I0904 13:53:08.748968 95857 client-internal.cc:283] Unable to determine the new leader Master: Not authorized: Client connection negotiation failed: client connection to 10.164.44.13:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired
I0904 13:53:10.389009 95861 status.cc:125] Unable to open Kudu table: Timed out: GetTableSchema timed out after deadline expired
 @ 0x95b1e9 impala::Status::Status()
 @ 0xff22d4 impala::KuduScanNodeBase::Open()
 @ 0xff101e impala::KuduScanNode::Open()
 @ 0xb73ced impala::FragmentInstanceState::Open()
 @ 0xb7532b impala::FragmentInstanceState::Exec()
 @ 0xb64ae8 impala::QueryState::ExecFInstance()
 @ 0xd15193 impala::Thread::SuperviseThread()
 @ 0xd158d4 boost::detail::thread_data<>::run()
 @ 0x129188a (unknown)
 @ 0x7f717ceade25 start_thread
 @ 0x7f717cbdb34d __clone

Unable to open Kudu table: Timed out: GetTableSchema timed out after deadline expired

Affected Versions: From CDH 5.11 through CDH 6.0

Fixed Versions: CDH 5.15.2, CDH 5.16.1

Apache Issue: KUDU-2580

If the Kudu master is configured with the -log_force_fsync_all option, tablet servers and clients will experience frequent timeouts, and the cluster may become unusable.

If a tablet server has a very large number of tablets, it may take several minutes to start up. It is recommended to limit the number of tablets per server to 1000 or fewer. The maximum allowed number of tablets is 2000 per server. Consider this limitation when pre-splitting your tables. If you notice slow start-up times, you can monitor the number of tablets per server in the web UI.

• Kerberos authentication does not function correctly on hosts which contain capital letters in their hostname.

• Kerberos authentication does not function correctly if rdns = false is configured in krb5.conf.

Unlike regular scans, fault tolerant scans will allocate all required memory when the scan begins rather than as it progresses. This can be significant for big tablets. Moreover, this memory usage isn't counted towards the tablet server's overall memory limit, raising the likelihood of the tablet server being out-of-memory killed by the kernel.

Affected Versions: CDH 6.2 / Kudu 1.9 and lower

Apache Issue: KUDU-2466

Descriptions in the Cloudera Manager Admin Console for TLS/SSL settings are confusing, and will be replaced in a future release. The correct information about the settings is in the Usage Notes column: