What's New in Apache Kudu

• The scan request timeout was added to Kudu scan token API.
• Two new configuration parameters have been added to the Spark connector to allow better tuning to avoid scan timeouts under high load: socketReadTimeoutMs and scanRequestTimeout
• The kudu tablet unsafe_replace_tablet tool was added to replace a tablet with a new one. Use this tool to recover a table when one of its tablets has permanently lost all replicas. The data in the replaced tablet is lost, so this tool should only be used as a last resort.
• Two new tools allow superusers to retrieve all the values of command line flags from remote Kudu processes: kudu tserver get_flags and kudu master get_flags. The get_flag tools support filtering the returned flags by tag. By default, they will return only flags that were explicitly set.
• A new CLI tool for manual data rebalancing can be used to redistribute table replicas among tablet servers. The rebalancing tool can be run via kudu cluster rebalance sub-command. Using the new tool, it’s possible to rebalance Kudu clusters running CDH 5.12 (Kudu 1.4.0) and newer.

• An optimization was added to improve the performance when scanning tables with large consecutive runs of deleted rows. For example, users may use DELETE all rows in a table or partition before re-adding them, or they may delete all data corresponding to some prefix of the primary key.
• The default timeout for starting tablet copy was increased from 3 to 30 seconds to make automatic tablet re-replication more robust when there is a heavy disk activity on the target server. Also, corresponding tablet server’s flag --tablet_copy_begin_session_timeout_ms is now available as an advanced option.
• The kudu cluster ksck tool has been significantly enhanced.
  • ksck now checks master health and consensus status, displays any unsafe or hidden flags set in the cluster, and produces a summary of the Kudu versions running on the master and tablet servers.
  • ksck now supports JSON output, both in pretty-printed and compact form.
  • The output format is controlled by the -ksck_format flag.
• The output of kudu local_replica dump rowset is now more human-readable.

• Kudu now supports the decimal column type. The decimal type is a numeric data type with fixed scale and precision suitable for financial and other arithmetic calculations where the imprecise representation and rounding behavior of float and double make those types impractical. The decimal type is also useful for integers larger than int64 and cases with fractional values in a primary key. See Decimal Type for more details.
• The strategy Kudu uses for automatically healing tablets which have lost a replica due to server or disk failures has been improved. The new re-replication strategy, or replica management scheme, first adds a replacement tablet replica before evicting the failed one. With the previous replica management scheme, the system first evicts the failed replica and then adds a replacement. The new replica management scheme allows for much faster recovery of tablets in scenarios where one tablet server goes down and then returns back shortly after 5 minutes or so. The new scheme also provides substantially better overall stability on clusters with frequent server failures. See KUDU-1097 for more information.
• The kudu fs update_dirs tool now supports removing directories. Unless the --force flag is specified, Kudu will not allow the removal of a directory across which tablets are configured to spread data. If specified, all tablet replicas configured to use that directory will fail upon starting up and be replicated elsewhere, provided a majority exists elsewhere.
• Users can use the new --fs_metadata_dir to specify the directory in which to place tablet-specific metadata. It is recommended that this be placed on a high-performance drive with high bandwidth and low latency, such as a solid-state drive. If not specified, metadata will be placed in the directory specified by --fs_wal_dir, or in the directory specified by the first entry of --fs_data_dirs if metadata already exists there from an earlier deployment. Kudu does not automatically move existing metadata based on this configuration.
• READ_YOUR_WRITES is a new scan read mode. You can specify READ_YOUR_WRITES when creating a new scanner in C++, Java, and Python clients. If this mode is used, the client performs a read such that it follows all previously known writes and reads from this client. Reads in this mode ensure "read-your-writes" and "read-your-reads" session guarantees while minimizing latency caused by waiting for outstanding write transactions to complete. Note that this is still an experimental feature which may be stabilized in future releases.

• The tablet server web UI scans dashboard (/scans) has been improved with several new features:

  • Shows the most recently completed scans
  • Pseudo-SQL scan descriptor that concisely shows the selected columns and applied predicates
  • More complete, better documented scan statistics
• Kudu daemons now expose a web page /stacks which dumps the current stack trace of every thread running in the server. This information can be helpful when diagnosing performance issues.

• By default, each tablet replica now stripes data blocks across three data directories instead of all data directories. This decreases the likelihood that any given tablet is affected in the event of a single disk failure. No substantial performance impact is expected due to this feature based on performance testing. This change only affects new replicas created after upgrading to Kudu 1.7.
• Kudu servers previously offered the ability to enable a separate metrics log which stores periodic snapshots of all metrics available on a server. This functionality is now available as part of a more general “diagnostics log” which is enabled by default. The diagnostics log includes periodic dumps of server metrics as well as collections of thread stack traces. The default configuration ensures that no more than 640MB of diagnostics logs are retained, and typically the space consumption is significantly less due to compression. The format and contents of this log file are documented in the Apache Kudu Administration.
• The handling of errors in the synchronous Java client has been improved so that when an exception is thrown, the stack trace indicates the correct location where the client function was invoked rather than a call stack of an internal worker thread. The original call stack from the worker thread is available as a suppressed exception.
• The logging of errors in the Java client has been improved to exclude exception stack traces for expected scenarios such as failure to connect to a server in a cluster. Instead, only a single line informational message will be logged in such cases to aid in debugging.
• The Java client now uses a predefined prioritized list of TLS ciphers when establishing an encrypted connection to Kudu servers. This cipher list matches the list of ciphers preferred for server-to-server communication and ensures that the most efficient and secure ciphers are preferred. When the Kudu client is running on Java 8 or newer, this provides a substantial speed-up to read and write performance.
• Reporting for the kudu cluster ksck tool has been updated so tablets and tables with on-going tablet copies are shown as recovering. Additional reporting changes have been made to make various common scenarios, particularly tablet copies, less alarming.
• The performance of inserting rows containing many string or binary columns has been improved, especially in the case of highly concurrent write workloads.
• By default, Spark tasks that scan Kudu are now able to scan non-leader replicas. This allows Spark to more easily schedule kudu-spark tasks local to the data. You can disable this behavior by passing leader_only to the kudu.scanLocality option.
• The number of OS threads used in the steady state and during bursts of activity, such as in Raft leader elections triggered by a node failure, has been drastically reduced and should no longer exceed the value of ulimit -u. It should no longer be necessary to increase the value of ulimit -u or of /proc/sys/kernel/threads-max in order to run a Kudu tablet server in most cases. See KUDU-1913 for more information.
• An issue where sparse column predicates could cause excessive data-block reads has been fixed. Previously in certain scans with sparsely matching predicates on multiple columns, Kudu read and decoded the same data blocks many times. This improvement typically results in a 5-10x performance increase for the affected scans. See KUDU-2231 for more information.
• The efficiency and on-disk size of large updated values has been improved. This improves update-heavy workloads which overwrite large (1KiB+) values. See KUDU-2253 for more information.

Kudu 1.7.0 / CDH 5.15.0 is wire-compatible with previous versions of Kudu:

• Kudu 1.7 clients can connect to servers running Kudu 1.0 or later. If the client uses features that are not available on the target server, an error is returned.
• Rolling upgrade to Kudu 1.7 has not been sufficiently tested. We recommend against rolling upgrades in Kudu. You should shut down all nodes in the cluster, upgrade the software, and then restart the daemons on the new version.
• Kudu 1.0 clients may connect to servers running Kudu 1.7 with the exception of the below-mentioned restrictions regarding secure clusters.

The authentication features introduced in Kudu 1.3 place the following limitations on wire compatibility between Kudu 1.7 and versions earlier than 1.3:

• If a Kudu 1.7 cluster is configured with authentication or encryption is set to required, clients older than Kudu 1.3 are unable to connect.
• If a Kudu 1.7 cluster is configured with authentication and encryption is set to optional or disabled, older clients are still able to connect.

• The Kudu 1.7 Java client library is API- and ABI-compatible with Kudu 1.6. Applications written against Kudu 1.6 compile and run against the Kudu 1.7 client library and vice-versa.
• The Kudu 1.7 C++ client is API- and ABI-forward-compatible with Kudu 1.6. Applications written and compiled against the Kudu 1.6 client library run without modification against the Kudu 1.7 client library. Applications written and compiled against the Kudu 1.7 client library run without modification against the Kudu 1.6 client library.
• The Kudu 1.7 Python client is API-compatible with Kudu 1.6. Applications written against Kudu 1.6 continue to run against the Kudu 1.7 client and vice-versa.
• Kudu 1.7 clients that attempt to create a table with a decimal column on a target server running Kudu 1.6 or earlier receive an error response. Similarly Kudu clients running Kudu 1.6 or earlier result in an error when attempting to access any table containing a decimal column.

• The newly introduced replica management scheme is not compatible with the old scheme, so it is not possible to run pre-1.7 Kudu masters with 1.7 Kudu tablet servers or vice versa. This is a server-side incompatibility only and it does not affect client compatibility. Kudu clients of prior versions are compatible with upgraded Kudu clusters:

  • Kudu masters of 1.7 version do not register Kudu tablet servers of 1.6 and prior versions.
  • Kudu tablet servers of 1.7 version does not work with Kudu masters of 1.6 and prior versions.
• The format of the previously-optional metrics log has changed to include a human-readable timestamp on each line. The path of the log file has also changed with the word “diagnostics” replacing the word “metrics” in the file name. The metrics log has been optimized to only include those metrics which have changed in between successive samples, and to not include entity attributes such as tablet partition information in the log. See KUDU-2297 for more information.

• Cloudera Manager now supports installing Kudu as part of the Express Wizard installation.
• Cloudera Manager has improved monitoring of Kudu node health. Cloudera Manager will indicate whether the health of the service is concerning or bad according to the number of master or tablet server roles that are not running.

• The disk failure tolerance feature is enabled by default on tablet servers, and the capability was extended to handle data directory failures at runtime. In the event of a runtime disk failure, any tablets with data on a failed disk will be shut down and restarted on another tablet server. By default, tablets are striped across all available disks. Note that the first configured data directory and the WAL directory cannot currently tolerate disk failures. This will be further improved in future Kudu releases.

• Kudu servers can now adopt new data directories via the kudu fs update_dirs tool. The new directory will be used by new tablet replicas only. Removing directories is not yet supported (KUDU-2202).

• Kudu servers have two new flags to control the webui TLS/HTTPS settings: --webserver_tls_ciphers and --webserver_tls_min_protocol. These flags allow the advertised TLS ciphers and TLS protocol versions to be configured. Additionally, the webserver now excludes insecure legacy ciphers by default (KUDU-2190).

• Kudu servers can now tolerate short interruptions in NTP clock synchronization. NTP synchronization is still required when any Kudu daemon starts up. If NTP synchronization is not available, diagnostic information is now logged to help pinpoint the issue (KUDU-1578).

• Tablet server startup time has been improved significantly on servers containing large numbers of blocks.

• The log block manager now performs disk data deletion in batches. This optimization can significantly reduce the time taken to delete data on a tablet.

• The usage of sensitive data redaction flag has been changed. By setting the --redact=log flag, redaction will be disabled in the web UI but retained for server logs. Alternatively, --redact=none can be used to disable redaction completely.

• The Spark DataSource integration now can take advantage of scan locality for better scan performance. The scan will take place at the closest replica instead of going to the leader.

• Various optimizations were made to reduce the 99th percentile latency of writes on tablet servers. This can also improve throughput on certain write workloads, particularly on larger clusters.

• You can configure Kudu to ignore the system-wide auth_to_local mappings configured in /etc/krb5.conf by setting the configuration flag --use_system_auth_to_local=false (KUDU-2198).

• The performance of the compaction scheduler has been improved. In previous versions, certain types of time series workloads could cause compaction scheduling to take tens of seconds. These workloads now schedule compactions an order of magnitude more efficiently.

• The compaction scheduler has been improved to avoid running a compaction when the benefit of that compaction is extremely small.

• Tablet servers now consider the health of all replicas of a tablet before deciding to evict one. This can improve stability of the Kudu cluster after experiencing multiple simultaneous daemon failures (KUDU-2048).

• Several performance improvements have been made to the Kudu master, particularly in concurrency of clients opening tables. This should improve performance in highly concurrent workloads.

• The on-disk size metric for a tablet now includes all data and metadata. Previously, it excluded WAL segments and consensus metadata (KUDU-1755).

• Added a verbose mode for the kudu cluster ksck command to enable output of detailed information on the cluster’s metadata, even when no errors are detected.

Kudu 1.6.0 is wire-compatible with previous versions of Kudu:

• Kudu 1.6 clients may connect to servers running Kudu 1.0 or later. If the client uses features that are not available on the target server, an error will be returned.

• Rolling upgrade between Kudu 1.5 and Kudu 1.6 servers may be possible though has not been sufficiently tested. Users are encouraged to shut down all nodes in the cluster, upgrade the software, and then restart the daemons on the new version.

• Kudu 1.0 clients may connect to servers running Kudu 1.6 with the exception of the below-mentioned restrictions regarding secure clusters.

The authentication features introduced in Kudu 1.3 place the following limitations on wire compatibility between Kudu 1.6 and versions earlier than 1.3:

• If a Kudu 1.6 cluster is configured with authentication or encryption set to "required", clients older than Kudu 1.3 will be unable to connect.

• If a Kudu 1.6 cluster is configured with authentication and encryption set to "optional" or "disabled", older clients will still be able to connect.

• The Kudu 1.6 Java client library is API- and ABI-compatible with Kudu 1.5. Applications written against Kudu 1.5 will compile and run against the Kudu 1.6 client library and vice-versa.

• The Kudu 1.6 C++ client is API- and ABI-forward-compatible with Kudu 1.5. Applications written and compiled against the Kudu 1.5 client library will run without modification against the Kudu 1.6 client library. Applications written and compiled against the Kudu 1.6 client library will run without modification against the Kudu 1.5 client library.

• The Kudu 1.6 Python client is API-compatible with Kudu 1.5. Applications written against Kudu 1.5 will continue to run against the Kudu 1.6 client and vice-versa.

• New Command-line Debugging Tool - The kudu fs list command line tool replaces exploratory usages of kudu fs dump and kudu local_replica dump. Output is formatted with default pretty-printing with options to output in CSV and JSON for scripts. Results can easily be filtered to a specific table, tablet, column, rowset, or block using flags. The tool can output many different fields: table, table-id, tablet-id, partition, rowset-id, block-id, block-kind, column, column-id, cfile-data-type, cfile-encoding, cfile-compression, cfile-num-values, cfile-size, cfile-incompatible-features, cfile-compatible-features, cfile-min-key, cfile-max-key, and cfile-delta-stats.

• Integrated into CDH - Starting with Apache Kudu 1.5.0 / CDH 5.13.0, Kudu is fully integrated into CDH and ships as part of the CDH parcel and packages. On a cluster managed by Cloudera Manager, Kudu will be installed as part of CDH and does not need to be installed separately.

• Kudu Web UI Improvements - Kudu server web UIs have a new configuration dashboard (/config) which provides a high level summary of important security configuration values, such as whether RPC authentication is required, or web server HTTPS encryption is enabled. Other types of configuration will be added in future releases.

• New Command-line Tools - The kudu command line tool has two new features:

  • kudu tablet change_config move_replica - This tool moves a tablet replica from one tablet server to another, under the condition that the tablet is healthy. An operator can use this tool to rebalance tablet replicas between tablet servers.
  • kudu local_replica data_size - This tool summarizes the space usage of a tablet, breaking it down by type of file, column, and rowset.

• The log block manager now performs disk synchronization in batches. This optimization can significantly reduce the time taken to copy tablet data from one server to another; in one case tablet copy time is reduced by 35%. It also improves the general performance of flushes and compactions.

• A new feature referred to as "tombstoned voting" is added to the Raft consensus subsystem to allow tablet replicas in the TABLET_DATA_TOMBSTONED state to vote in tablet leader elections. This feature increases Kudu’s stability and availability by improving the likelihood that Kudu will be able to self-heal in more edge-case scenarios, such as when tablet copy operations fail. See KUDU-871 for details.

• The tablet on-disk size metric has been made more accurate. Previously, the metric included only REDO deltas; it now counts all deltas. Additionally, the metric includes the size of bloomfiles, ad hoc indexes, and the tablet superblock. WAL segments and consensus metadata are still not counted. The latter is very small compared to the size of data, but the former may be significant depending on the workload (this will be resolved in a future release).

• The number of threads used by the Kudu tablet server has been further reduced. Previously, each follower tablet replica used a dedicated thread to detect leader tablet replica failures, and each leader replica used one dedicated thread per follower to send Raft heartbeats to that follower. The work performed by these dedicated threads has been reassigned to other threads. Other improvements were made to facilitate better thread sharing by tablets. For the purpose of capacity planning, expect the Kudu tablet server to create one thread for every five "cold" (i.e. those not servicing writes) tablets, and an additional three threads for every "hot" tablet. This will be further improved upon in future Kudu releases.

Kudu 1.5.0 is wire-compatible with previous versions of Kudu:

• Kudu 1.5 clients may connect to servers running Kudu 1.0 or later. If the client uses features that are not available on the target server, an error will be returned.

• Rolling upgrade between Kudu 1.4 and Kudu 1.5 servers is believed to be possible though has not been sufficiently tested. Users are encouraged to shut down all nodes in the cluster, upgrade the software, and then restart the daemons on the new version.

• Kudu 1.0 clients may connect to servers running Kudu 1.5 with the exception of the below-mentioned restrictions regarding secure clusters.

The authentication features introduced in Kudu 1.3 place the following limitations on wire compatibility between Kudu 1.5 and versions earlier than 1.3:

• If a Kudu 1.5 cluster is configured with authentication or encryption set to "required", clients older than Kudu 1.3 will be unable to connect.

• If a Kudu 1.5 cluster is configured with authentication and encryption set to "optional" or "disabled", older clients will still be able to connect.

• The Kudu 1.5 Java client library is API- and ABI-compatible with Kudu 1.4. Applications written against Kudu 1.4 will compile and run against the Kudu 1.5 client library and vice-versa, unless one of the following newly added APIs is used:

• The Kudu 1.5 C++ client is API- and ABI-forward-compatible with Kudu 1.4. Applications written and compiled against the Kudu 1.4 client library will run without modification against the Kudu 1.5 client library. Applications written and compiled against the Kudu 1.5 client library will run without modification against the Kudu 1.4 client library.

• The Kudu 1.5 Python client is API-compatible with Kudu 1.4. Applications written against Kudu 1.4 will continue to run against the Kudu 1.5 client and vice-versa.

For a complete list of known issues for Apache Kudu, see Apache Kudu Known Issues.

Kudu servers, by default, will now only allow unencrypted or unauthenticated connections from trusted subnets, which are private networks (127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16) and local subnets of all local network interfaces. Unencrypted or unauthenticated connections from publicly routable IPs will be rejected, even if encryption and authentication are not configured.

The trusted subnets can be configured using the --trusted_subnets flag, which can be set to IP blocks represented in CIDR notation separated by comma. Set it to '0.0.0.0/0' to allow unauthenticated connections from all remote IP addresses. However, if network access is not otherwise restricted by a firewall, malicious users may be able to gain unauthorized access. This can be mitigated if authentication and encryption are configured to be required.

For a complete list of schema design and usage limitations for Apache Kudu, see Apache Kudu Known Issues.

Kudu 1.3.0 is wire-compatible with previous versions of Kudu:

• Kudu 1.3.0 clients can connect to servers running Kudu 1.0.0. If the client uses features that are not available on the target server, an error will be returned.

• Kudu 1.0.0 clients can connect to servers running Kudu 1.3.0 with the exception of the below-mentioned restrictions regarding secure clusters.

• Rolling upgrade between Kudu 1.2.0 and Kudu 1.3.0 servers is believed to be possible though has not been sufficiently tested. Users are encouraged to shut down all nodes in the cluster, upgrade the software, and then restart the daemons on the new version.

Security - The authentication features introduced in Kudu 1.3.0 place the following limitations on wire compatibility with older versions:

• If a Kudu 1.3.0 cluster is configured with authentication or encryption set to required, older clients will not be able to connect.

• If a Kudu 1.3.0 cluster is configured with authentication and encryption set to optional or disabled, older clients will still be able to connect.

• Due to storage format changes in Kudu 1.3, downgrading from Kudu 1.3 to earlier versions is not supported. After upgrading to Kudu 1.3, attempting to restart with an earlier version will result in an error.

• In order to support running MapReduce and Spark jobs on secure clusters, these frameworks now connect to the cluster at job submission time to retrieve authentication credentials which can later be used by the tasks to be spawned. This means that the process submitting jobs to Kudu clusters must have direct access to that cluster.

• The embedded web servers in Kudu processes now specify the X-Frame-Options: DENY HTTP header which prevents embedding of Kudu web pages in HTML iframe elements.

For a complete list of schema design and usage limitations for Apache Kudu, see Apache Kudu Known Issues.

See also Issues resolved for Kudu 1.2.0 and Git changes between 1.1.x and 1.2.x.

See Issues resolved for Kudu 1.2.0 and Git changes between 1.1.x and 1.2.x.

• KUDU-1508 - Fixed a long-standing issue in which running Kudu on ext4 file systems could cause file system corruption. While this issue has been known to still manifest in certain rare cases, the corruption is harmless and can be repaired as part of a regular fsck. Switching from ext4 to xfs will also solve the problem.
• KUDU-1399 - Implemented an LRU cache for open files, which prevents running out of file descriptors on long-lived Kudu clusters. By default, Kudu will limit its file descriptor usage to half of its configured ulimit.
• Gerrit #5192 - Fixed an issue which caused data corruption and crashes in the case that a table had a non-composite (single-column) primary key, and that column was specified to use DICT_ENCODING or BITSHUFFLE encodings. If a table with an affected schema was written in previous versions of Kudu, the corruption will not be automatically repaired; users are encouraged to re-insert such tables after upgrading to Kudu 1.2 or later.
• Gerrit #5541 - Fixed a bug in the Spark KuduRDD implementation which could cause rows in the result set to be silently skipped in some cases.
• KUDU-1551 - Fixed an issue in which the tablet server would crash on restart in the case that it had previously crashed during the process of allocating a new WAL segment.
• KUDU-1764 - Fixed an issue where Kudu servers would leak approximately 16-32MB of disk space for every 10GB of data written to disk. After upgrading to Kudu 1.2 or later, any disk space leaked in previous versions will be automatically recovered on startup.
• KUDU-1750 - Fixed an issue where the API to drop a range partition would drop any partition with a matching lower _or_ upper bound, rather than any partition with matching lower _and_ upper bound.
• KUDU-1766 - Fixed an issue in the Java client where equality predicates which compared an integer column to its maximum possible value (e.g. Integer.MAX_VALUE) would return incorrect results.
• KUDU-1780 - Fixed the kudu-client Java artifact to properly shade classes in the com.google.thirdparty namespace. The lack of proper shading in prior releases could cause conflicts with certain versions of Google Guava.
• Gerrit #5327 - Fixed shading issues in the kudu-flume-sink Java artifact. The sink now expects that Hadoop dependencies are provided by Flume, and properly shades the Kudu client's dependencies.
• Fixed a few issues using the Python client library from Python 3.

Apache Kudu 1.2.0 introduces the following incompatible changes:

• The replication factor of tables is now limited to a maximum of 7. In addition, it is no longer allowed to create a table with an even replication factor.
• The GROUP_VARINT encoding is now deprecated. Kudu servers have never supported this encoding, and now the client-side constant has been deprecated to match the server's capabilities.
• Client Library Compatibility
  • The Kudu 1.2 Java client is API- and ABI-compatible with Kudu 1.1. Applications written against Kudu 1.1 will compile and run against the Kudu 1.2 client and vice-versa.
  • The Kudu 1.2 cpp client is API- and ABI-forward-compatible with Kudu 1.1. Applications written and compiled against the Kudu 1.1 client will run without modification against the Kudu 1.2 client. Applications written and compiled against the Kudu 1.2 client will run without modification against the Kudu 1.1 client unless they use one of the following new APIs:
    • kudu::DisableSaslInitialization()
    • KuduSession::SetErrorBufferSpace(...)
  • The Kudu 1.2 Python client is API-compatible with Kudu 1.1. Applications written against Kudu 1.1 will continue to run against the Kudu 1.2 client and vice-versa.

• KUDU-1893 - Certain query results incorrectly include rows with null values for predicates.

  Solution - Upgrade to the latest maintenance release.

• Schema and Usage Limitations - For a complete list of schema design and usage limitations for Apache Kudu, see Apache Kudu Known Issues.