What's New in Apache Kudu

Schema Design and Usage Limitations

The Apache Kudu Known Issues and Usage Limitations topics describes known issues and limitations with respect to schema design, integration with Apache Impala, and security in Kudu, as of the current release.

Kudu 1.7.0 / CDH 5.16.1 Release Notes

The release of Kudu in CDH 5.16.1 is based on Apache Kudu 1.7 with a few additional changes highlighted in these release notes.

Wire protocol compatibility, client library compatibility, incompatible changes and the issues stay the same as described in Kudu 1.7.0 / CDH 5.15.0 Release Notes.

New Features in Kudu 1.7.0 / CDH 5.16.1

  • The scan request timeout was added to Kudu scan token API.
  • Two new configuration parameters have been added to the Spark connector to allow better tuning to avoid scan timeouts under high load: socketReadTimeoutMs and scanRequestTimeout
  • The kudu tablet unsafe_replace_tablet tool was added to replace a tablet with a new one. Use this tool to recover a table when one of its tablets has permanently lost all replicas. The data in the replaced tablet is lost, so this tool should only be used as a last resort.
  • Two new tools allow superusers to retrieve all the values of command line flags from remote Kudu processes: kudu tserver get_flags and kudu master get_flags. The get_flag tools support filtering the returned flags by tag. By default, they will return only flags that were explicitly set.
  • A new CLI tool for manual data rebalancing can be used to redistribute table replicas among tablet servers. The rebalancing tool can be run via kudu cluster rebalance sub-command. Using the new tool, it’s possible to rebalance Kudu clusters running CDH 5.12 (Kudu 1.4.0) and newer.

Optimizations and Improvements in Kudu 1.7.0 / CDH 5.16.1

  • An optimization was added to improve the performance when scanning tables with large consecutive runs of deleted rows. For example, users may use DELETE all rows in a table or partition before re-adding them, or they may delete all data corresponding to some prefix of the primary key.
  • The default timeout for starting tablet copy was increased from 3 to 30 seconds to make automatic tablet re-replication more robust when there is a heavy disk activity on the target server. Also, corresponding tablet server’s flag --tablet_copy_begin_session_timeout_ms is now available as an advanced option.
  • The kudu cluster ksck tool has been significantly enhanced.
    • ksck now checks master health and consensus status, displays any unsafe or hidden flags set in the cluster, and produces a summary of the Kudu versions running on the master and tablet servers.
    • ksck now supports JSON output, both in pretty-printed and compact form.
    • The output format is controlled by the -ksck_format flag.
  • The output of kudu local_replica dump rowset is now more human-readable.

Kudu 1.7.0 / CDH 5.15.0 Release Notes

New Features in Kudu 1.7.0 / CDH 5.15.0

  • Kudu now supports the decimal column type. The decimal type is a numeric data type with fixed scale and precision suitable for financial and other arithmetic calculations where the imprecise representation and rounding behavior of float and double make those types impractical. The decimal type is also useful for integers larger than int64 and cases with fractional values in a primary key. See Decimal Type for more details.
  • The strategy Kudu uses for automatically healing tablets which have lost a replica due to server or disk failures has been improved. The new re-replication strategy, or replica management scheme, first adds a replacement tablet replica before evicting the failed one. With the previous replica management scheme, the system first evicts the failed replica and then adds a replacement. The new replica management scheme allows for much faster recovery of tablets in scenarios where one tablet server goes down and then returns back shortly after 5 minutes or so. The new scheme also provides substantially better overall stability on clusters with frequent server failures. See KUDU-1097 for more information.
  • The kudu fs update_dirs tool now supports removing directories. Unless the --force flag is specified, Kudu will not allow the removal of a directory across which tablets are configured to spread data. If specified, all tablet replicas configured to use that directory will fail upon starting up and be replicated elsewhere, provided a majority exists elsewhere.
  • Users can use the new --fs_metadata_dir to specify the directory in which to place tablet-specific metadata. It is recommended that this be placed on a high-performance drive with high bandwidth and low latency, such as a solid-state drive. If not specified, metadata will be placed in the directory specified by --fs_wal_dir, or in the directory specified by the first entry of --fs_data_dirs if metadata already exists there from an earlier deployment. Kudu does not automatically move existing metadata based on this configuration.
  • READ_YOUR_WRITES is a new scan read mode. You can specify READ_YOUR_WRITES when creating a new scanner in C++, Java, and Python clients. If this mode is used, the client performs a read such that it follows all previously known writes and reads from this client. Reads in this mode ensure "read-your-writes" and "read-your-reads" session guarantees while minimizing latency caused by waiting for outstanding write transactions to complete. Note that this is still an experimental feature which may be stabilized in future releases.
  • The tablet server web UI scans dashboard (/scans) has been improved with several new features:

    • Shows the most recently completed scans
    • Pseudo-SQL scan descriptor that concisely shows the selected columns and applied predicates
    • More complete, better documented scan statistics
  • Kudu daemons now expose a web page /stacks which dumps the current stack trace of every thread running in the server. This information can be helpful when diagnosing performance issues.

Optimizations and Improvements in Kudu 1.7.0 / CDH 5.15.0

  • By default, each tablet replica now stripes data blocks across three data directories instead of all data directories. This decreases the likelihood that any given tablet is affected in the event of a single disk failure. No substantial performance impact is expected due to this feature based on performance testing. This change only affects new replicas created after upgrading to Kudu 1.7.
  • Kudu servers previously offered the ability to enable a separate metrics log which stores periodic snapshots of all metrics available on a server. This functionality is now available as part of a more general “diagnostics log” which is enabled by default. The diagnostics log includes periodic dumps of server metrics as well as collections of thread stack traces. The default configuration ensures that no more than 640MB of diagnostics logs are retained, and typically the space consumption is significantly less due to compression. The format and contents of this log file are documented in the Apache Kudu Administration.
  • The handling of errors in the synchronous Java client has been improved so that when an exception is thrown, the stack trace indicates the correct location where the client function was invoked rather than a call stack of an internal worker thread. The original call stack from the worker thread is available as a suppressed exception.
  • The logging of errors in the Java client has been improved to exclude exception stack traces for expected scenarios such as failure to connect to a server in a cluster. Instead, only a single line informational message will be logged in such cases to aid in debugging.
  • The Java client now uses a predefined prioritized list of TLS ciphers when establishing an encrypted connection to Kudu servers. This cipher list matches the list of ciphers preferred for server-to-server communication and ensures that the most efficient and secure ciphers are preferred. When the Kudu client is running on Java 8 or newer, this provides a substantial speed-up to read and write performance.
  • Reporting for the kudu cluster ksck tool has been updated so tablets and tables with on-going tablet copies are shown as recovering. Additional reporting changes have been made to make various common scenarios, particularly tablet copies, less alarming.
  • The performance of inserting rows containing many string or binary columns has been improved, especially in the case of highly concurrent write workloads.
  • By default, Spark tasks that scan Kudu are now able to scan non-leader replicas. This allows Spark to more easily schedule kudu-spark tasks local to the data. You can disable this behavior by passing leader_only to the kudu.scanLocality option.
  • The number of OS threads used in the steady state and during bursts of activity, such as in Raft leader elections triggered by a node failure, has been drastically reduced and should no longer exceed the value of ulimit -u. It should no longer be necessary to increase the value of ulimit -u or of /proc/sys/kernel/threads-max in order to run a Kudu tablet server in most cases. See KUDU-1913 for more information.
  • An issue where sparse column predicates could cause excessive data-block reads has been fixed. Previously in certain scans with sparsely matching predicates on multiple columns, Kudu read and decoded the same data blocks many times. This improvement typically results in a 5-10x performance increase for the affected scans. See KUDU-2231 for more information.
  • The efficiency and on-disk size of large updated values has been improved. This improves update-heavy workloads which overwrite large (1KiB+) values. See KUDU-2253 for more information.

Wire Protocol Compatibility

Kudu 1.7.0 / CDH 5.15.0 is wire-compatible with previous versions of Kudu:

  • Kudu 1.7 clients can connect to servers running Kudu 1.0 or later. If the client uses features that are not available on the target server, an error is returned.
  • Rolling upgrade to Kudu 1.7 has not been sufficiently tested. We recommend against rolling upgrades in Kudu. You should shut down all nodes in the cluster, upgrade the software, and then restart the daemons on the new version.
  • Kudu 1.0 clients may connect to servers running Kudu 1.7 with the exception of the below-mentioned restrictions regarding secure clusters.

The authentication features introduced in Kudu 1.3 place the following limitations on wire compatibility between Kudu 1.7 and versions earlier than 1.3:

  • If a Kudu 1.7 cluster is configured with authentication or encryption is set to required, clients older than Kudu 1.3 are unable to connect.
  • If a Kudu 1.7 cluster is configured with authentication and encryption is set to optional or disabled, older clients are still able to connect.

Client Library Compatibility

  • The Kudu 1.7 Java client library is API- and ABI-compatible with Kudu 1.6. Applications written against Kudu 1.6 compile and run against the Kudu 1.7 client library and vice-versa.
  • The Kudu 1.7 C++ client is API- and ABI-forward-compatible with Kudu 1.6. Applications written and compiled against the Kudu 1.6 client library run without modification against the Kudu 1.7 client library. Applications written and compiled against the Kudu 1.7 client library run without modification against the Kudu 1.6 client library.
  • The Kudu 1.7 Python client is API-compatible with Kudu 1.6. Applications written against Kudu 1.6 continue to run against the Kudu 1.7 client and vice-versa.
  • Kudu 1.7 clients that attempt to create a table with a decimal column on a target server running Kudu 1.6 or earlier receive an error response. Similarly Kudu clients running Kudu 1.6 or earlier result in an error when attempting to access any table containing a decimal column.

Kudu 1.7.0 / CDH 5.15.0 Incompatible Changes

  • The newly introduced replica management scheme is not compatible with the old scheme, so it is not possible to run pre-1.7 Kudu masters with 1.7 Kudu tablet servers or vice versa. This is a server-side incompatibility only and it does not affect client compatibility. Kudu clients of prior versions are compatible with upgraded Kudu clusters:

    • Kudu masters of 1.7 version do not register Kudu tablet servers of 1.6 and prior versions.
    • Kudu tablet servers of 1.7 version does not work with Kudu masters of 1.6 and prior versions.
  • The format of the previously-optional metrics log has changed to include a human-readable timestamp on each line. The path of the log file has also changed with the word “diagnostics” replacing the word “metrics” in the file name. The metrics log has been optimized to only include those metrics which have changed in between successive samples, and to not include entity attributes such as tablet partition information in the log. See KUDU-2297 for more information.

Kudu 1.6.0 / CDH 5.14.2 Release Notes

New Features in Kudu 1.6.0 / CDH 5.14.2

  • Cloudera Manager now supports installing Kudu as part of the Express Wizard installation.
  • Cloudera Manager has improved monitoring of Kudu node health. Cloudera Manager will indicate whether the health of the service is concerning or bad according to the number of master or tablet server roles that are not running.

Kudu 1.6.0 / CDH 5.14.0 Release Notes

Apache Kudu 1.6.0 / CDH 5.14.0 is the first Kudu release to be supported with Cloudera Manager 5.14.x and CDH 5.14.x. Any future Kudu 1.6.x maintenance releases will be supported by corresponding CDH and Cloudera Manager 5.14.x maintenance releases.

New Features in Kudu 1.6.0 / CDH 5.14.0

  • The disk failure tolerance feature is enabled by default on tablet servers, and the capability was extended to handle data directory failures at runtime. In the event of a runtime disk failure, any tablets with data on a failed disk will be shut down and restarted on another tablet server. By default, tablets are striped across all available disks. Note that the first configured data directory and the WAL directory cannot currently tolerate disk failures. This will be further improved in future Kudu releases.

  • Kudu servers can now adopt new data directories via the kudu fs update_dirs tool. The new directory will be used by new tablet replicas only. Removing directories is not yet supported (KUDU-2202).

  • Kudu servers have two new flags to control the webui TLS/HTTPS settings: --webserver_tls_ciphers and --webserver_tls_min_protocol. These flags allow the advertised TLS ciphers and TLS protocol versions to be configured. Additionally, the webserver now excludes insecure legacy ciphers by default (KUDU-2190).

Optimization and Improvements in Kudu 1.6.0 / CDH 5.14.0

  • Kudu servers can now tolerate short interruptions in NTP clock synchronization. NTP synchronization is still required when any Kudu daemon starts up. If NTP synchronization is not available, diagnostic information is now logged to help pinpoint the issue (KUDU-1578).

  • Tablet server startup time has been improved significantly on servers containing large numbers of blocks.

  • The log block manager now performs disk data deletion in batches. This optimization can significantly reduce the time taken to delete data on a tablet.

  • The usage of sensitive data redaction flag has been changed. By setting the --redact=log flag, redaction will be disabled in the web UI but retained for server logs. Alternatively, --redact=none can be used to disable redaction completely.

  • The Spark DataSource integration now can take advantage of scan locality for better scan performance. The scan will take place at the closest replica instead of going to the leader.

  • Various optimizations were made to reduce the 99th percentile latency of writes on tablet servers. This can also improve throughput on certain write workloads, particularly on larger clusters.

  • You can configure Kudu to ignore the system-wide auth_to_local mappings configured in /etc/krb5.conf by setting the configuration flag --use_system_auth_to_local=false (KUDU-2198).

  • The performance of the compaction scheduler has been improved. In previous versions, certain types of time series workloads could cause compaction scheduling to take tens of seconds. These workloads now schedule compactions an order of magnitude more efficiently.

  • The compaction scheduler has been improved to avoid running a compaction when the benefit of that compaction is extremely small.

  • Tablet servers now consider the health of all replicas of a tablet before deciding to evict one. This can improve stability of the Kudu cluster after experiencing multiple simultaneous daemon failures (KUDU-2048).

  • Several performance improvements have been made to the Kudu master, particularly in concurrency of clients opening tables. This should improve performance in highly concurrent workloads.

  • The on-disk size metric for a tablet now includes all data and metadata. Previously, it excluded WAL segments and consensus metadata (KUDU-1755).

  • Added a verbose mode for the kudu cluster ksck command to enable output of detailed information on the cluster’s metadata, even when no errors are detected.

Wire Protocol Compatibility

Kudu 1.6.0 is wire-compatible with previous versions of Kudu:

  • Kudu 1.6 clients may connect to servers running Kudu 1.0 or later. If the client uses features that are not available on the target server, an error will be returned.

  • Rolling upgrade between Kudu 1.5 and Kudu 1.6 servers may be possible though has not been sufficiently tested. Users are encouraged to shut down all nodes in the cluster, upgrade the software, and then restart the daemons on the new version.

  • Kudu 1.0 clients may connect to servers running Kudu 1.6 with the exception of the below-mentioned restrictions regarding secure clusters.

The authentication features introduced in Kudu 1.3 place the following limitations on wire compatibility between Kudu 1.6 and versions earlier than 1.3:

  • If a Kudu 1.6 cluster is configured with authentication or encryption set to "required", clients older than Kudu 1.3 will be unable to connect.

  • If a Kudu 1.6 cluster is configured with authentication and encryption set to "optional" or "disabled", older clients will still be able to connect.

Client Library Compatibility

  • The Kudu 1.6 Java client library is API- and ABI-compatible with Kudu 1.5. Applications written against Kudu 1.5 will compile and run against the Kudu 1.6 client library and vice-versa.

  • The Kudu 1.6 C++ client is API- and ABI-forward-compatible with Kudu 1.5. Applications written and compiled against the Kudu 1.5 client library will run without modification against the Kudu 1.6 client library. Applications written and compiled against the Kudu 1.6 client library will run without modification against the Kudu 1.5 client library.

  • The Kudu 1.6 Python client is API-compatible with Kudu 1.5. Applications written against Kudu 1.5 will continue to run against the Kudu 1.6 client and vice-versa.

Kudu 1.6.0 / CDH 5.13.2 Release Notes

New Features in Kudu 1.6.0 / CDH 5.13.2

  • New Command-line Debugging Tool - The kudu fs list command line tool replaces exploratory usages of kudu fs dump and kudu local_replica dump. Output is formatted with default pretty-printing with options to output in CSV and JSON for scripts. Results can easily be filtered to a specific table, tablet, column, rowset, or block using flags. The tool can output many different fields: table, table-id, tablet-id, partition, rowset-id, block-id, block-kind, column, column-id, cfile-data-type, cfile-encoding, cfile-compression, cfile-num-values, cfile-size, cfile-incompatible-features, cfile-compatible-features, cfile-min-key, cfile-max-key, and cfile-delta-stats.

Kudu 1.5.0 / CDH 5.13.0 Release Notes

Apache Kudu 1.5.0 / CDH 5.13.0 is the first Kudu release to be supported with Cloudera Manager 5.13.x and CDH 5.13.x. Any future Kudu 1.5.x maintenance releases will be supported by corresponding CDH and Cloudera Manager 5.13.x maintenance releases.

New Features in Kudu 1.5.0 / CDH 5.13.0

  • Integrated into CDH - Starting with Apache Kudu 1.5.0 / CDH 5.13.0, Kudu is fully integrated into CDH and ships as part of the CDH parcel and packages. On a cluster managed by Cloudera Manager, Kudu will be installed as part of CDH and does not need to be installed separately.

  • Kudu Web UI Improvements - Kudu server web UIs have a new configuration dashboard (/config) which provides a high level summary of important security configuration values, such as whether RPC authentication is required, or web server HTTPS encryption is enabled. Other types of configuration will be added in future releases.

  • New Command-line Tools - The kudu command line tool has two new features:
    • kudu tablet change_config move_replica - This tool moves a tablet replica from one tablet server to another, under the condition that the tablet is healthy. An operator can use this tool to rebalance tablet replicas between tablet servers.
    • kudu local_replica data_size - This tool summarizes the space usage of a tablet, breaking it down by type of file, column, and rowset.

Optimizations and Improvements in Kudu 1.5.0 / CDH 5.13.0

  • The log block manager now performs disk synchronization in batches. This optimization can significantly reduce the time taken to copy tablet data from one server to another; in one case tablet copy time is reduced by 35%. It also improves the general performance of flushes and compactions.

  • A new feature referred to as "tombstoned voting" is added to the Raft consensus subsystem to allow tablet replicas in the TABLET_DATA_TOMBSTONED state to vote in tablet leader elections. This feature increases Kudu’s stability and availability by improving the likelihood that Kudu will be able to self-heal in more edge-case scenarios, such as when tablet copy operations fail. See KUDU-871 for details.

  • The tablet on-disk size metric has been made more accurate. Previously, the metric included only REDO deltas; it now counts all deltas. Additionally, the metric includes the size of bloomfiles, ad hoc indexes, and the tablet superblock. WAL segments and consensus metadata are still not counted. The latter is very small compared to the size of data, but the former may be significant depending on the workload (this will be resolved in a future release).

  • The number of threads used by the Kudu tablet server has been further reduced. Previously, each follower tablet replica used a dedicated thread to detect leader tablet replica failures, and each leader replica used one dedicated thread per follower to send Raft heartbeats to that follower. The work performed by these dedicated threads has been reassigned to other threads. Other improvements were made to facilitate better thread sharing by tablets. For the purpose of capacity planning, expect the Kudu tablet server to create one thread for every five "cold" (i.e. those not servicing writes) tablets, and an additional three threads for every "hot" tablet. This will be further improved upon in future Kudu releases.

Wire Protocol Compatibility

Kudu 1.5.0 is wire-compatible with previous versions of Kudu:

  • Kudu 1.5 clients may connect to servers running Kudu 1.0 or later. If the client uses features that are not available on the target server, an error will be returned.

  • Rolling upgrade between Kudu 1.4 and Kudu 1.5 servers is believed to be possible though has not been sufficiently tested. Users are encouraged to shut down all nodes in the cluster, upgrade the software, and then restart the daemons on the new version.

  • Kudu 1.0 clients may connect to servers running Kudu 1.5 with the exception of the below-mentioned restrictions regarding secure clusters.

The authentication features introduced in Kudu 1.3 place the following limitations on wire compatibility between Kudu 1.5 and versions earlier than 1.3:

  • If a Kudu 1.5 cluster is configured with authentication or encryption set to "required", clients older than Kudu 1.3 will be unable to connect.

  • If a Kudu 1.5 cluster is configured with authentication and encryption set to "optional" or "disabled", older clients will still be able to connect.

Client Library Compatibility

  • The Kudu 1.5 Java client library is API- and ABI-compatible with Kudu 1.4. Applications written against Kudu 1.4 will compile and run against the Kudu 1.5 client library and vice-versa, unless one of the following newly added APIs is used:

  • The Kudu 1.5 C++ client is API- and ABI-forward-compatible with Kudu 1.4. Applications written and compiled against the Kudu 1.4 client library will run without modification against the Kudu 1.5 client library. Applications written and compiled against the Kudu 1.5 client library will run without modification against the Kudu 1.4 client library.

  • The Kudu 1.5 Python client is API-compatible with Kudu 1.4. Applications written against Kudu 1.4 will continue to run against the Kudu 1.5 client and vice-versa.

Known Issues and Limitations in Kudu 1.5.0 / CDH 5.13.0

For a complete list of known issues for Apache Kudu, see Apache Kudu Known Issues.

Kudu 1.4.0 / CDH 5.12.1 Release Notes

Apache Kudu 1.4.0 / CDH 5.12.1 is a bug-fix release which includes the following fixes:

  • KUDU-2085 - Fixed a bug that caused crashes when seeking past the end of prefix-encoded blocks.
  • KUDU-2087 - Fixed an issue where Kudu would fail to start when Kerberos was enabled in FreeIPA-configured deployments.
  • KUDU-2053 - Fixed a race condition in the Java RequestTracker.
  • KUDU-2049 - Fixed an issue where scans on RLE-encoded integer columns would sometimes cause CHECK failures due to the CHECK condition being too strict.
  • KUDU-2052 - Kudu now uses XFS_IOC_UNRESVSP64 ioctl to punch holes on xfs filesystems. This fixes an issue with slow startup times on xfs when hole punching was done via fallocate().
  • KUDU-1956 - Kudu will no longer crash if faced with a race condition when selecting rowsets for compaction.

Kudu 1.4.0 / CDH 5.12.0 Release Notes

Apache Kudu 1.4.0 / CDH 5.12.0 release is the first Kudu release to be supported with Cloudera Manager 5.12.x and CDH 5.12.x. Any future Kudu 1.4.x maintenance releases will be supported by corresponding CDH and Cloudera Manager 5.12.x maintenance releases.

New Features in Kudu 1.4.0 / CDH 5.12.0

  • The C++ and Java client libraries now support the ability to alter the storage attributes (for example, encoding and compression) and default value of existing columns. Additionally, it is now possible to rename a column which is part of a table's primary key.
  • The C++ client library now includes an experimental KuduPartitioner API which can be used to efficiently map rows to their associated partitions and hosts. This can be used to achieve better locality or distribution of writes in client applications.
  • The Java client library now supports enabling fault tolerance on scanners. Fault tolerant scanners are able to transparently recover from concurrent server crashes at the cost of some performance overhead. See the Java API documentation for more details on usage.
  • Kudu 1.4.0 includes the optional ability to compute, store, and verify checksums on all pieces of data stored on a server. Prior versions only performed checksums on certain portions of the stored data. This feature is not enabled by default since it makes a backward-incompatible change to the on-disk formats and thus prevents downgrades.
  • Kudu Command Line Tools
    • The kudu command line tool now includes a new advanced administrative command, kudu remote_replica unsafe_change_config. This command can be used to force a tablet to perform an unsafe change of its Raft replication configuration. This can then be used to recover from scenarios such as a loss of a majority of replicas, at the risk of losing edits.
    • A new kudu fs check command can be use to perform various offline consistency checks on the local on-disk storage of a Kudu Tablet Server or Master. In addition to detecting various inconsistencies or corruptions, it can also detect and remove data blocks that are no longer referenced by any tablet but were not fully removed from disk due to a crash or a bug in prior versions of Kudu.
    • The kudu command line tool can now be used to list the addresses and identifiers of the servers in the cluster using either kudu master list or kudu tserver list.

    For a complete list of Kudu's command line tools and usage information, see the Command Line Reference documentation.

Optimizations and Improvements in Kudu 1.4.0 / CDH 5.12.0

  • kudu cluster ksck now detects and reports new classes of inconsistencies and issues. In particular, it is better able to detect cases where a configuration change such as a replica eviction or addition is pending but is unable to be committed. It also properly detects and reports cases where a tablet has no elected leader.
  • The default size for Write Ahead Log (WAL) segments has been reduced from 64MB to 8MB. Additionally, in the case that all replicas of a tablet are fully up to date and data has been flushed from memory, servers will now retain only a single WAL segment rather than two. These changes are expected to reduce the average consumption of disk space on the configured WAL disk by 16x, as well as improve the startup speed of tablet servers by reducing the number and size of WAL segments that need to be re-read.
  • The default on-disk storage system used by Kudu servers (Log Block Manager) has been improved to compact its metadata and remove dead containers. This compaction and garbage collection occurs only at startup. Thus, the first startup after upgrade is expected to be longer than usual, and subsequent restarts should be shorter.
  • The usability of the Kudu web interfaces has been improved, particularly for cases where a server hosts many tablets or a table has many partitions. Pages that list tablets now include a top-level summary of tablet status and show the complete list under a toggleable section.
  • The maintenance manager features improved utilization of the configured maintenance threads. Previously, maintenance work would only be scheduled a maximum of 4 times per second, but now maintenance work will be scheduled immediately whenever any configured thread is available. This can improve the throughput of write-heavy workloads.
  • The maintenance manager aggressively schedules flushes of in-memory data when memory consumption crosses 60 percent of the configured process-wide memory limit. The backpressure mechanism which begins to throttle client writes was also adjusted to not begin throttling until memory consumption reaches 80 percent of the configured limit. These two changes together result in improved write throughput, more consistent latency, and fewer timeouts due to memory exhaustion.
  • Many performance improvements were made to improve write performance. Applications which send large batches of writes to Kudu should see substantially improved throughput in Kudu 1.4.0.
  • Several improvements were made to reduce the memory consumption of Kudu Tablet Servers which hold large volumes of data. The specific amount of memory saved varies depending on workload, but the expectation is that approximately 350MB of excess peak memory usage has been eliminated per TB of data stored.
  • The number of threads used by the Kudu Tablet Server has been reduced. Previously, each tablet used a dedicated thread to append to its WAL. Those threads now automatically stop running if there is no activity on a given tablet for a short period of time.

Fixed Issues in Kudu 1.4.0 / CDH 5.12.0

  • KUDU-2020 - Fixed an issue where re-replication after a failure would proceed significantly slower than expected. This bug caused many tablets to be unnecessarily copied multiple times before successfully being considered re-replicated, resulting in significantly more network and IO bandwidth usage than expected. Mean time to recovery on clusters with large amounts of data is improved by up to 10x by this fix.
  • KUDU-1982 - Fixed an issue where the Java client would call NetworkInterface.getByInetAddress very often, causing performance problems particularly on Windows where this function can be quite slow.
  • KUDU-1755 - Improved the accuracy of the on_disk_size replica metrics to include the size consumed by bloom filters, primary key indexes, superblock metadata, and delta files. Note that because the size metric is now more accurate, the reported values are expected to increase after upgrading to Kudu 1.4.0. This does not indicate that replicas are using more space after the upgrade; rather, it is now accurately reporting the amount of space that has always been used.
  • KUDU-1192 - Kudu servers will now periodically flush their log messages to disk even if no WARNING-level messages have been logged. This makes it easier to tail the logs to see progress output during normal startup.

Wire Protocol Compatibility in Kudu 1.4.0 / CDH 5.12.0

  • Kudu 1.4.0 clients can connect to servers running Kudu 1.0.0 or later. If the client uses features that are not available on the target server, an error will be returned.
  • Kudu 1.0.0 clients can connect to servers running Kudu 1.4.0 with the exception of the restrictions on secure clusters mentioned further on in this list.
  • Rolling upgrade between Kudu 1.3.0 and Kudu 1.4.0 servers is believed to be possible though it has not been sufficiently tested. Users are encouraged to shut down all nodes in the cluster, upgrade the software, and then restart the daemons on the new version.
  • The authentication features introduced in Kudu 1.3.0 place the following limitations on wire compatibility between Kudu 1.4.0 and versions earlier than 1.3.0:
    • If a Kudu 1.4.0 cluster is configured with authentication or encryption set to required, clients older than Kudu 1.3.0 will be unable to connect.
    • If a Kudu 1.4.0 cluster is configured with authentication and encryption set to optional or disabled, older clients will still be able to connect.

Incompatible Changes in Kudu 1.4.0 / CDH 5.12.0

Kudu servers, by default, will now only allow unencrypted or unauthenticated connections from trusted subnets, which are private networks (127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16) and local subnets of all local network interfaces. Unencrypted or unauthenticated connections from publicly routable IPs will be rejected, even if encryption and authentication are not configured.

The trusted subnets can be configured using the --trusted_subnets flag, which can be set to IP blocks represented in CIDR notation separated by comma. Set it to '0.0.0.0/0' to allow unauthenticated connections from all remote IP addresses. However, if network access is not otherwise restricted by a firewall, malicious users may be able to gain unauthorized access. This can be mitigated if authentication and encryption are configured to be required.

Client Library Compatibility

  • Java - The Kudu 1.4.0 Java client library is API- and ABI-compatible with Kudu 1.3.0. Applications written against Kudu 1.3.0 will compile and run against the Kudu 1.4.0 client library and vice-versa, unless one of the following newly-added APIs is used:
    • [Async]KuduScannerBuilder.setFaultTolerant(...)
    • New methods in AlterTableOptions: removeDefault, changeDefault, changeDesiredBlockSize, changeEncoding, changeCompressionAlgorithm
    • KuduClient.updateLastPropagatedTimestamp
    • KuduClient.getLastPropagatedTimestamp
    • New getters in PartialRow: getBoolean, getByte, getShort, getInt, getLong, getFloat, getDouble, getString, getBinaryCopy, getBinary, isNull, isSet.
  • C++ - The Kudu 1.4.0 C++ client is API- and ABI-forward-compatible with Kudu 1.3.0. Applications written and compiled against the Kudu 1.3.0 client library will run without modification against the Kudu 1.4.0 client library. Applications written and compiled against the Kudu 1.4.0 client library will run without modification against the Kudu 1.3.0 client library unless they use one of the following new APIs:
    • KuduPartitionerBuilder
    • KuduPartitioner
    • KuduScanner::SetRowFormatFlags (unstable API)
    • KuduScanBatch::direct_data, KuduScanBatch::indirect_data (unstable API)
  • Python - The Kudu 1.4.0 Python client is API-compatible with Kudu 1.3.0. Applications written against Kudu 1.3.0 will continue to run against the Kudu 1.4.0 client and vice-versa.

Known Issues and Limitations in Kudu 1.4.0 / CDH 5.12.0

For a complete list of schema design and usage limitations for Apache Kudu, see Apache Kudu Known Issues.

Kudu 1.3.0 / CDH 5.11.2 Release Notes

Apache Kudu 1.3.0 / CDH 5.11.2 is a bug-fix release which includes the following fixes:

  • KUDU-2053 - Fixed a race condition in the Java RequestTracker.
  • KUDU-2049 - Fixed an issue where scans on RLE-encoded integer columns would sometimes cause CHECK failures due to the CHECK condition being too strict.
  • KUDU-1963 - Fixed an issue where the Java client misdiagnoses an error and logs a NullPointerException when a connection is closed by client while a negotiation is in progress.
  • KUDU-1853 - Fixed an issue where data blocks could be orphaned after a failed tablet copy.

Kudu 1.3.0 / CDH 5.11.1 Release Notes

Apache Kudu 1.3.0 / CDH 5.11.1 is a bug-fix release which includes the following fixes:

  • KUDU-1999 - Fixed an issue where the Kudu Spark connector would fail to kinit with the principal and keytab provided to a job.

  • KUDU-1993 - Fixed a validation issue with grouped gflags.

  • KUDU-1981 - Fixed an issue where Kudu server components would fail to start on machines with fully-qualified domain names longer than 64 characters when security was enabled. This was due to hard-coded restrictions in the OpenSSL library.

  • KUDU-1607 - Fixed a case in which a tablet replica on a tablet server could retain blocks of data which prevented it from being fully deleted.
  • KUDU-1933 - Fixed an issue in which a tablet server would crash and fail to restart after a single tablet received more than two billion write operations.

Kudu 1.3.0 / CDH 5.11.0 Release Notes

New Features in Kudu 1.3.0 / CDH 5.11.0

  • The Kudu CSD is now included in Cloudera Manager. Manual installation of the Kudu CSD is not required.
  • Kudu 1.3 supports strong authentication using Kerberos. This feature allows users to authenticate themselves to Kudu using Kerberos tickets, and also provides mutual authentication of servers using Kerberos credentials stored in keytabs.

  • Added support for encryption of data on the network using Transport Layer Security (TLS). Kudu will now use TLS to encrypt all network traffic between clients and servers as well as any internal traffic among servers, with the exception of traffic determined to be within a localhost network connection. Encryption is enabled by default whenever it can be determined that both the client and server support the feature.

  • Added support for coarse-grained service-level authorization checks for access to the cluster. The operator may set up lists of permitted users who may act as administrators and as clients of the cluster. Combined with the strong authentication feature described above, this can enable a secure environment for some use cases. Note that fine-grained access control , for example table-level or column-level, is not yet supported.

  • A background task was added to tablet servers that removes historical versions of data that have fallen behind the configured data retention time. This reduces disk space usage in all workloads, but particularly in those with a higher volume of updates or upserts. After upgrading, you might see disk usage go down a lot.

  • Kudu now incorporates Google Breakpad, a library that writes crash reports in the case of a server crash. These reports can be found by default under the minidumps directory, or within a configured log directory, and can be useful during bug diagnosis.

  • As of Kudu 1.3, multi-word flags can use a dash '-' separator in lieu of the underscore '_' separator. For example, the --memory_limit_hard_bytes flag can now be specified as --memory-limit-hard-bytes, or even --memory_limit-hard_bytes.

Optimizations and Improvements in Kudu 1.3.0 / CDH 5.11.0

  • Kudu servers will now change the file permissions of data directories and contained data files based on a new configuration flag --umask. As a result, after upgrading, permissions on disk may be more restrictive than in previous versions. The new default configuration improves data security.

  • Kudu web UIs will now redact strings that might include sensitive user data. For example, the monitoring page, which shows in-progress scans, no longer includes the scanner predicate values. The tracing and RPC diagnostics endpoints no longer include contents of RPCs, which may include table data.

  • By default, Kudu now reserves 1% of each configured data volume as free space. If a volume is seen to have less than 1% of disk space free, Kudu stops writing to that volume to avoid completely filling up the disk.

  • The default encoding for numeric columns (int, float, and double) has been changed to BIT_SHUFFLE. The default encoding for binary and string columns has been changed to DICT_ENCODING. Dictionary encoding automatically falls back to the old default (PLAIN) when cardinality is too high to be effectively encoded. These new defaults match the default behavior of other storage mechanisms, such as Apache Parquet, and are likely to perform better out of the box.

    Existing tables with AUTO_ENCODING-configured columns will only be encoded during DiskRowSet compactions, or if new data is written to them. New tables and columns will have the new defaults.

  • Kudu now uses LZ4 compression when writing its Write Ahead Log (WAL). This improves write performance and stability for many use cases.

  • Kudu now uses LZ4 compression when writing delta files. This can improve both read and write performance as well as save substantial disk usage, especially for workloads involving a high number of updates or upserts containing compressible data.

    Existing delta files will not get compressed until they are compacted. New files on both old and new tables alike will be compressed.

  • The Kudu API now supports the ability to express IS NULL and IS NOT NULL predicates on scanners. The Spark DataSource integration will take advantage of these new predicates when possible.

  • Both C++ and Java clients have been optimized to prune partitions more effectively when performing scans using the IN (…​) predicate.

  • The exception messages produced by the Java client are now truncated to a maximum length of 32KB.

Fixed Issues in Kudu 1.3.0 / CDH 5.11.0

  • KUDU-1968 - Fixed an issue in which the tablet server would delete an incorrect set of data blocks after an aborted attempt to copy a tablet from another server. This would produce data loss in unrelated tablets.
  • KUDU-1962 - Fixed a NullPointerException in the Java client in the case that the Kudu master is overloaded at the time the client requests location information. This could cause client applications to hang indefinitely regardless of configured timeouts.
  • KUDU-1893 - Fixed a critical bug in which wrong results would be returned when evaluating predicates applied to columns added using the ALTER TABLE operation.

  • KUDU-1905 - Fixed an issue where Kudu would crash after reinserts that resulted in an empty change list. This occurred in cases where the primary key was composed of all columns.

  • KUDU-1899 - Fixed a crash that occurred after inserting a row with an empty string as the single-column primary key.

  • KUDU-1904 - Fixed a potential crash when performing random reads against a column using RLE encoding and containing long runs of NULL values.

  • KUDU-1856 - Fixed an issue in which disk space could be leaked by Kudu servers storing data on partitions using the XFS file system. Any leaked disk space will now be automatically recovered upon upgrade.

  • KUDU-1888, KUDU-1906 - Fixed multiple issues in the Java client where operation callbacks would never be triggered, causing the client to hang.

Wire Protocol Compatibility

Kudu 1.3.0 is wire-compatible with previous versions of Kudu:

  • Kudu 1.3.0 clients can connect to servers running Kudu 1.0.0. If the client uses features that are not available on the target server, an error will be returned.

  • Kudu 1.0.0 clients can connect to servers running Kudu 1.3.0 with the exception of the below-mentioned restrictions regarding secure clusters.

  • Rolling upgrade between Kudu 1.2.0 and Kudu 1.3.0 servers is believed to be possible though has not been sufficiently tested. Users are encouraged to shut down all nodes in the cluster, upgrade the software, and then restart the daemons on the new version.

Security - The authentication features introduced in Kudu 1.3.0 place the following limitations on wire compatibility with older versions:

  • If a Kudu 1.3.0 cluster is configured with authentication or encryption set to required, older clients will not be able to connect.

  • If a Kudu 1.3.0 cluster is configured with authentication and encryption set to optional or disabled, older clients will still be able to connect.

Incompatible Changes in Kudu 1.3.0 / CDH 5.11.0

  • Due to storage format changes in Kudu 1.3, downgrading from Kudu 1.3 to earlier versions is not supported. After upgrading to Kudu 1.3, attempting to restart with an earlier version will result in an error.

  • In order to support running MapReduce and Spark jobs on secure clusters, these frameworks now connect to the cluster at job submission time to retrieve authentication credentials which can later be used by the tasks to be spawned. This means that the process submitting jobs to Kudu clusters must have direct access to that cluster.

  • The embedded web servers in Kudu processes now specify the X-Frame-Options: DENY HTTP header which prevents embedding of Kudu web pages in HTML iframe elements.

Client Library Compatibility

  • Java - The Kudu 1.3.0 Java client library is API- and ABI-compatible with Kudu 1.2.0. Applications written against Kudu 1.2.0 will compile and run against the Kudu 1.3.0 client library and vice-versa, unless one of the following newly-added APIs is used:

    • [Async]KuduClient.exportAuthenticationCredentials(…​) (unstable API)
    • [Async]KuduClient.importAuthenticationCredentials(…​) (unstable API)
    • [Async]KuduClient.getMasterAddressesAsString()
    • KuduPredicate.newIsNotNullPredicate()
    • KuduPredicate.newIsNullPredicate()
  • C++ - The Kudu 1.3.0 C++ client is API- and ABI-forward-compatible with Kudu 1.2.0. Applications written and compiled against the Kudu 1.2.0 client library will run without modification against the Kudu 1.3.0 client library. Applications written and compiled against the Kudu 1.3.0 client library will run without modification against the Kudu 1.2.0 client library unless they use one of the following new APIs:

    • kudu::DisableOpenSSLInitialization()
    • KuduClientBuilder::import_authentication_credentials(…​)
    • KuduClient::ExportAuthenticationCredentials(…​)
    • KuduClient::NewIsNotNullPredicate(…​)
    • KuduClient::NewIsNullPredicate(…​)
  • Python - The Kudu 1.3.0 Python client is API-compatible with Kudu 1.2.0. Applications written against Kudu 1.2.0 will continue to run against the Kudu 1.3.0 client and vice-versa.

Known Issues and Limitations in Kudu 1.3.0 / CDH 5.11.0

For a complete list of schema design and usage limitations for Apache Kudu, see Apache Kudu Known Issues.

Kudu 1.2.0 / CDH 5.10.2 Release Notes

Apache Kudu 1.2.x / CDH 5.10.2 includes the following fixed issues.

  • KUDU-1933 - Fixed an issue that truncated the 64-bit log index in the OpId to 32 bits, causing overflow of the log index.
  • KUDU-1607 - Fixed an issue where Kudu could not delete failed tablets using the DROP TABLE command.
  • KUDU-1905 - Allow reinserts on tables when all columns are part of the primary key.
  • KUDU-1893 - Made a fix to avoid incorrect NULL results and ensure evaluation of predicates for columns added after table creation.

Kudu 1.2.0 / CDH 5.10.1 Release Notes

Apache Kudu 1.2.x / CDH 5.10.1 includes the following fixed issues.

  • KUDU-1904 - Fixed a bug where RLE columns with only NULL values would crash on scan.
  • KUDU-1899 - Fixed an issue where tablet servers would crash after inserting an empty string primary key ("").
  • KUDU-1851 - Fixed an issue with the Python client which would crash whenever a TableAlterer is instantiated directly.
  • KUDU-1852 - KuduTableAlterer will no longer crash when given nullptr range bound arguments.
  • KUDU-1821 - Improved warnings when the catalog manager starts.

Kudu 1.2.0 / CDH 5.10.0 Release Notes

New Features and Improvements in Kudu 1.2.0 / CDH 5.10.0

See also Issues resolved for Kudu 1.2.0 and Git changes between 1.1.x and 1.2.x.

New Features
  • Kudu clients and servers now redact user data such as cell values from log messages, Java exception messages, and Status strings. User metadata such as table names, column names, and partition bounds are not redacted.
  • Kudu's ability to provide consistency guarantees has been substantially improved:
    • Replicas now correctly track their "safe timestamp". This timestamp is the maximum timestamp at which reads are guaranteed to be repeatable.
    • A scan created using the SCAN_AT_SNAPSHOT mode will now either wait for the requested snapshot to be "safe" at the replica being scanned, or be re-routed to a replica where the requested snapshot is "safe". This ensures that all such scans are repeatable.
    • Kudu Tablet Servers now properly retain historical data when a row with a given primary key is inserted and deleted, followed by the insertion of a new row with the same key. Previous versions of Kudu would not retain history in such situations. This allows the server to return correct results for snapshot scans with a timestamp in the past, even in the presence of such "reinsertion" scenarios.
    • The Kudu clients now automatically retain the timestamp of their latest successful read or write operation. Scans using the READ_AT_SNAPSHOT mode without a client-provided timestamp automatically assign a timestamp higher than the timestamp of their most recent write. Writes also propagate the timestamp, ensuring that sequences of operations with causal dependencies between them are assigned increasing timestamps. Together, these changes allow clients to achieve read-your-writes consistency, and also ensure that snapshot scans performed by other clients return causally-consistent results.
  • User data in log files is now redacted by default.
  • Kudu servers now automatically limit the number of log files being stored. By default, 10 log files will be retained at each severity level.
Optimizations and Improvements
  • The logging in the Java and cpp clients has been substantially quieted. Clients no longer log messages in normal operation unless there is some kind of error.
  • The cpp client now includes a KuduSession::SetErrorBufferSpace API which can limit the amount of memory used to buffer errors from asynchronous operations.
  • The Java client now fetches tablet locations from the Kudu Master in batches of 1000, increased from batches of 10 in prior versions. This can substantially improve the performance of Spark and Impala queries running against Kudu tables with large numbers of tablets.
  • Table metadata lock contention in the Kudu Master was substantially reduced. This improves the performance of tablet location lookups on large clusters with a high degree of concurrency.
  • Lock contention in the Kudu Tablet Server during high-concurrency write workloads was also reduced. This can reduce CPU consumption and improve performance when a large number of concurrent clients are writing to a smaller number of a servers.
  • Lock contention when writing log messages has been substantially reduced. This source of contention could cause high tail latencies on requests, and when under high load could contribute to cluster instability such as election storms and request timeouts.
  • The BITSHUFFLE column encoding has been optimized to use the AVX2 instruction set present on processors including Intel(R) Haswell and later. Scans on BITSHUFFLE-encoded columns are now up to 30% faster.
  • The kudu tool now accepts hyphens as an alternative to underscores when specifying actions. For example, kudu local-replica copy-from-remote may be used as an alternative to kudu local_replica copy_from_remote.

Issues Fixed in Kudu 1.2.0 / CDH 5.10.0

See Issues resolved for Kudu 1.2.0 and Git changes between 1.1.x and 1.2.x.

  • KUDU-1508 - Fixed a long-standing issue in which running Kudu on ext4 file systems could cause file system corruption. While this issue has been known to still manifest in certain rare cases, the corruption is harmless and can be repaired as part of a regular fsck. Switching from ext4 to xfs will also solve the problem.
  • KUDU-1399 - Implemented an LRU cache for open files, which prevents running out of file descriptors on long-lived Kudu clusters. By default, Kudu will limit its file descriptor usage to half of its configured ulimit.
  • Gerrit #5192 - Fixed an issue which caused data corruption and crashes in the case that a table had a non-composite (single-column) primary key, and that column was specified to use DICT_ENCODING or BITSHUFFLE encodings. If a table with an affected schema was written in previous versions of Kudu, the corruption will not be automatically repaired; users are encouraged to re-insert such tables after upgrading to Kudu 1.2 or later.
  • Gerrit #5541 - Fixed a bug in the Spark KuduRDD implementation which could cause rows in the result set to be silently skipped in some cases.
  • KUDU-1551 - Fixed an issue in which the tablet server would crash on restart in the case that it had previously crashed during the process of allocating a new WAL segment.
  • KUDU-1764 - Fixed an issue where Kudu servers would leak approximately 16-32MB of disk space for every 10GB of data written to disk. After upgrading to Kudu 1.2 or later, any disk space leaked in previous versions will be automatically recovered on startup.
  • KUDU-1750 - Fixed an issue where the API to drop a range partition would drop any partition with a matching lower _or_ upper bound, rather than any partition with matching lower _and_ upper bound.
  • KUDU-1766 - Fixed an issue in the Java client where equality predicates which compared an integer column to its maximum possible value (e.g. Integer.MAX_VALUE) would return incorrect results.
  • KUDU-1780 - Fixed the kudu-client Java artifact to properly shade classes in the com.google.thirdparty namespace. The lack of proper shading in prior releases could cause conflicts with certain versions of Google Guava.
  • Gerrit #5327 - Fixed shading issues in the kudu-flume-sink Java artifact. The sink now expects that Hadoop dependencies are provided by Flume, and properly shades the Kudu client's dependencies.
  • Fixed a few issues using the Python client library from Python 3.

Incompatible Changes in Kudu 1.2.0 / CDH 5.10.0

Apache Kudu 1.2.0 introduces the following incompatible changes:

  • The replication factor of tables is now limited to a maximum of 7. In addition, it is no longer allowed to create a table with an even replication factor.
  • The GROUP_VARINT encoding is now deprecated. Kudu servers have never supported this encoding, and now the client-side constant has been deprecated to match the server's capabilities.
  • Client Library Compatibility
    • The Kudu 1.2 Java client is API- and ABI-compatible with Kudu 1.1. Applications written against Kudu 1.1 will compile and run against the Kudu 1.2 client and vice-versa.
    • The Kudu 1.2 cpp client is API- and ABI-forward-compatible with Kudu 1.1. Applications written and compiled against the Kudu 1.1 client will run without modification against the Kudu 1.2 client. Applications written and compiled against the Kudu 1.2 client will run without modification against the Kudu 1.1 client unless they use one of the following new APIs:
      • kudu::DisableSaslInitialization()
      • KuduSession::SetErrorBufferSpace(...)
    • The Kudu 1.2 Python client is API-compatible with Kudu 1.1. Applications written against Kudu 1.1 will continue to run against the Kudu 1.2 client and vice-versa.

Known Issues and Limitations in Kudu 1.2.0 / CDH 5.10.0

  • KUDU-1893 - Certain query results incorrectly include rows with null values for predicates.

    Solution - Upgrade to the latest maintenance release.

  • Schema and Usage Limitations - For a complete list of schema design and usage limitations for Apache Kudu, see Apache Kudu Known Issues.