Apache Kudu Administration

Kudu master processes serve their web interface on port 8051. The interface exposes several pages with information about the state of the cluster.

• A list of tablet servers, their host names, and the time of their last heartbeat.

• A list of tables, including schema and tablet location information for each.

• SQL code which you can paste into Impala Shell to add an existing table to Impala’s list of known data sources.

Each tablet server serves a web interface on port 8050. The interface exposes information about each tablet hosted on the server, its current state, and debugging information about maintenance background operations.

Both Kudu masters and tablet servers expose the following information via their web interfaces:

• HTTP access to server logs.

• An /rpcz endpoint which lists currently running RPCs via JSON.

• Details about the memory usage of different components of the process.

• The current set of configuration flags.

• Currently running threads and their resource consumption.

• A JSON endpoint exposing metrics about the server.

• The version number of the daemon deployed on the cluster.

These interfaces are linked from the landing page of each daemon’s web UI.

The full set of available metrics for a Kudu server can be dumped using a special command line flag:

$ kudu-tserver --dump_metrics_json
$ kudu-master --dump_metrics_json

This will output a large JSON document. Each metric indicates its name, label, description, units, and type. Because the output is JSON-formatted, this information can easily be parsed and fed into other tooling which collects metrics from Kudu servers.

If you are using Cloudera Manager, see Cloudera Manager Metrics for Kudu for the complete list of metrics collected by Cloudera Manager for a Kudu service.

Metrics can be collected from a server process via its HTTP interface by visiting /metrics. The output of this page is JSON for easy parsing by monitoring services. This endpoint accepts several GET parameters in its query string:

• /metrics?metrics=<substring1>,<substring2>,…​ - Limits the returned metrics to those which contain at least one of the provided substrings. The substrings also match entity names, so this may be used to collect metrics for a specific tablet.

• /metrics?include_schema=1 - Includes metrics schema information such as unit, description, and label in the JSON output. This information is typically omitted to save space.

• /metrics?compact=1 - Eliminates unnecessary whitespace from the resulting JSON, which can decrease bandwidth when fetching this page from a remote host.

• /metrics?include_raw_histograms=1 - Include the raw buckets and values for histogram metrics, enabling accurate aggregation of percentile metrics over time and across hosts.

For example:

$ curl -s 'http://example-ts:8050/metrics?include_schema=1&metrics=connections_accepted'

[
    {
        "type": "server",
        "id": "kudu.tabletserver",
        "attributes": {},
        "metrics": [
            {
                "name": "rpc_connections_accepted",
                "label": "RPC Connections Accepted",
                "type": "counter",
                "unit": "connections",
                "description": "Number of incoming TCP connections made to the RPC server",
                "value": 92
            }
        ]
    }
]

$ curl -s 'http://example-ts:8050/metrics?metrics=log_append_latency'

[
    {
        "type": "tablet",
        "id": "c0ebf9fef1b847e2a83c7bd35c2056b1",
        "attributes": {
            "table_name": "lineitem",
            "partition": "hash buckets: (55), range: [(<start>), (<end>))",
            "table_id": ""
        },
        "metrics": [
            {
                "name": "log_append_latency",
                "total_count": 7498,
                "min": 4,
                "mean": 69.3649,
                "percentile_75": 29,
                "percentile_95": 38,
                "percentile_99": 45,
                "percentile_99_9": 95,
                "percentile_99_99": 167,
                "max": 367244,
                "total_sum": 520098
            }
        ]
    }
]

Kudu can be configured to periodically dump all of its metrics to a local log file using the --metrics_log_interval_ms flag. Set this flag to the interval at which metrics should be written to a log file.

The metrics log will be written to the same directory as the other Kudu log files, and with the same naming format. After any metrics log file reaches 64MB uncompressed, the log will be rolled and the previous file will be gzip-compressed.

For high availability and to avoid a single point of failure, Kudu clusters should be created with multiple masters. Many Kudu clusters were created with just a single master, either for simplicity or because Kudu multi-master support was still experimental at the time. This workflow demonstrates how to migrate to a multi-master configuration.

Kudu multi-master deployments function normally in the event of a master loss. However, it is important to replace the dead master; otherwise a second failure may lead to a loss of availability, depending on the number of available masters. This workflow describes how to replace the dead master.

Due to KUDU-1620, it is not possible to perform this workflow without also restarting the live masters. As such, the workflow requires a maintenance window, albeit a brief one as masters generally restart quickly.

The kudu CLI includes a tool called ksck which can be used for monitoring cluster health and data integrity. ksck will identify issues such as under-replicated tablets, unreachable tablet servers, or tablets without a leader.

ksck should be run from the command line, and requires you to specify the complete list of Kudu master addresses:

kudu cluster ksck master-01.example.com,master-02.example.com,master-03.example.com

To see the full list of the options available with ksck, either use the --help flag or visit the Kudu command line reference documentation.

If the cluster is healthy, ksck will print a success message, and return a zero (success) exit status.

Connected to the Master
Fetched info from all 1 Tablet Servers
Table IntegrationTestBigLinkedList is HEALTHY (1 tablet(s) checked)

The metadata for 1 table(s) is HEALTHY
OK

If the cluster is unhealthy, for instance if a tablet server process has stopped, ksck will report the issue(s) and return a non-zero exit status:

Connected to the Master
WARNING: Unable to connect to Tablet Server 8a0b66a756014def82760a09946d1fce
(tserver-01.example.com:7050): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 192.168.0.2:7050: connect: Connection refused (error 61)
WARNING: Fetched info from 0 Tablet Servers, 1 weren't reachable
Tablet ce3c2d27010d4253949a989b9d9bf43c of table 'IntegrationTestBigLinkedList'
is unavailable: 1 replica(s) not RUNNING
  8a0b66a756014def82760a09946d1fce (tserver-01.example.com:7050): TS unavailable [LEADER]
  Table IntegrationTestBigLinkedList has 1 unavailable tablet(s)

  WARNING: 1 out of 1 table(s) are not in a healthy state
  ==================
  Errors:
  ==================
  error fetching info from tablet servers: Network error: Not all Tablet Servers are reachable
  table consistency check error: Corruption: 1 table(s) are bad

  FAILED
  Runtime error: ksck discovered errors

To verify data integrity, the optional --checksum-scan flag can be set, which will ensure that the cluster has consistent data by scanning each tablet replica and comparing results. The --tables and --tablets flags can be used to limit the scope of the checksum scan to specific tables or tablets, respectively.

For example, use the following command to check the integrity of data in the IntegrationTestBigLinkedList table:

kudu cluster ksck --checksum-scan --tables IntegrationTestBigLinkedList master-01.example.com,master-02.example.com,master-03.example.com

Kudu tablet servers are not resistant to disk failure. When a disk containing a data directory or the write-ahead log (WAL) dies, the entire tablet server must be rebuilt. Kudu will automatically re-replicate tablets on other servers after a tablet server fails, but manual intervention is needed in order to restore the failed tablet server to a running state.

The first step to restoring a tablet server after a disk failure is to replace the failed disk, or remove the failed disk from the data-directory and/or WAL configuration. Next, the existing data directories and WAL directory must be removed. For example, if the tablet server is configured with --fs_wal_dir=/data/0/kudu-tserver-wal and --fs_data_dirs=/data/1/kudu-tserver,/data/2/kudu-tserver, the following commands will remove the existing data directories and WAL directory:

rm -rf /data/0/kudu-tserver-wal /data/1/kudu-tserver /data/2/kudu-tserver

After the WAL and data directories are removed, the tablet server process can be started. When Kudu is installed using system packages, service is typically used as follows:

sudo service kudu-tserver start

Once the tablet server is running again, new tablet replicas will be created on it as necessary.

If you find that the size of your Kudu cloud deployment has exceeded previous expectations, or you simply wish to allocate more storage to Kudu, use the following set of high-level steps as a guide to increasing storage on your Kudu master or tablet server hosts. You must work with your cluster's Hadoop administrators and the system administrators to complete this process. Replace the file paths in the following steps to those relevant to your setup.