Changing directory configuration

For higher read parallelism and larger volumes of storage per server, you may want to configure servers to store data in multiple directories on different devices. You can add or remove data directories to an existing master or tablet server by updating the --fs_data_dirs Gflag configuration and restarting the server. Data is striped across data directories, and when a new data directory is added, new data will be striped across the union of the old and new directories.

If on a Kudu version below 1.12, once a server is started, users must go through the below steps to change the directory configuration:

  1. Use ksck to ensure the cluster is healthy, and establish a maintenance window to bring the tablet server offline.
  2. The tool can only run while the server is offline, so establish a maintenance window to update the server. The tool itself runs quickly, so this offline window should be brief, and as such, only the server to update needs to be offline.
    However, if the server is offline for too long (see the follower_unavailable_considered_failed_sec flag), the tablet replicas on it may be evicted from their Raft groups. To avoid this, it may be desirable to bring the entire cluster offline while performing the update.
  3. Run the tool with the desired directory configuration flags. For example, if a cluster was set up with --fs_wal_dir=/wals, ‑‑fs_metadata_dir=/meta, and ‑‑fs_data_dirs=/data/1,/data/2,/data/3, and /data/3 is to be removed (e.g. due to a disk error), run the command:
    $ sudo -u kudu kudu fs update_dirs --force --fs_wal_dir=/wals --fs_metadata_dir=/meta --fs_data_dirs=/data/1,/data/2
  4. Modify the value of the --fs_data_dirs flag for the updated sever. If using Cloudera Manager, make sure to only update the configurations of the updated server, rather than of the entire Kudu service.
  5. Once complete, the server process can be started. When Kudu is installed using system packages, service is typically used:
    $ sudo service kudu-tserver start
  6. Use ksck to ensure Kudu returns to a healthy state before resuming normal operation.