Rebuild a Kudu filesystem layout

In the event that critical files are lost, i.e. WALs or tablet-specific metadata, all Kudu directories on the server must be deleted and rebuilt to ensure correctness. Doing so will destroy the copy of the data for each tablet replica hosted on the local server. Kudu will automatically re-replicate tablet replicas removed in this way, provided the replication factor is at least three and all other servers are online and healthy.

  1. The first step to rebuilding a server with a new directory configuration is emptying all of the server’s existing directories. For example, if a tablet server is configured with --fs_wal_dir=/data/0/kudu-tserver-wal, --fs_metadata_dir=/data/0/kudu-tserver-meta, and --fs_data_dirs=/data/1/kudu-tserver,/data/2/kudu-tserver, the following commands will remove the WAL directory’s and data directories' contents:
    # Note: this will delete all of the data from the local tablet server.
    $ rm -rf /data/0/kudu-tserver-wal/* /data/0/kudu-tserver-meta/* /data/1/kudu-tserver/* /data/2/kudu-tserver/*
  2. If using Cloudera Manager, update the configurations for the rebuilt server to include only the desired directories. Make sure to only update the configurations of servers to which changes were applied, rather than of the entire Kudu service.
  3. Run the following command to delete the tablet server's UUID entry in the Kudu master system catalog:
    kudu tserver unregister <master_addresses> <tserver_uuid>
  4. After you delete the directories and erase the UUID entry from the masters, you can start the server process with the new directory configuration. Kudu creates the appropriate sub-directories during startup, and the server registers with the Kudu master by providing the new generated UUID.