Perform the removal

When you remove any Kudu masters from a multi-master deployment, you need to rewrite the Raft configuration on the remaining masters, remove data and WAL directories from the unwanted masters, and finaly modify the value of the tserver_master_addrs configuration parameter for the tablet servers to remove the unwanted masters. You need to bring the Kudu clusters down. Therefore, identify at least a one-hour maintenance window for this task.

  1. Rewrite the Raft configuration on the remaining masters to include only the remaining masters.
    $ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dir>] <tablet_id> <all_masters>
    master_data_dir

    The existing master’s previously recorded data directory

    tablet_id

    This must be set to the string, 00000000000000000000000000000000.

    all_masters

    A space-separated list of masters, both new and existing. Each entry in the list must be a string of the form <uuid>:<hostname>:<port>.

    uuid

    The master’s previously recorded UUID.

    hostname

    The master’s previously recorded hostname or alias.

    port

    The master’s previously recorded RPC port number.

    For example:
    $ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/data/kudu/test/wal --fs_data_dirs=/data/kudu/test/data 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:master-1:7051 f5624e05f40649b79a757629a69d061e:master-2:7051 988d8ac6530f426cbe180be5ba52033d:master-3:7051
  2. Remove the data directories and WAL directory on the unwanted masters. This is a precaution to ensure that they cannot start up again and interfere with the new multi-master deployment.
  3. Modify the value of the master_addresses configuration parameter for the masters of the new multi-master deployment. If migrating to a single-master deployment, the master_addresses flag should be omitted entirely.
  4. Start all of the masters that were not removed.
  5. Modify the value of the tserver_master_addrs configuration parameter for the tablet servers to remove any unwanted masters.
  6. Start all of the tablet servers.
To verify that all masters are working properly, consider performing the following sanity checks:
  • Using a browser, visit each master’s web UI and navigate to the /masters page. All the masters should now be listed there with one master in the LEADER role and the others in the FOLLOWER role. The contents of /masters on each master should be the same.

  • Run a Kudu system check (ksck) on the cluster using the kudu command line tool.