Perform the removal

When you remove any Kudu masters from a multi-master deployment, you need to rewrite the Raft configuration on the remaining masters, remove data and WAL directories from the unwanted masters, and finaly modify the value of the tserver_master_addrs configuration parameter for the tablet servers to remove the unwanted masters. You need to bring the Kudu clusters down. Therefore, identify at least a one-hour maintenance window for this task.

  1. Perform the Raft configuration change by running the kudu master remove tool.
    Only a single master can be removed at a time. If multiple masters need to be removed, run the tool multiple times. In the following example master-2 is being removed from a Kudu cluster with two masters master-1, and master-2:
    $ sudo -u kudu kudu master remove master-1,master-2 master-2
  2. Remove the data directories and WAL directory on the unwanted masters. This is a precaution to ensure that they cannot start up again and interfere with the new multi-master deployment.
  3. Modify the value of the master_addresses configuration parameter for the masters of the new multi-master deployment.
  4. Restart all of the masters that were not removed.
  5. If you are not using Cloudera Manager: Modify the value of the tserver_master_addrs configuration parameter for the tablet servers to remove any unwanted masters.
  6. Retart all of the tablet servers.
To verify that all masters are working properly, consider performing the following checks:
  • Using a browser, visit each master’s web UI and navigate to the /masters page. All the masters should now be listed there with one master in the LEADER role and the others in the FOLLOWER role. The contents of /masters on each master should be the same.

  • Run a Kudu system check (ksck) on the cluster using the kudu command line tool.