Cloudera Docs

Perform the migration

Use the kudu master add CLI command to orchestrate the migration to multiple masters in an existing Kudu cluster.

The procedure does not require stopping all the Kudu processes in the entire cluster but once the migration procedure is complete, all the Kudu processes must be restarted to incorporate the newly added master which can be done without incurring downtime.

The procedure supports adding only one master at a time. In order to add multiple masters follow the same procedure again for the next new master.

  1. On the new master host run the kudu master add command to add the master.
    Look for any success or error messages on the console or the new master log file. The command is designed to be idempotent so in case of an error after the issue mentioned in the error messages is fixed, run the same command again to make forward progress. After the completion of the procedure irrespective of whether the procedure is successful, the new master is shutdown. The following example adds master-2 to an existing Kudu cluster with master-1: 
    $ sudo -u kudu kudu master add master-1 master-2 --fs_wal_dir=/data/kudu/master/wal \
--fs_data_dirs=/data/kudu/master/data
  2. Modify the value of the master_addresses configuration parameter for existing masters only as the new master is already configured with the updated master_addresses.
    The new value must be a comma-separated list of all of the masters. Each entry is a string of the form <hostname>:<port>:
    hostname

    The master's previously recorded hostname or alias

    port

    The master's previously recorded RPC port number

  3. Restart the existing masters one by one.
  4. Start all the new masters.
  5. Modify the value of the tserver_master_addrs configuration parameter for each tablet server.
    The new value must be a comma-separated list of masters where each entry is a string of the form <hostname>:<port>:
    hostname

    The master's previously recorded hostname or alias

    port

    The master's previously recorded RPC port number

  6. Restart all the tablet servers to pick up the new master configuration.
  7. If you have Kudu tables that are accessed from Impala and you didn’t set up DNS aliases, update the HMS database manually in the underlying database that provides the storage for HMS.
    • The following is an example SQL statement you would run in the HMS database: 
      UPDATE TABLE_PARAMS
SET PARAM_VALUE =
  'master-1.example.com,master-2.example.com'
WHERE PARAM_KEY = 'kudu.master_addresses' AND PARAM_VALUE = 'master-1.example.com';
    • In impala-shell run the following command: 
      INVALIDATE METADATA;
To verify that all masters are working properly, consider performing the following checks:

  • Using a browser, visit each master’s web UI and navigate to the /masters page. All the masters should be listed there with one master in the LEADER role and the others in the FOLLOWER role. The contents of /masters on each master should be the same.

  • Run a Kudu system check (ksck) on the cluster using the kudu command line tool.