Perform the migration

For migrating to multiple Kudu masters, you need to bring the Kudu clusters down. Therefore, identify at least a one-hour maintenance window for this task.

  1. Stop all the Kudu processes in the entire cluster.
  2. Format the data directory on each new master machine, and record the generated UUID. Use the following commands:
    $ sudo -u kudu kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dir>]
    $ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dir>] 2>/dev/null
    master_data_dir

    The new master’s previously recorded data directory.

    For example:
    $ sudo -u kudu kudu fs format --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data
    sudo -u kudu kudu fs dump uuid --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 2>/dev/null
    f5624e05f40649b79a757629a69d061e
  3. If you are using Cloudera Manager, add the new Kudu master roles now, but do not start them.
    • If using DNS aliases, override the empty value of the Master Address parameter for each role (including the existing master role) with that master’s alias.

    • Add the port number (separated by a colon) if using a non-default RPC port value.

  4. Rewrite the master’s Raft configuration with the following command, executed on the existing master:
    $ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dir>] <tablet_id> <all_masters>
    master_data_dir

    The existing master’s previously recorded data directory

    tablet_id

    This must be set to the string, 00000000000000000000000000000000.

    all_masters

    A space-separated list of masters, both new and existing. Each entry in the list must be a string of the form <uuid>:<hostname>:<port>.

    uuid

    The master’s previously recorded UUID.

    hostname

    The master’s previously recorded hostname or alias.

    port

    The master’s previously recorded RPC port number.

    For example:
    $ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:master-1:7051 f5624e05f40649b79a757629a69d061e:master-2:7051 988d8ac6530f426cbe180be5ba52033d:master-3:7051
  5. Modify the value of the master_addresses configuration parameter for both existing master and new masters. The new value must be a comma-separated list of all of the masters. Each entry is a string of the form, <hostname>:<port>.
    hostname

    The master's previously recorded hostname or alias.

    port

    The master's previously recorded RPC port number.

  6. Start the existing master.
  7. Copy the master data to each new master with the following command, executed on each new master machine.
    $ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=<master_data_dir> <tablet_id> <existing_master>
    master_data_dir

    The new master's previously recorded data directory.

    tablet_id

    Must be set to the string, 00000000000000000000000000000000.

    existing_master

    RPC address of the existing master. It must be a string of the form <hostname>:<port>.

    hostname

    The existing master's previously recorded hostname or alias.

    port

    The existing master's previously recorded RPC port number.

    Example
    $ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data  00000000000000000000000000000000 master-1:7051
  8. Start all the new masters.
  9. Modify the value of the tserver_master_addrs configuration parameter for each tablet server. The new value must be a comma-separated list of masters where each entry is a string of the form <hostname>:<port>
    hostname

    The master's previously recorded hostname or alias

    port

    The master's previously recorded RPC port number

  10. Start all the tablet servers.
  11. If you have Kudu tables that are accessed from Impala and you didn’t set up DNS aliases, update the HMS database manually in the underlying database that provides the storage for HMS.
    • The following is an example SQL statement you would run in the HMS database:
      UPDATE TABLE_PARAMS
      SET PARAM_VALUE =
        'master-1.example.com,master-2.example.com,master-3.example.com'
      WHERE PARAM_KEY = 'kudu.master_addresses' AND PARAM_VALUE = 'old-master';
    • Invalidate the metadata by running the command in impala-shell:
      INVALIDATE METADATA;
To verify that all masters are working properly, consider performing the following sanity checks:
  • Using a browser, visit each master’s web UI and navigate to the /masters page. All the masters should now be listed there with one master in the LEADER role and the others in the FOLLOWER role. The contents of /masters on each master should be the same.

  • Run a Kudu system check (ksck) on the cluster using the kudu command line tool.