Perform the migration

For migrating to multiple Kudu masters, you need to bring the Kudu clusters down. Therefore, identify at least a one-hour maintenance window for this task.

Stop all the Kudu processes in the entire cluster.

Format the data directory on each new master machine, and record the generated UUID. Use the following commands:

$ sudo -u kudu kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dir>]
$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dir>] 2>/dev/null

master_data_dir

The new master’s previously recorded data directory.

For example:

$ sudo -u kudu kudu fs format --fs_wal_dir=/data/kudu/test/wal --fs_data_dirs=/data/kudu/test/data
sudo -u kudu kudu fs dump uuid --fs_wal_dir=/data/kudu/test/wal --fs_data_dirs=/data/kudu/test/data 2>/dev/null
f5624e05f40649b79a757629a69d061e

If you are using Cloudera Manager, add the new Kudu master roles now, but do not start them.
- If using DNS aliases, override the empty value of the Master Address parameter for each role (including the existing master role) with that master’s alias.
- Add the port number (separated by a colon) if using a non-default RPC port value.
Rewrite the master’s Raft configuration with the following command, executed on the existing master:
```
$ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dir>] <tablet_id> <all_masters>
```
master_data_dir

The existing master’s previously recorded data directory

tablet_id

This must be set to the string, 00000000000000000000000000000000.

all_masters

A space-separated list of masters, both new and existing. Each entry in the list must be a string of the form <uuid>:<hostname>:<port>.

uuid

The master’s previously recorded UUID.

hostname

The master’s previously recorded hostname or alias.

port

The master’s previously recorded RPC port number.

For example:
```
$ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/data/kudu/test/wal --fs_data_dirs=/data/kudu/test/data 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:master-1:7051 f5624e05f40649b79a757629a69d061e:master-2:7051 988d8ac6530f426cbe180be5ba52033d:master-3:7051
```
important
If you are using Cloudera Manager, skip the next step.
Modify the value of the master_addresses configuration parameter for both existing master and new masters. The new value must be a comma-separated list of all of the masters. Each entry is a string of the form, <hostname>:<port>.

hostname

The master's previously recorded hostname or alias.

port

The master's previously recorded RPC port number.
Start the existing master.
Copy the master data to each new master with the following command, executed on each new master machine.

important
If your Kudu cluster is secure, in addition to running as the Kudu UNIX user, you must authenticate as the Kudu service user prior to running this command.
```
$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=<master_data_dir> <tablet_id> <existing_master>
```
master_data_dir

The new master's previously recorded data directory.

tablet_id

Must be set to the string, 00000000000000000000000000000000.

existing_master

RPC address of the existing master. It must be a string of the form <hostname>:<port>.

hostname

The existing master's previously recorded hostname or alias.

port

The existing master's previously recorded RPC port number.

Example
```
$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/data/kudu/test/wal --fs_data_dirs=/data/kudu/test/data  00000000000000000000000000000000 master-1:7051
```
Start all the new masters.

important
If you are using Cloudera Manager, skip the next step.
Modify the value of the tserver_master_addrs configuration parameter for each tablet server. The new value must be a comma-separated list of masters where each entry is a string of the form <hostname>:<port>

hostname

The master's previously recorded hostname or alias

port

The master's previously recorded RPC port number
Start all the tablet servers.
If you have Kudu tables that are accessed from Impala and you didn’t set up DNS aliases, update the HMS database manually in the underlying database that provides the storage for HMS.
- The following is an example SQL statement you would run in the HMS database:
```
UPDATE TABLE_PARAMS
SET PARAM_VALUE =
  'master-1.example.com,master-2.example.com,master-3.example.com'
WHERE PARAM_KEY = 'kudu.master_addresses' AND PARAM_VALUE = 'old-master';
```
- Invalidate the metadata by running the command in impala-shell:
```
INVALIDATE METADATA;
```

To verify that all masters are working properly, consider performing the following sanity checks:

Using a browser, visit each master’s web UI and navigate to the /masters page. All the masters should now be listed there with one master in the LEADER role and the others in the FOLLOWER role. The contents of /masters on each master should be the same.
Run a Kudu system check (ksck) on the cluster using the kudu command line tool.