Perform the migration

When migrating to a multiple Kudu master cluster configuration, you first need to add the new Kudu master role at the new master node using the Cloudera Manager UI and then use the kudu master add CLI command.

The migration procedure does not require stopping all the Kudu processes in the entire cluster but once the migration is complete, all the Kudu processes must be restarted to incorporate the newly added masters. The restarting of the Kudu processes can be done without incurring downtime.

The procedure supports adding only one master at a time. In order to add multiple masters follow the same procedure again for the next new master.

In Cloudera Manager, add a new Kudu Master role to the selected new master node:
1. Select the Kudu service.
2. Select Instances.
3. Click Add Role Instances.
4. Click Select host.
5. Select the host node and click OK.
6. Click Continue.
7. Review the changes and if everything is correct, click Finish.
Optional: Adding a new Kudu Master in a secure cluster
1. From the Cloudera Manager, start the newly added Kudu Master role. This action initiates the start of a kudu-master process.
2. Run the following command to list all flags used by the kudu-master process.
```
ps axww | grep kudu-master
```
3. Stop the newly added Kudu Master role started in step a.
4. Run the following command to view the credentials.
```
sudo -u kudu klist -kt <keytab file path>
```
5. Run kinit with the Kudu keytab and principal to authenticate.
```
sudo -u kudu kinit -kt <keytab file path> <principal>
```
  Ensure the command completes successfully.
On the new master node, run the kudu master add command to add the master. If your Kudu cluster is secure, you must ensure to use all the flags while running kudu master add command. For more information, see Step 2(b).
The kudu master add command is designed to be idempotent. In case of an error, fix the issue mentioned in the error message, and then run the kudu master addcommand again to make forward progress.
After the completion of this step irrespective of whether it is successful, the new master is shut down.
The following example adds master-2.example.com to an existing Kudu cluster with master-1.example.com:
```
$ sudo -u kudu kudu master add master-1.example.com:7051 master-2.example.com:7051 --fs_wal_dir=/data/kudu/master \
--fs_data_dirs=/data/kudu/master
```
The following example adds master-2.example.com to an existing secure Kudu cluster using flags from Step 2(b):
```
$ sudo -u kudu kudu master add master-1.example.com:7051 master-2.example.com:7051 --fs_wal_dir=/data/kudu/master \
--fs_data_dirs=/data/kudu/master \
--rpc_authentication=required \
--rpc_encryption=required \
--webserver_require_spnego=true \
--keytab_file=/path/to/kudu.keytab ...
```
Start the newly added master.
Optional: If you have Kudu tables that are accessed from Impala and you did not set up DNS aliases, update the HMS database manually in the underlying database that provides the storage for HMS:
1. Connect to the HMS database.
2. Run an SQL statement similar to the following example:
```
UPDATE TABLE_PARAMS
SET PARAM_VALUE =
  'master-1.example.com,master-2.example.com'
WHERE PARAM_KEY = 'kudu.master_addresses' AND PARAM_VALUE = 'master-1.example.com';
```
3. In impala-shell run the following command: INVALIDATE METADATA;

To verify that all masters are working properly, consider performing the following checks:

Using a browser, visit each master’s web UI and navigate to the /masters page.

All the masters should be listed there with one master in the LEADER role and the others in the FOLLOWER role. The contents of /masters on each master should be the same.
Run a Kudu system check (ksck) on the cluster using the kudu command line tool.
If applicable, run a few quick SQL queries against a couple of migrated Kudu tables using impala-shell or Hue.