Perform the migration

When migrating to a multiple Kudu master cluster configuration, you first need to add the new Kudu master role at the new master node using the Cloudera Manager UI and then use the kudu master add CLI command.

The migration procedure does not require stopping all the Kudu processes in the entire cluster but once the migration is complete, all the Kudu processes must be restarted to incorporate the newly added masters. The restarting of the Kudu processes can be done without incurring downtime.

The procedure supports adding only one master at a time. In order to add multiple masters follow the same procedure again for the next new master.

In Cloudera Manager, add a new Kudu Master role to the selected new master node:
1. Select the Kudu service.
2. Select Instances.
3. Click Add Role Instances.
4. Click Select host.
5. Select the host node and click OK.
6. Click Continue.
7. Review the changes and if everything is correct, click Finish.
Optional: If you have a secure Kudu cluster, run the kinit command on the new master node. That provides Keberos credentials for the kudu master add CLI tool to be run in the same shell session.
On the new master node, run the kudu master add command to add the master.
The kudu master add command is designed to be idempotent. In case of an error, fix the issue mentioned in the error message, and then run the kudu master addcommand again to make forward progress.
After the completion of this step irrespective of whether it is successful, the new master is shut down.
The following example adds master-2.example.com to an existing Kudu cluster with master-1.example.com:
```
$ sudo -u kudu kudu master add master-1.example.com:7051 master-2.example.com:7051 --fs_wal_dir=/data/kudu/master \
--fs_data_dirs=/data/kudu/master
```
note
If your Kudu cluster is secure, in addition to running as the kudu UNIX user, you must authenticate as the Kudu service user prior to running this command. For more information, see Step 2.
Start the newly added master.
Optional: If you have Kudu tables that are accessed from Impala and you did not set up DNS aliases, update the HMS database manually in the underlying database that provides the storage for HMS:
1. Connect to the HMS database.
2. Run an SQL statement similar to the following example:
```
UPDATE TABLE_PARAMS
SET PARAM_VALUE =
  'master-1.example.com,master-2.example.com'
WHERE PARAM_KEY = 'kudu.master_addresses' AND PARAM_VALUE = 'master-1.example.com';
```
3. In impala-shell run the following command: INVALIDATE METADATA;

To verify that all masters are working properly, consider performing the following checks:

Using a browser, visit each master’s web UI and navigate to the /masters page.

All the masters should be listed there with one master in the LEADER role and the others in the FOLLOWER role. The contents of /masters on each master should be the same.
Run a Kudu system check (ksck) on the cluster using the kudu command line tool.
If applicable, run a few quick SQL queries against a couple of migrated Kudu tables using impala-shell or Hue.