Perform the migration
For migrating to multiple Kudu masters, you need to bring the Kudu clusters down. Therefore, identify at least a one-hour maintenance window for this task.
- Stop all the Kudu processes in the entire cluster.
-
Format the data directory on each new master machine, and record the generated UUID.
Use the following commands:
$ sudo -u kudu kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dir>] $ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dir>] 2>/dev/null
- master_data_dir
-
The new master’s previously recorded data directory.
For example:$ sudo -u kudu kudu fs format --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data sudo -u kudu kudu fs dump uuid --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 2>/dev/null f5624e05f40649b79a757629a69d061e
-
If you are using Cloudera Manager, add the new Kudu master roles now, but do not start
them.
-
If using DNS aliases, override the empty value of the
Master Address
parameter for each role (including the existing master role) with that master’s alias. -
Add the port number (separated by a colon) if using a non-default RPC port value.
-
-
Rewrite the master’s Raft configuration with the following command, executed on the
existing master:
$ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dir>] <tablet_id> <all_masters>
- master_data_dir
-
The existing master’s previously recorded data directory
- tablet_id
-
This must be set to the string,
00000000000000000000000000000000
. - all_masters
-
A space-separated list of masters, both new and existing. Each entry in the list must be a string of the form
<uuid>:<hostname>:<port>
.- uuid
-
The master’s previously recorded UUID.
- hostname
-
The master’s previously recorded hostname or alias.
- port
-
The master’s previously recorded RPC port number.
- For example:
-
$ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:master-1:7051 f5624e05f40649b79a757629a69d061e:master-2:7051 988d8ac6530f426cbe180be5ba52033d:master-3:7051
-
Modify the value of the
master_addresses
configuration parameter for both existing master and new masters. The new value must be a comma-separated list of all of the masters. Each entry is a string of the form,<hostname>:<port>
.- hostname
-
The master's previously recorded hostname or alias.
- port
-
The master's previously recorded RPC port number.
- Start the existing master.
-
Copy the master data to each new master with the following command, executed on each
new master machine.
$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=<master_data_dir> <tablet_id> <existing_master>
- master_data_dir
-
The new master's previously recorded data directory.
- tablet_id
-
Must be set to the string,
00000000000000000000000000000000
. - existing_master
-
RPC address of the existing master. It must be a string of the form
<hostname>:<port>
.- hostname
-
The existing master's previously recorded hostname or alias.
- port
-
The existing master's previously recorded RPC port number.
- Example
-
$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 master-1:7051
-
Start all the new masters.
-
Modify the value of the
tserver_master_addrs
configuration parameter for each tablet server. The new value must be a comma-separated list of masters where each entry is a string of the form<hostname>:<port>
- hostname
-
The master's previously recorded hostname or alias
- port
-
The master's previously recorded RPC port number
- Start all the tablet servers.
-
If you have Kudu tables that are accessed from Impala and you didn’t set up DNS
aliases, update the HMS database manually in the underlying database that provides the
storage for HMS.
- The following is an example SQL statement you would run in the HMS database:
UPDATE TABLE_PARAMS SET PARAM_VALUE = 'master-1.example.com,master-2.example.com,master-3.example.com' WHERE PARAM_KEY = 'kudu.master_addresses' AND PARAM_VALUE = 'old-master';
- Invalidate the metadata by running the command in
impala-shell
:INVALIDATE METADATA;
- The following is an example SQL statement you would run in the HMS database:
-
Using a browser, visit each master’s web UI and navigate to the
/masters
page. All the masters should now be listed there with one master in theLEADER
role and the others in theFOLLOWER
role. The contents of/masters
on each master should be the same. -
Run a Kudu system check (
ksck
) on the cluster using thekudu
command line tool.