Migrating Brokers in a Cluster

Brokers can be moved to a new host in a Kafka cluster. This might be needed in the case of catastrophic hardware failure. Make sure the following are true before starting:

  • Make sure the cluster is healthy.
  • Make sure all replicas are in sync.
  • Perform the migration when there is minimal load on the cluster.

Brokers need to be moved one-by-one. There are two techniques available:

Using kafka-reassign-partitions tool

This method involves more manual work to modify JSON, but does not require manual edits to configuration files. For more information, see kafka-reassign-partitions.

Modify the broker IDs in meta.properties

This technique involves less manual work, but requires modifying an internal configuration file.

  1. Start up the new broker as a member of the old cluster.

    This creates files in the data directory.

  2. Stop both the new broker and the old broker that it is replacing.
  3. Change broker.id of the new broker to the broker.id of the old one both in Cloudera Manager and in data directory/meta.properties.
  4. (Optional) Run rsync to copy files from one broker to another.

    See Using rsync to Copy Files from One Broker to Another.

  5. Start up the new broker.

    It re-replicates data from the other nodes.

Note that data intensive administration operations such as rebalancing partitions, adding a broker, removing a broker, or bootstrapping a new machine can cause significant additional load on the cluster.

To avoid performance degradation of business workloads, you can limit the resources that these background processes can consume by specifying the -throttleparameter when running kafka-reassign-partitions.

Using rsync to Copy Files from One Broker to Another

You can run rsync command to copy over all data from an old broker to a new broker, preserving modification times and permissions. Using rsync allows you to avoid having to re-replicate the data from the leader. You have to ensure that the disk structures match between the two brokers, or you have to verify the meta.properties file between the source and destination brokers (because there is one meta.properties file for each data directory).

Run the following command on destination broker:

rsync -avz
        src_broker:src_data_dir
        dest_data_dir

If you plan to change the broker ID, edit dest_data_dir/meta.properties.