kafka-reassign-partitions

Learn about the kafka-reassign-partitions tool, which is used to reassign partitions between brokers in a cluster.

This tool provides substantial control over partitions in a Kafka cluster. It is mainly used to balance storage loads across brokers through the following reassignment actions:

Change the ordering of the partition assignment list. Used to control leader imbalances between brokers.
Reassign partitions from one broker to another. Used to expand existing clusters.
Increase the replication factor of partitions. Used to expand existing clusters.
Reassign partitions between log directories on the same broker. Used to resolve storage load imbalance among available disks in the broker.
Reassign partitions between log directories across multiple brokers. Used to resolve storage load imbalance across multiple brokers.

The tool uses two JSON files for input. Both of these are created by the user. The two files are the following:

Topics-to-Move JSON
Reassignment Configuration JSON

Topics-to-Move JSON

This JSON file specifies the topics that you want to reassign. This a simple file that tells the kafka-reassign-partitions tool which partitions it should look at when generating a proposal for the reassignment configuration. The user has to create the topics-to-move JSON file from scratch.

The format of the file is the following:

{"topics":  [{"topic": "mytopic1"},
             {"topic": "mytopic2"}],
 "version":1
}

Reassignment Configuration JSON

This JSON file is a configuration file that contains the parameters used in the reassignment process. This file is created by the user, however, a proposal for its contents is generated by the tool. When the kafka-reasssign-partitions tool is executed with the --generate option, it generates a proposed configuration which can be fine-tuned and saved as a JSON file. The file created this way is the reassignment configuration JSON. To generate a proposal, the tool requires a topics-to-move file as input.

The format of the file is the following:

{"version":1,
 "partitions":
   [{"topic":"mytopic1","partition":3,"replicas":[4,5],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":1,"replicas":[5,4],"log_dirs":["any","any"]},
    {"topic":"mytopic2","partition":2,"replicas":[6,5],"log_dirs":["any","any"]}]
}

The reassignment configuration contains multiple properties that each control and specify an aspect of the configuration. The Reassignment Configuration Properties table lists each property and its description.

Table 1. Reassignment Configuration Properties
Property	Description
`topic`	Specifies the topic.
`partition`	Specifies the partition.
`replicas`	Specifies the brokers that the selected partition is assigned to. The brokers are listed in order, which means that the first broker in the list is always the leader for that partition. Change the order of brokers to resolve any leader balancing issues among brokers. Change the broker IDs to reassign partitions to different brokers.
`log_dirs`	Specifies the log directory of the brokers. The log directories are listed in the same order as the brokers. By default `any` is specified as the log directory, which means that the broker is free to choose where it places the replica. By default, the current broker implementation selects the log directory using a round-robin algorithm. An absolute path beginning with a `/` can be used to explicitly set where to store the partition replica.

Notes and Recommendations:

Cloudera recommends that you minimize the volume of replica changes per command instance. Instead of moving 10 replicas with a single command, move two at a time in order to save cluster resources.
This tool cannot be used to make an out-of-sync replica into the leader partition.
Use this tool only when all brokers and topics are healthy.
Anticipate system growth. Redistribute the load when the system is at 70% capacity. Waiting until redistribution becomes necessary due to reaching resource limits can make the redistribution process extremely time consuming.

Tool usage

Learn how to reassign partitions with the kafka-reassign-partitions-tool.

Create a topics-to-move JSON file that specifies the topics you want to reassign.
Use the following format:
```
{"topics":  [{"topic": "mytopic1"},
             {"topic": "mytopic2"}],
 "version":1
}
```

Generate the content for the reassignment configuration JSON with the following command:

kafka-reassign-partitions --bootstrap-server [***HOSTNAME***]:[***PORT***] --topics-to-move-json-file [***TOPICS TO MOVE.JSON***] --broker-list [***BROKER 1***], [***BROKER 2***] --generate

Running the command lists the distribution of partition replicas on your current brokers followed by a proposed partition reassignment configuration.

Current partition replica assignment
{"version":1,
 "partitions":
   [{"topic":"mytopic2","partition":1,"replicas":[2,3],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},
    {"topic":"mytopic2","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":2,"replicas":[3,1],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":1,"replicas":[2,3],"log_dirs":["any","any"]}]
}

Proposed partition reassignment configuration

{"version":1,
 "partitions":
   [{"topic":"mytopic1","partition":0,"replicas":[4,5],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":2,"replicas":[4,5],"log_dirs":["any","any"]},
    {"topic":"mytopic2","partition":1,"replicas":[4,5],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":1,"replicas":[5,4],"log_dirs":["any","any"]},
    {"topic":"mytopic2","partition":0,"replicas":[5,4],"log_dirs":["any","any"]}]
}

In this example, the tool proposed a configuration which reassigns existing partitions on broker 1, 2, and 3 to brokers 4 and 5.

Copy and paste the proposed partition reassignment configuration into an empty JSON file.
Review, and if required, modify the suggested reassignment configuration.
Save the file.

Start the redistribution process with the following command:

kafka-reassign-partitions --bootstrap-server [***HOSTNAME***]:[***PORT***] --reassignment-json-file [***REASSIGNMENT CONFIGURATION.JSON***] --execute

The tool prints a list containing the original replica assignment and a message that reassignment has started. Example output:

Current partition replica assignment

{"version":1,
 "partitions":
   [{"topic":"mytopic2","partition":1,"replicas":[2,3],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},
    {"topic":"mytopic2","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":2,"replicas":[3,1],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":1,"replicas":[2,3],"log_dirs":["any","any"]}]
}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

Verify the status of the reassignment with the following command:

kafka-reassign-partitions --bootstrap-server [***HOSTNAME***]:[***PORT***] --reassignment-json-file [***REASSIGNMENT CONFIGURATION.JSON***] --verify

The tool prints the reassignment status of all partitions.

Status of partition reassignment: 
Reassignment of partition mytopic2-1 completed successfully
Reassignment of partition mytopic1-0 completed successfully
Reassignment of partition mytopic2-0 completed successfully
Reassignment of partition mytopic1-2 completed successfully
Reassignment of partition mytopic1-1 completed successfully

Partitions are reassigned.

Reassignment examples

A collection of examples that demonstrate how users can modify the proposed configuration file generated by the kafka-reassign-partitions.

There are multiple ways to modify the configuration file. The following list of examples shows how a user can modify a proposed configuration and what these changes do.

Suppose that the kafka-reassign-partitions tool generated the following proposed reassignment configuration:

{"version":1,
 "partitions":
   [{"topic":"mytopic1","partition":0,"replicas":[1,2],"log_dirs":["any","any"]}]}

Now let's look at how this reassignment configuration has to be changed in different reassignment scenarios.

Reassign partitions between brokers

To reassign partitions from one broker to another, change the broker ID specified in replicas. For example:

{"topic":"mytopic1","partition":0,"replicas":[5,2],"log_dirs":["any","any"]}

This reassignment configuration moves partition mytopic1-0 from broker 1 to broker 5.

Reassign partitions to another log directory on the same broker

To reassign partitions between log directories on the same broker, change the appropriate any entry to an absolute path. For example:

{"topic":"mytopic1","partition":0,"replicas":[1,2],"log_dirs":["/log/directory1","any"]}

This reassignment configuration moves partition mytopic1-0 to the /log/directory1 log directory.

Reassign partitions between log directories across multiple brokers

To reassign partitions between log directories across multiple brokers, change the broker ID specified in replicas and the appropriate any entry to an absolute path. For example:

{"topic":"mytopic1","partition":0,"replicas":[5,2],"log_dirs":["/log/directory1","any"]}

This reassignment configuration moves partition mytopic1-0 to /log/directory1 on broker 5.

Change partition assignment order (elect a new leader)

To change the ordering of the partition assignment list, change the order of the brokers in replicas. For example:

{"topic":"mytopic1","partition":0,"replicas":[2,1],"log_dirs":["any","any"]}

This reassignment configuration elects broker 2 as the new leader.

Increase the replication factor of a topic

To increase the replication factor of a topic, add one or more new brokers to replicas. For example:

{"topic":"mytopic1","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]}