Cloudera Docs

Tool usage

Learn how to reassign partitions with the kafka-reassign-partitions-tool.

  1. Create a topics-to-move JSON file that specifies the topics you want to reassign.
    Use the following format:
    {"topics":  [{"topic": "mytopic1"},
             {"topic": "mytopic2"}],
 "version":1
}
  2. Generate the content for the reassignment configuration JSON with the following command:
    kafka-reassign-partitions --zookeeper [***HOSTNAME***]:[***PORT***]/[***ZOOKEEPER KAFKA CHROOT***] --topics-to-move-json-file [***TOPICS TO MOVE.JSON***] --broker-list [***BROKER 1***], [***BROKER 2***] --generate

    Running the command lists the distribution of partition replicas on your current brokers followed by a proposed partition reassignment configuration. 

    Current partition replica assignment
{"version":1,
 "partitions":
   [{"topic":"mytopic2","partition":1,"replicas":[2,3],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},
    {"topic":"mytopic2","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":2,"replicas":[3,1],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":1,"replicas":[2,3],"log_dirs":["any","any"]}]
}

Proposed partition reassignment configuration

{"version":1,
 "partitions":
   [{"topic":"mytopic1","partition":0,"replicas":[4,5],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":2,"replicas":[4,5],"log_dirs":["any","any"]},
    {"topic":"mytopic2","partition":1,"replicas":[4,5],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":1,"replicas":[5,4],"log_dirs":["any","any"]},
    {"topic":"mytopic2","partition":0,"replicas":[5,4],"log_dirs":["any","any"]}]
}
    In this example, the tool proposed a configuration which reassigns existing partitions on broker 1, 2, and 3 to brokers 4 and 5.
  3. Copy and paste the proposed partition reassignment configuration into an empty JSON file.
  4. Review, and if required, modify the suggested reassignment configuration.
  5. Save the file.
  6. Start the redistribution process with the following command: 
    kafka-reassign-partitions --zookeeper [***HOSTNAME***]:[***PORT***]/[***ZOOKEEPER KAFKA CHROOT***]  --reassignment-json-file [***REASSIGNMENT CONFIGURATION.JSON***] --bootstrap-server [***HOSTNAME***]:[***PORT***] --execute
    The tool prints a list containing the original replica assignment and a message that reassignment has started. Example output: 
    Current partition replica assignment

{"version":1,
 "partitions":
   [{"topic":"mytopic2","partition":1,"replicas":[2,3],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},
    {"topic":"mytopic2","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":2,"replicas":[3,1],"log_dirs":["any","any"]},
    {"topic":"mytopic1","partition":1,"replicas":[2,3],"log_dirs":["any","any"]}]
}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
  7. Verify the status of the reassignment with the following command: 
    kafka-reassign-partitions --zookeeper [***HOSTNAME***]:[***PORT***]/[***ZOOKEEPER KAFKA CHROOT***] --reassignment-json-file [***REASSIGNMENT CONFIGURATION.JSON***] --bootstrap-server [***HOSTNAME***]:[***PORT***] --verify
    The tool prints the reassignment status of all partitions. 
    Status of partition reassignment: 
Reassignment of partition mytopic2-1 completed successfully
Reassignment of partition mytopic1-0 completed successfully
Reassignment of partition mytopic2-0 completed successfully
Reassignment of partition mytopic1-2 completed successfully
Reassignment of partition mytopic1-1 completed successfully
Partitions are reassigned.