Extract Kafka broker ID

You must extract the Kafka broker IDs before you upgrade the HDP cluster from 3.1.5.x to HDP intermediate bits.

You must extract the Kafka broker IDs manually from the HDP 3.1.5.x cluster. When you migrate to Cloudera Manager, you must manually enter the broker IDs to the real values in Kafka.

Procedure 1

To extract the broker IDs from the HDP 3.1.5 cluster, use the following commands: This procedure helps you only if the Kafka service is not stopped.

For Unsecure environments:sh /usr/hdp/current/kafka-broker/bin/kafka-broker-api-versions.sh --bootstrap-server <host:port> | grep "id" | awk -F'[=:]' '{print $1,$3}' | awk '{print $1,$2}' > /tmp/kafka-broker-ids.ini
For Secure environments: sh /usr/hdp/current/kafka-broker/bin/kafka-broker-api-versions.sh --bootstrap-server <host:port> --command-config <client.config> | grep "id" | awk -F'[=:]' '{print $1,$3}' | awk '{print $1,$2}' > /tmp/kafka-broker-ids.ini
For example, sh /usr/hdp/current/kafka-broker/bin/kafka-broker-api-versions.sh --bootstrap-server c6111-node2.coelab.cloudera.com:6667 --command-config <client.config> | grep "id" | awk -F'[=:]' '{print $1,$3}' | awk '{print $1,$2}' > /tmp/kafka-broker-ids.ini

where <client.config> contains environment specific Kafka client security configuration for connecting the source Kafka cluster. For more information on configuring Kafka client, see Kafka documentation and Kafka client configuration.
note
This command generates the kafka-broker-ids.ini file. Thekafka-broker-ids.ini file must be manually copied to $am2cm-1.2.0.0-xx/conf/ path before running the AM2CM tool.

Procedure 2

Create kafka-broker-ids.ini file
Navigate to each Kafka broker host > $log.dirs/meta.properties
Pick up the broker.id value.
Copy the hostname broker.id to kafka-broker-ids.ini file
An example of the file format:
ctr-e153-xxxxx-xxxx71.cloudera.site 1001
ctr-e153-xxxxx-xxxx72.cloudera.site 1002
ctr-e153-xxxxx-xxxx73.cloudera.site 1003