Cloudera Docs

Extract Kafka broker ID

You must extract the Kafka broker IDs before you upgrade the HDP cluster from 2.6.5 to HDP intermediate bits.

You must extract the Kafka broker IDs manually from the HDP 2.6.5 cluster. When you migrate to Cloudera Manager, you must manually enter the broker IDs to the real values in Kafka.

Procedure 1

To extract the broker IDs from the HDP 2.6.5 cluster, use the following commands: This procedure helps you only if the Kafka service is not stopped.
  • For Unsecure environments:sh /usr/hdp/current/kafka-broker/bin/kafka-broker-api-versions.sh --bootstrap-server <host:port> | grep "id" | awk -F'[=:]' '{print $1,$3}' | awk '{print $1,$2}' > /tmp/kafka-broker-ids.ini
  • For Secure environments: sh /usr/hdp/current/kafka-broker/bin/kafka-broker-api-versions.sh --bootstrap-server <host:port> --command-config <client.config> | grep "id" | awk -F'[=:]' '{print $1,$3}' | awk '{print $1,$2}' > /tmp/kafka-broker-ids.ini

  • For example, sh /usr/hdp/current/kafka-broker/bin/kafka-broker-api-versions.sh --bootstrap-server c6111-node2.coelab.cloudera.com:6667 --command-config <client.config> | grep "id" | awk -F'[=:]' '{print $1,$3}' | awk '{print $1,$2}' > /tmp/kafka-broker-ids.ini

    where <client.config> contains environment specific Kafka client security configuration for connecting the source Kafka cluster. For more information on configuring Kafka client, see Kafka documentation and Kafka client configuration.

Procedure 2

  1. Create kafka-broker-ids.ini file
  2. Navigate to each Kafka broker host > $log.dirs/meta.properties
  3. Pick up the broker.id value.
  4. Copy the hostname broker.id to kafka-broker-ids.ini file

    An example of the file format:

    ctr-e153-xxxxx-xxxx71.cloudera.site 1001

    ctr-e153-xxxxx-xxxx72.cloudera.site 1002

    ctr-e153-xxxxx-xxxx73.cloudera.site 1003