Learn how to use the
kafka-consumer-groups tool can be used to list all consumer groups,
describe a consumer group, delete consumer group info, or reset consumer group offsets. The
following topic gives an overview on how to describe or reset consumer group offsets.
This tool is primarily used for describing consumer groups and debugging any consumer offset issues, like consumer lag. The output from the tool shows the log and consumer offsets for each partition connected to the consumer group that is being described. You can see at a glance which consumers are current with their partition and which ones are lagging. From there, you can determine which partitions (and likely the corresponding brokers) are slow.
Using the tool on secure and unsecure clusters differs slightly. On secure clusters, you
have to use the
command-config option together with an appropriate property
- Describing offsets on an unsecure cluster
Use the following command to describe offsets committed to Kafka:
kafka-consumer-groups --bootstrap-server [HOST]:9092 --describe --group [CONSUMER GROUP]Output Example:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER group1 topic1 0 1 3 2 test-consumer-group_postamac.local-1456198719410-29ccd54f-0
- Describing offsets on a secure cluster
- In order to describe offsets on a secure Kafka cluster, the consumer-groups tool has
to be run with the
command-configoption specifies the property file that contains the necessary configurations to run the tool on a secure cluster. Which properties are configured in this file depends on the security configuration of your cluster.
This example shows what properties you have to set when both Kerberos and TLS/SSL are configured.
exclude.internal.topics=false security.protocol = SASL_SSL sasl.kerberos.service.name = kafka ssl.truststore.location = /var/private/ssl/kafka.client.truststore.jks ssl.truststore.password = test1234To describe offsets do the following:
- Pass the
jaas.conffile location as a JVM parameter.
export KAFKA_OPTS='-Djava.security.auth.login.config=[PATH TO JAAS.CONF]
- Create a
Use the example above. Make changes as necessary.
- Run the tool with the
kafka-consumer-groups --bootstrap-server [HOST]:9093 --describe --command-config client.properties --group [CONSUMER GROUP]
- Pass the
You can use the
--reset-offset option to reset the offsets of a consumer
group to a particular value. The tool can be used to reset all offsets on all topics.
However, this is something you probably won’t ever want to do. Therefore, it is highly
recommended that you exercise caution when resetting offsets.
To reset offsets you need to define a scope, an execution option, and a scenario.
- There are two supported scopes:
--topic: Restricts the change to a specific topic or a specific set of partitions within a topic.
--all-topics: Executes the change for all topics.
- Execution option
- There are three execution options:
--dry-run --reset-offsets: Default option. Displays which offsets will be reset if the process is executed.
--execute --reset-offsets: Executes the process.
--export --reset-offsets: Exports results in CSV format.
- There are a number of supported scenarios. Scenarios control what value the offsets are reset to. Specifying a scenario is mandatory.
kafka-consumer-groups --dry-run --reset-offsets --bootstrap-server [HOST]:9092 --group [CONSUMER GROUP] --topic [TOPIC] --to-current