Cloudera Docs

kafka-consumer-groups

Learn how to use the kafka-consumer-groups tool.

The kafka-consumer-groups tool can be used to list all consumer groups, describe a consumer group, delete consumer group info, or reset consumer group offsets. The following topic gives an overview on how to describe or reset consumer group offsets.

Describe Offsets

This tool is primarily used for describing consumer groups and debugging any consumer offset issues, like consumer lag. The output from the tool shows the log and consumer offsets for each partition connected to the consumer group that is being described. You can see at a glance which consumers are current with their partition and which ones are lagging. From there, you can determine which partitions (and likely the corresponding brokers) are slow.

Using the tool on secure and unsecure clusters differs slightly. On secure clusters, you have to use the command-config option together with an appropriate property file.

Describing offsets on an unsecure cluster
Use the following command to describe offsets committed to Kafka:
kafka-consumer-groups --bootstrap-server [HOST]:9092 --describe --group [CONSUMER GROUP]
Output Example:
GROUP   TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     OWNER
group1   topic1     0          1               3               2       test-consumer-group_postamac.local-1456198719410-29ccd54f-0
Describing offsets on a secure cluster
In order to describe offsets on a secure Kafka cluster, the consumer-groups tool has to be run with the command-config option. The command-config option specifies the property file that contains the necessary configurations to run the tool on a secure cluster. Which properties are configured in this file depends on the security configuration of your cluster.
Example client.properties file:
exclude.internal.topics=false
security.protocol = SASL_SSL
sasl.kerberos.service.name = kafka
ssl.truststore.location = /var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password = test1234
This example shows what properties you have to set when both Kerberos and TLS/SSL are configured.
To describe offsets do the following:
  1. Pass the jaas.conf file location as a JVM parameter.
    export KAFKA_OPTS='-Djava.security.auth.login.config=[PATH TO JAAS.CONF]
  2. Create a client.properties file.

    Use the example above. Make changes as necessary.

  3. Run the tool with the command-config option.
    kafka-consumer-groups --bootstrap-server [HOST]:9093 --describe --command-config client.properties --group [CONSUMER GROUP]

Resetting Offsets

You can use the --reset-offset option to reset the offsets of a consumer group to a particular value. The tool can be used to reset all offsets on all topics. However, this is something you probably won’t ever want to do. Therefore, it is highly recommended that you exercise caution when resetting offsets.

To reset offsets you need to define a scope, an execution option, and a scenario.

Scope
There are two supported scopes:
  • --topic: Restricts the change to a specific topic or a specific set of partitions within a topic.
  • --all-topics: Executes the change for all topics.
Execution option
There are three execution options:
  • --dry-run --reset-offsets: Default option. Displays which offsets will be reset if the process is executed.
  • --execute --reset-offsets : Executes the process.
  • --export --reset-offsets : Exports results in CSV format.
Scenarios
There are a number of supported scenarios. Scenarios control what value the offsets are reset to. Specifying a scenario is mandatory.
  • --to-datetime
  • --by-period
  • --to-earliest
  • --to-latest
  • --shift-by
  • --from-file
  • --to-current
Example: 
kafka-consumer-groups --dry-run --reset-offsets --bootstrap-server [HOST]:9092 --group [CONSUMER GROUP] --topic [TOPIC] --to-current