kafka-consumer-groups
Learn how to use the kafka-consumer-groups
tool.
The kafka-consumer-groups
tool can be used to list all consumer groups,
describe a consumer group, delete consumer group info, or reset consumer group offsets. The
following topic gives an overview on how to describe or reset consumer group offsets.
Describe Offsets
This tool is primarily used for describing consumer groups and debugging any consumer offset issues, like consumer lag. The output from the tool shows the log and consumer offsets for each partition connected to the consumer group that is being described. You can see at a glance which consumers are current with their partition and which ones are lagging. From there, you can determine which partitions (and likely the corresponding brokers) are slow.
Using the tool on secure and unsecure clusters differs slightly. On secure clusters, you
have to use the command-config
option together with an appropriate property
file.
- Describing offsets on an unsecure cluster
-
Use the following command to describe offsets committed to Kafka:
kafka-consumer-groups --bootstrap-server [HOST]:9092 --describe --group [CONSUMER GROUP]
Output Example:GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER group1 topic1 0 1 3 2 test-consumer-group_postamac.local-1456198719410-29ccd54f-0
- Describing offsets on a secure cluster
- In order to describe offsets on a secure Kafka cluster, the consumer-groups tool has
to be run with the
command-config
option. Thecommand-config
option specifies the property file that contains the necessary configurations to run the tool on a secure cluster. Which properties are configured in this file depends on the security configuration of your cluster.
Resetting Offsets
You can use the --reset-offset
option to reset the offsets of a consumer
group to a particular value. The tool can be used to reset all offsets on all topics.
However, this is something you probably won’t ever want to do. Therefore, it is highly
recommended that you exercise caution when resetting offsets.
To reset offsets you need to define a scope, an execution option, and a scenario.
- Scope
- There are two supported scopes:
--topic
: Restricts the change to a specific topic or a specific set of partitions within a topic.--all-topics
: Executes the change for all topics.
- Execution option
- There are three execution options:
--dry-run --reset-offsets
: Default option. Displays which offsets will be reset if the process is executed.--execute --reset-offsets
: Executes the process.--export --reset-offsets
: Exports results in CSV format.
- Scenarios
- There are a number of supported scenarios. Scenarios control what value the offsets are reset to. Specifying a scenario is mandatory.
kafka-consumer-groups --dry-run --reset-offsets --bootstrap-server [HOST]:9092 --group [CONSUMER GROUP] --topic [TOPIC] --to-current