Metrics

Learn about Kafka metrics and how to view them.

Kafka uses Yammer metrics to record internal performance measurements. The metrics are exposed via Java Management Extensions (JMX) and can be read with a JMX console.

Metrics Categories

There are metrics available in the various components of Kafka. In addition, there are some metrics specific to how Cloudera Manager and Kafka interact. This table has pointers to both the Apache Kafka metrics names and the Cloudera Manager metric names.

Table 1. Metrics by Category
Category Cloudera Manager Metrics Doc Apache Kafka Metrics Doc

Cloudera Manager

Kafka Service

Base Metrics
Broker Broker Metrics, Broker Topic Metrics, Replica Metrics, Broker Topic Partition Metrics Broker

Common

Producer/Consumer

Client

Client-to-Broker

Producer Producer Metrics

Producer

Producer Sender

Consumer Consumer Metrics,Consumer Group Metrics

Consumer Group

Consumer Fetch

Mirror Maker Mirror Maker Metrics Same as Producer or Consumer tables
Note the following about Cloudera Manager metrics:
Broker Topic Partitions Metrics:
Required for Streams Messaging Manager. These metrics have to be specifically enabled within the host's configuration.
Producer Metrics:
Required for Streams Messaging Manager. These metrics have to be specifically enabled within Kafka's configuration and are different from the metrics available on client instances.
Consumer and Consumer Groups Metrics
Generated by and required for Streams Messaging Manager. These metrics are different from the metrics available on client instances.

Viewing Metrics

Cloudera Manager records most of these metrics and makes them available via Chart Builder.

Because Cloudera Manager cannot track metrics on any clients (that is, producer or consumer), you may wish to use an alternative JMX console program to check metrics. There are several JMX console options, for example:

  • JConsole, which comes bundled with the JDK.
  • VisualVM, with the MBeans plugin.