Kafka Metrics

Kafka uses Yammer metrics to record internal performance measurements. The metrics are exposed via Java Management Extensions (JMX) and can be read with a JMX console.

Metrics Categories

There are metrics available in the various components of Kafka. In addition, there are some metrics specific to how Cloudera Manager and Kafka interact. This table has pointers to both the Apache Kafka metrics names and the Cloudera Manager metric names.

Metrics by Category
Category Cloudera Manager Metrics Doc Apache Kafka Metrics Doc

Cloudera Manager

Kafka Service

Base Metrics  
Broker Broker Metrics

Broker Topic Metrics

Replica Metrics

Broker

Common

Producer/Consumer

 

Client

Client-to-Broker

Producer  

Producer

Producer Sender

Consumer  

Consumer Group

Consumer Fetch

Mirror Maker Mirror Maker Metrics Same as Producer or Consumer tables

Viewing Metrics

Cloudera Manager records most of these metrics and makes them available via Chart Builder.

Because Cloudera Manager cannot track metrics on any clients (that is, producer or consumer), you may wish to use an alternative JMX console program to check metrics. There are several JMX console options:

Building Cloudera Manager Charts with Kafka Metrics

The Charts edit menu looks like a small pencil icon in the Charts page of the Cloudera Manager console. From there, choose Add from Chart Builder and enter a query for the appropriate metric.

SELECT
    metric
WHERE
    filter

Some specific examples of queries for Cloudera Metrics are:

Controllers across all brokers
This chart shows the active controller across all brokers. It is useful for checking active controller status (should be one at any given time, transitions should be fast).
SELECT
    kafka_active_controller
WHERE
    roleType=KAFKA_BROKER
Network idle rate
>Chart showing the network processor idle rate across all brokers. If idle time is always zero, then probably the num.network.threads property may need to be increased.
SELECT
    kafka_network_processor_avg_idle_rate
WHERE
    roleType=KAFKA_BROKER
Partitions per broker
Chart showing the number of partitions per broker. It is useful for detecting partition imbalances early.
SELECT
    kafka_partitions
WHERE
    roleType=KAFKA_BROKER
Partition activity
Chart tracking partition activity on a single broker.
SELECT
    kafka_partitions, kafka_under_replicated_partitions
WHERE
    hostname=host1.domain.com
Mirror Maker activity
Chart for tracking Mirror Maker behavior. Since Mirror Maker has one or more consumers and a single producer, most consumer or metrics should be usable with this query.
SELECT
    producer or consumer metric
WHERE
    roleType=KAFKA_MIRROR_MAKER