Configuring Kafka for Prometheus monitoring

To monitor your Kafka cluster with Prometheus, you must configure your Kafka cluster to expose the necessary metric endpoints that integrate with your Prometheus deployment. This is done by configuring metricsConfig properties for components in your Kafka resource.

By default cluster components deployed with your Kafka resource do not expose metrics that Prometheus can scrape. In order to use Prometheus to monitor your Kafka cluster, you must enable and expose these metrics. This is done by adding a metricsConfig property to the spec of each cluster component in your Kafka resource.

Specifying metricsConfig in the Kafka resource enables the Prometheus JMX Exporter which exposes metrics through a HTTP endpoint. The metrics are exposed on port 9094. The metricsConfig property can reference a ConfigMap that holds your JMX metrics configuration or will include the metrics configurations in-line. The following steps demonstrate the configuration by referencing a ConfigMap.

A Prometheus deployment that can connect to the metric endpoints of the Kafka cluster running in the Kubernetes environment is required. Any properly configured Prometheus deployment can be used to monitor Kafka. You can find additional information and examples on Prometheus setup in the Strimzi documentation.

Create a ConfigMap with JMX metrics configuration for both Kafka and ZooKeeper.
```
kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
data:
  kafka-metrics-config.yml: |
    [***KAFKA METRICS CONFIGURATION***]
  zookeeper-metrics-config.yml: |
    [***ZOOKEEPER METRICS CONFIGURATION***]
```
Replace [***KAFKA METRICS CONFIGURATION***] and [***ZOOKEEPER METRICS CONFIGURATION***] with your JMX Prometheus metrics configurations.

Update your Kafka resource with metricsConfig property.

Add metricsConfig to the spec of both Kafka and ZooKeeper. The property needs to reference the ConfigMap you created in Step 1.

#...
kind: Kafka
spec:
  kafka:
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml

Configure Prometheus and specify alert rules to start scraping metrics from the ZooKeeper and Kafka pods. You can find an example rules file (prometheus-rules.yaml) as well as various other configuration examples on the Cloudera Archive. Examples related to Prometheus are located in the /csm-operator/1.2/examples/metrics directory.
Review Cloudera recommendations on what alerts and metrics to configure. See Monitoring with Prometheus.