Configuring Kafka Connect for Prometheus monitoring

To monitor Kafka Connect with Prometheus, you must configure your Kafka Connect cluster to expose the necessary metric endpoints that integrate with your Prometheus deployment. This is done by configuring the metricsConfig property in your KafkaConnect resource.

By default a Kafka Connect cluster you deploy with a KafkaConnect resource does not expose metrics that Prometheus can scrape. In order to use Prometheus to monitor your Kafka Connect cluster, you must enable and expose these metrics. This is done by adding a metricsConfig property to the spec of your KafkaConnect resource.

Specifying metricsConfig in the KafkaConnect resource enables the Prometheus JMX Exporter which exposes metrics through a HTTP endpoint. The metrics are exposed on port 9094. The metricsConfig property can reference a ConfigMap that holds your JMX metrics configuration or will include the metrics configurations in-line. The following steps demonstrate the configuration by referencing a ConfigMap.

A Prometheus deployment that can connect to the metric endpoints of the Kafka connect cluster running in the Kubernetes environment is required. Any properly configured Prometheus deployment can be used to monitor Kafka Connect. You can find additional information and examples on Prometheus setup in the Strimzi documentation.

Create a ConfigMap with JMX metrics configuration for Kafka Connect.
```
kind: ConfigMap
apiVersion: v1
metadata:
  name: connect-metrics
  labels:
    app: strimzi
data:
  metrics-config.yml: |
    [***KAFKA-CONNECT-METRICS-CONFIGURATION***]
```
Replace [***KAFKA-CONNECT-METRICS-CONFIGURATION***] with your JMX Prometheus metrics configuration.

Update your KafkaConnect resource with a metricsConfig property. The property needs to reference the ConfigMap you created in 1.

#...
kind: KafkaConnect
spec:
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        name: connect-metrics
        key: metrics-config.yml

Configure Prometheus and specify alert rules to start scraping metrics from the Kafka Connect pods. You can find an example rules file (prometheus-rules.yaml) as well as various other configuration examples on the Cloudera Archive. Examples related to Prometheus are located in the /csm-operator/1.1/examples/metrics directory.
Review Cloudera recommendations on what alerts and metrics to configure. See Monitoring with Prometheus.