Configuring Kafka Connect for Prometheus monitoring
To monitor Kafka Connect with Prometheus, you must configure your Kafka Connect cluster to expose the necessary metric endpoints that integrate with your Prometheus deployment. This is done by configuring the metricsConfig property in your KafkaConnect resource.
By default a Kafka Connect cluster you deploy with a KafkaConnect
resource does not expose metrics that Prometheus can scrape. In order to use
Prometheus to monitor your Kafka Connect cluster, you must enable and expose these
metrics. This is done by adding a metricsConfig
property to the
spec of your KafkaConnect resource.
Specifying metricsConfig
in the KafkaConnect
resource enables the Prometheus JMX Exporter which exposes metrics through a HTTP
endpoint. The metrics are exposed on port 9094. The metricsConfig
property can reference a ConfigMap that holds your JMX metrics
configuration or will include the metrics configurations in-line. The following
steps demonstrate the configuration by referencing a
ConfigMap.
- Configure Prometheus and specify alert rules to start scraping metrics from the
Kafka Connect pods. You can find an example rules file
(
prometheus-rules.yaml
) as well as various other configuration examples on the Cloudera Archive. Examples related to Prometheus are located in the /csm-operator/1.1/examples/metrics directory. - Review Cloudera recommendations on what alerts and metrics to configure. See Monitoring with Prometheus.