Prometheus properties configuration
Learn the properties that you need to configure in the prometheus.yml
file before you start using the Prometheus metric store for Streams Messaging Manager
(SMM).
Configure the following properties in the prometheus.yml
file:
- Set the
scrape_interval
property value to 60 seconds in theprometheus.yml
file.scrape_interval: 60s
- Prometheus automatically assigns the instance label to metrics. However, in case partition
leader change reassignment happens, it means an excessive amount of metrics being created.
This property is recommended for large clusters.
metric_relabel_configs: - source_labels: [__name__] regex: ^(broker_producer_messagesinpersec_total|topic_partition_messagesinpersec_total|topic_partition_bytesinpersec_total|topic_partition_bytesoutpersec_total)$ target_label: instance replacement: 'no-instance'
- Set Kafka host and metrics port
values.
You can find the metric port used by Kafka in .['luigi-1.luigi.root.hwx.site:24042','luigi-2.luigi.root.hwx.site:24042','luigi-3.luigi.root.hwx.site:24042']
- Set Kafka Connect host (deployed on same hosts as Kafka) and the metrics port
values.
['luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087']
The metrics port used by Kafka Connect is set in Jetty Metrics Port. Use Secure Jetty Metrics Port if the Kafka Connect metrics reporter uses TLS/SSL, otherwise, use Jetty Metrics Port. The value of
and28087
in this example is the default secure port. - Set Prometheus node exporter host (deployed on same hosts as Kafka) and the Prometheus node
exporter metrics port
values.
['luigi-1.luigi.root.hwx.site:9100','luigi-2.luigi.root.hwx.site:9100','luigi-3.luigi.root.hwx.site:9100']
update=true
parameter should be only used by Prometheus. Querying Kafka Prometheus endpoint with this flag set to true updates the internal metrics cache for stateful metrics.update: ['true']
- If Basic Authentication is enabled for the Kafka Connect metrics reporter, add the
following parameters:
Ensure that you specify the username and password that is configured for the Kafka Connect metrics reporter. You can find the username and password in and Jetty Metrics Password.basic_auth: username: 'email@username.me' password: 'password'
Configuration example
The following is an example of a Prometheus configuration YAML. You can use this example as a template and make changes as necessary. Ensure that you replace host and port values as well as the Basic Authentication credentials with your own values. This example uses the default ports.
# my global config
global:
scrape_interval: 60s
scrape_timeout: 55s
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
- job_name: 'kafka'
metrics_path: '/api/prometheus-metrics'
params:
update: ['true']
static_configs:
- targets: ['luigi-1.luigi.root.hwx.site:24042','luigi-2.luigi.root.hwx.site:24042','luigi-3.luigi.root.hwx.site:24042']
metric_relabel_configs:
- source_labels: [__name__]
regex: ^(broker_producer_messagesinpersec_total|topic_partition_messagesinpersec_total|topic_partition_bytesinpersec_total|topic_partition_bytesoutpersec_total)$
target_label: instance
replacement: 'no-instance'
- job_name: 'kafka_connect'
metrics_path: '/prometheus-metrics'
basic_auth:
username: 'email@username.me'
password: 'password'
static_configs:
- targets: ['luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087']
- job_name: 'system_metrics'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['luigi-1.luigi.root.hwx.site:9100','luigi-2.luigi.root.hwx.site:9100','luigi-3.luigi.root.hwx.site:9100']
['luigi-1.luigi.root.hwx.site:9100','luigi-2.luigi.root.hwx.site:9100','luigi-3.luigi.root.hwx.site:9100']
= Kafka host + Node exporter metrics port['luigi-1.luigi.root.hwx.site:24042','luigi-2.luigi.root.hwx.site:24042','luigi-3.luigi.root.hwx.site:24042']
= Kafka host + HTTP Metric Report Port['luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087']
= Kafka Connect host + Secure Jetty Metrics Port