Prometheus properties configuration

Learn the properties that you need to configure in the prometheus.yml file before you start using the Prometheus metric store for Streams Messaging Manager (SMM).

Configure the following properties in the prometheus.yml file:

Set the scrape_interval property value to 60 seconds in the prometheus.yml file.
```
scrape_interval: 60s
```

Prometheus automatically assigns the instance label to metrics. However, in case partition leader change reassignment happens, it means an excessive amount of metrics being created. This property is recommended for large clusters.

    metric_relabel_configs:
    - source_labels: [__name__]
      regex: ^(broker_producer_messagesinpersec_total|topic_partition_messagesinpersec_total|topic_partition_bytesinpersec_total|topic_partition_bytesoutpersec_total)$
      target_label: instance
      replacement: 'no-instance'

Set Kafka hosts and metrics port values.
```
['luigi-1.luigi.root.hwx.site:24042','luigi-2.luigi.root.hwx.site:24042','luigi-3.luigi.root.hwx.site:24042']
```
The metrics port you must set here is the value of the kafka.http.metrics.port Kafka property. You can find the property and its value in Cloudera Manager > Kafka service > Configuration.
Set Kafka Connect hosts (deployed on same hosts as Kafka) and metrics port values.
```
['luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087']
```
The metric port you must set here is the value of the metrics.jetty.server.port or metrics.jetty.server.secureport Kafka properties. You can find these properties and their values in Cloudera Manager > Kafka service > Configuration.
Use metrics.jetty.server.secureport if the Kafka Connect metrics reporter uses TLS/SSL, otherwise, use metrics.jetty.server.port. The value of 28087 in this example is the default secure port.
Set Prometheus node exporter hosts (deployed on same hosts as Kafka) and Prometheus node exporter metrics port values.
```
['luigi-1.luigi.root.hwx.site:9100','luigi-2.luigi.root.hwx.site:9100','luigi-3.luigi.root.hwx.site:9100']
```
update=true parameter should be only used by Prometheus. Querying Kafka Prometheus endpoint with this flag set to true updates the internal metrics cache for stateful metrics.
```
update: ['true']
```

For example, configure Prometheus with the following YAML file:

# my global config
global:
  scrape_interval: 60s 
  scrape_timeout: 55s

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'kafka'
    
    metrics_path: '/api/prometheus-metrics'
    params:
      update: ['true']
    
    static_configs:
    - targets: ['luigi-1.luigi.root.hwx.site:24042','luigi-2.luigi.root.hwx.site:24042','luigi-3.luigi.root.hwx.site:24042']

  - job_name: 'kafka_connect'

    metrics_path: '/prometheus-metrics'
    static_configs:
    - targets: ['luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087']

  - job_name: 'system_metrics'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['luigi-1.luigi.root.hwx.site:9100','luigi-2.luigi.root.hwx.site:9100','luigi-3.luigi.root.hwx.site:9100']
  - job_name: 'kafka'
    metric_relabel_configs:
    - source_labels: [__name__]
      regex: ^(broker_producer_messagesinpersec_total|topic_partition_messagesinpersec_total|topic_partition_bytesinpersec_total|topic_partition_bytesoutpersec_total)$
      target_label: instance
      replacement: 'no-instance'

Where,

['luigi-1.luigi.root.hwx.site:9100','luigi-2.luigi.root.hwx.site:9100','luigi-3.luigi.root.hwx.site:9100'] = Kafka host + Node exporter metrics port
['luigi-1.luigi.root.hwx.site:24042','luigi-2.luigi.root.hwx.site:24042','luigi-3.luigi.root.hwx.site:24042'] = Kafka host + kafka.http.metrics.port
['luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087'] = Kafka Connect host + metrics.jetty.server.secureport