You can configure an external Prometheus service to scrape NiFi metrics for Cloudera DataFlow
deployments. To do that, you need to generate a password and add a job for each deployment
to your Prometheus configuration.
You have the DFAdmin user role for the
environment where you want to configure access for NiFi metrics
scraping.
In Cloudera DataFlow, from the Environments page, select the
Environment where you want to configure NiFi metrics scraping.
From the Actions menu select Access NiFi Metrics.
Depending on whether you are setting up access for the first time or updating
an existing one, select Initial configuration or
Manage existing.
Copy the generated password. The
username is nifi-metrics for all jobs, do
not change it.
Create a new job for each deployment where you want to perform
metrics scraping and add it to your Prometheus configuration.
Depending on your use case, either append it to an existing
configuration or you can create a new one. You can use the
provided Sample Prometheus scrape
configuration. Figure 1. Sample metrics configuration code snippet
scrape_configs:
- job_name: 'nifi-metrics-[***DEPLOYMENT-NAME***]'
scrape_interval: 15s
scheme: https
honor_labels: true
metrics_path: /dfx-[***DEPLOYMENT-NAME***]-ns/federate
basic_auth:
# Use ‘nifi-metrics’ as the username for all jobs.
[ username: nifi-metrics ]
[ password: [***GENERATED PASSWORD***] ]
params:
'match[]':
# This parameter is mandatory, because Cloudera's Prometheus instance also scrapes cadvisor and Prometheus itself.
- '{job="dfx-nifi-web"}'
static_configs:
- targets: ['https://dfx.qbllchii.xcu2-8y8x.dev.cldr.work']
You need to replace
[***DEPLOYMENT-NAME***] with the
encoded deployment name. (For example 'Some DataFlow
Deployment' is encoded as
'some-dataflow-deployment')
[***GENERATED PASSWORD***] with the
generated password.
Depending on your Prometheus setup, you may need to make
further additions to the job definition.
Add the newly created or updated configuration file to your
Prometheus service.
To turn on or off NiFi metrics scraping for all flow deployments
in an environment, toggle the Access NiFi
Metrics switch. Turning metrics scraping off
stops exposing the Prometheus endpoint.
If you need to reset the generated credentials, click
Regenerate Credentials. Copy the
generated password. The username is
nifi-metrics for all jobs, do not
change it. You will not be able to access the generated
credentials after closing the dialog box.
Do not forget to update your Prometheus
configuration with the regenerated credentials.
To add new jobs or remove existing ones, modify the Prometheus
configuration file.
Create a new job for each deployment where you want to
perform metrics scraping and add it to your Prometheus
configuration. Add it to the existing configuration. You
can use the provided Sample Prometheus scrape
configuration. Figure 2. Sample metrics configuration code
snippet
scrape_configs:
- job_name: 'nifi-metrics-[***DEPLOYMENT-NAME***]'
scrape_interval: 15s
scheme: https
honor_labels: true
metrics_path: /dfx-[***DEPLOYMENT-NAME***]-ns/federate
basic_auth:
# Use ‘nifi-metrics’ as the username for all jobs.
[ username: nifi-metrics ]
[ password: [***GENERATED PASSWORD***] ]
params:
'match[]':
# This parameter is mandatory, because Cloudera's Prometheus instance also scrapes cadvisor and Prometheus itself.
- '{job="dfx-nifi-web"}'
static_configs:
- targets: ['https://dfx.qbllchii.xcu2-8y8x.dev.cldr.work']
You need to replace
[***DEPLOYMENT-NAME***] with
the encoded deployment name. (For example 'Some
DataFlow Deployment' is encoded as
'some-dataflow-deployment')
[***GENERATED PASSWORD***]
with the generated password.
Depending on your Prometheus setup, you may need to
make further additions to the job definition.
Add the updated configuration file to your Prometheus
service.