Configuring Flow Management clusters to hot load custom NARs

Hot loading custom NARs allows you to store NiFi custom processor NAR files in an external folder and load them dynamically from that location for use on the NiFi canvas. This makes it easy to add or update custom processors without having to restart your Flow Management cluster.

Collect your Kerberos information.

In Cloudera Management Console, click Environments.
From the Actions drop-down, click Manage Access.
Create a Machine User with the EnvironmentUser Resource Role and set its Workload password.
note

It is best practice to have a dedicated Machine User in this role. This Machine User is in charge of accessing the object store resources for NiFi, having a dedicated user ensures that there is least privilege access and separation of duties.
Note down the user you create there and password, for your safety valves.

You have created an ID Broker mapping.

Access IDBroker mappings.
1. To access IDBroker mappings in your environment, click Actions > Manage Access.
2. Choose the IDBroker Mappings tab where you can provide mappings for users or groups and click Edit.
Add your Cloudera Machine User and the corresponding AWS role that provides write access to your folder in your S3 bucket to the Current Mappings section by clicking the blue + sign.
note
You can get the AWS IAM role ARN from the Roles Summary page in AWS and can copy it into the IDBroker role field. The selected AWS IAM role must have a trust policy allowing IDBroker to assume this role.
Click Save and Sync.

You can configure NiFi to hot load custom NARs in all supported cloud providers.
You can only access resources in the object store of the cloud provider in which your environment is running. But you can have a development and a production environment, for example, and configure both environments to access the same location to hot load custom NARs
It is possible to define multiple locations if desired by repeating the configuration properties with a different “[**custom-name**]” for each location. But all locations should be in the object store of the underlying cloud provider of where the NiFi cluster is located.
NiFi polls for new NARs in the configured location(s) every 5 minutes.

In Cloudera Management Console, select Data Hub Clusters, then go to the Flow Management cluster, then NiFi Service.
Click Configuration.
Search for Safety Valve and locate NiFi Node Advanced Configuration Snippet (Safety Valve) for staging/nifi.properties.xml.

Add the following new properties:

Name: nifi.nar.library.provider.[**custom-name**].implementation
Example: nifi.nar.library.provider.narstorage1.implementation
Value: org.apache.nifi.nar.hadoop.HDFSNarProvider

Path to the configuration files
Name: nifi.nar.library.provider.[**custom-name**].resources
Example: nifi.nar.library.provider.narstorage1.resources
Value, by default: /etc/hadoop/conf.cloudera.core_settings/core-site.xml

Storage location for the NARs (the bucket name in S3 for example)
Name: nifi.nar.library.provider.[**custom-name**].storage.location
Example: nifi.nar.library.provider.narstorage1.storage.location
Value: s3a://my-custom-nars-bucket

Directory library with path to the NARs relative to the storage location
Name: nifi.nar.library.provider.[**custom-name**].source.directory
Example: nifi.nar.library.provider.narstorage1.source.directory
Value: /myNARs

Workload username and password of the machine user to use to access the resources

Name: nifi.nar.library.provider.[**custom-name**].kerberos.principal
Example: nifi.nar.library.provider.narstorage1.kerberos.principal
Value: srv_my_s3_nar_user

Name: nifi.nar.library.provider.[**custom-name**].kerberos.password
Example: nifi.nar.library.provider.narstorage1.kerberos.password
Value: myPassword123

note

Cloudera recommends that you use a workload user name and password dedicated solely to accessing custom NARs, with permissions limited to the designated object store location.

Alternatively, you can also use the keytab file. If using a keytab, ensure it is available on all NiFi nodes with proper file permissions for the nifi user. This approach may be problematic when scaling a cluster.

Name: nifi.nar.library.provider.[**custom-name**].kerberos.keytab
Example: nifi.nar.library.provider.narstorage1.kerberos.keytab
Value: /path/to/my/file.keytab

Restart the NiFi service in the Cloudera Manager.
Refresh the NiFi UI.

You can now drag a new processor onto NiFi canvas and search for your custom processor.