Configuring Flow Management clusters to hot load custom NARs

Hot loading custom NiFi archives (NARs) allows you to store NiFi custom processor NAR files in an external folder and load them dynamically from that location for use on the NiFi canvas. This makes it easy to add or update custom processors without having to restart your Flow Management cluster.

Collect your Kerberos information.

In Cloudera Management Console, click Environments.
From the Actions drop-down, click Manage Access.
Create a Machine User with the EnvironmentUser Resource Role and set its Workload password.
note

It is best practice to have a dedicated Machine User in this role. This Machine User is in charge of accessing the object store resources for NiFi, having a dedicated user ensures that there is least privilege access and separation of duties.
Note down the user you create there and password, for your safety valves.

Create an ID Broker mapping.

Access IDBroker mappings.
1. To access IDBroker mappings in your environment, click Actions > Manage Access.
2. Choose the IDBroker Mappings tab where you can provide mappings for users or groups and click Edit.
Add your Cloudera Machine User and the corresponding AWS role that provides write access to your folder in your S3 bucket to the Current Mappings section by clicking the blue + sign.
note
You can get the AWS IAM role ARN from the Roles Summary page in AWS and can copy it into the IDBroker role field. The selected AWS IAM role must have a trust policy allowing IDBroker to assume this role.
Click Save and Sync.

You can configure NiFi to hot load custom NARs in all supported cloud providers.
You can only access resources in the object store of the cloud provider in which your environment is running. But you can have a development and a production environment, for example, and configure both environments to access the same location to hot load custom NARs
It is possible to define multiple locations if desired by repeating the configuration properties with a different “[**custom-name**]” for each location. But all locations should be in the object store of the underlying cloud provider of where the NiFi cluster is located.
NiFi polls for new NARs in the configured location(s) every 5 minutes.

In Cloudera Management Console, select Data Hub Clusters, then go to the Flow Management cluster, then NiFi Service.
Click Configuration.
Search for Safety Valve and locate NiFi Node Advanced Configuration Snippet (Safety Valve) for staging/nifi.properties.xml.
Add the following new properties:
- nifi.nar.library.provider.[**custom-name**].implementation
  - Example: nifi.nar.library.provider.narstorage1.implementation
  - Value: org.apache.nifi.nar.hadoop.HDFSNarProvider
  - Purpose: Defines the Java class that implements the NAR provider.
- nifi.nar.library.provider.[**custom-name**].resources
  - Example: nifi.nar.library.provider.narstorage1.resources
  - Value: /etc/hadoop/conf.cloudera.core_settings/core-site.xml
  - Purpose: Shows the path to the configuration files.
- nifi.nar.library.provider.[**custom-name**].storage.location
  - Example: nifi.nar.library.provider.narstorage1.storage.location
  - Value: s3a://my-custom-nars-bucket
  - Purpose: Shows the storage location for the NARs (the bucket name in S3 for example).
- nifi.nar.library.provider.[**custom-name**].source.directory
  - Example: nifi.nar.library.provider.narstorage1.source.directory
  - Value: /myNARs
  - Purpose: Shows the directory library with path to the NARs relative to the storage location.
- nifi.nar.library.provider.[**custom-name**].kerberos.principal
  - Example: nifi.nar.library.provider.narstorage1.kerberos.principal
  - Value: srv_my_s3_nar_user
  - Purpose: Defines the workload username of the machine user to access the resources.
- nifi.nar.library.provider.[**custom-name**].kerberos.password
  - Example: nifi.nar.library.provider.narstorage1.kerberos.password
  - Value: myPassword123
  - Purpose: Defines the workload password of the machine user to access the resources.
note
Cloudera recommends that you use a workload user name and password dedicated solely to accessing custom NARs, with permissions limited to the designated object store location.

Alternatively, you can also use the keytab file. If using a keytab, ensure it is available on all NiFi nodes with proper file permissions for the nifi user. This approach may be problematic when scaling a cluster.
```
Name: nifi.nar.library.provider.[**custom-name**].kerberos.keytab
Example: nifi.nar.library.provider.narstorage1.kerberos.keytab
Value: /path/to/my/file.keytab
```
Restart the NiFi service in Cloudera Manager.
Refresh the NiFi UI.

You can now drag a new processor onto NiFi canvas and search for your custom processor.