Configuring Flow Management clusters to hot load custom NARs

Hot loading custom NARs means that you can store the NARs for your NiFi custom processors in an external folder, and load them from that directory for use on the NiFi canvas. This makes it easy to edit custom processors and add new ones without having to restart your Flow Management cluster.

  • You can configure NiFi to hot load custom NARs in all supported cloud providers.
  • You can only access resources in the object store of the cloud provider in which you environment is running. But but you can have a development and a production environment, for example, and configure both environments to access the same location to hot load custom NARs
  • It is possible to define multiple locations if desired by repeating the configuration properties with a different “[**custom-name**]” for each location. But all locations should be in the object store of the underlying cloud provider of where the NiFi cluster is located.

  • NiFi polls for new NARs in the configured location(s) every 5 minutes.

  1. In Management Console, select Data Hub Clusters, then go to the Flow Management cluster, then NiFi Service.
  2. Click Configuration.
  3. Search for Safety Valve and then Find: NiFi Node Advanced Configuration Snippet (Safety Valve) for staging/nifi.properties.xml
  4. Create the following new properties:
    Name: nifi.nar.library.provider.[**custom-name**].implementation
    Example: nifi.nar.library.provider.narstorage1.implementation
    Value: org.apache.nifi.nar.hadoop.HDFSNarProvider
    
    Path to the configuration files
    Name: nifi.nar.library.provider.[**custom-name**].resources
    Example: nifi.nar.library.provider.narstorage1.resources
    Value, by default: /etc/hadoop/conf.cloudera.core_settings/core-site.xml
    
    Storage location for the NARs (the bucket name in S3 for example)
    Name: nifi.nar.library.provider.[**custom-name**].storage.location
    Example: nifi.nar.library.provider.narstorage1.storage.location
    Value: s3a://my-custom-nars-bucket
    
    Directory library with path to the NARs relative to the storage location
    Name: nifi.nar.library.provider.[**custom-name**].source.directory
    Example: nifi.nar.library.provider.narstorage1.source.directory
    Value: /myNARs
    
    Workload username and password of the machine user to use to access the resources
    
    Name: nifi.nar.library.provider.[**custom-name**].kerberos.principal
    Example: nifi.nar.library.provider.narstorage1.kerberos.principal
    Value: srv_my_s3_nar_user
    
    Name: nifi.nar.library.provider.[**custom-name**].kerberos.password
    Name: nifi.nar.library.provider.narstorage1.kerberos.password
    Value: myPassword123
    
  5. Restart the NiFi service in the Cloudera Manager.

Refresh the NiFi UI and then drag and drop a new processor to NiFi canvas and search for your custom processor.