Cloudera Docs

Preparing cloud storage to deploy custom processors

To use a custom Apache NiFi processor or controller service in one of your Cloudera DataFlow flow deployments, add the NiFi Archive (NAR), Python file, or Python package containing the custom processor or controller service to a cloud storage location for later use during a flow deployment.

  1. Create your cloud storage location.
  2. Upload your NAR file, Python file, or Python package to the cloud storage location.
  3. Configure access to your cloud provider storage in one of two ways:

    • You have configured access to S3 buckets using ID Broker mapping.

      If your environment is not RAZ-enabled, you can configure access to S3 buckets using ID Broker mapping.
      1. Access IDBroker mappings.
        1. To access IDBroker mappings in your environment, click Actions > Manage Access.
        2. Choose the IDBroker Mappings tab where you can provide mappings for users or groups and click Edit.
      2. Add your Cloudera Workload User and the corresponding AWS role that provides write access to your folder in your S3 bucket to the Current Mappings section by clicking the blue + sign.
      3. Click Save and Sync.

    • You have configured access to S3 buckets with a RAZ enabled environment.

      It is a best practice to enable RAZ to control access to your object store buckets. This allows you to use your Cloudera credentials to access S3 buckets, increases auditability, and makes object store data ingest workflows portable across cloud providers.
      1. Ensure that Fine-grained access control is enabled for your Cloudera DataFlow environment.
      2. From the Ranger UI, navigate to the S3 repository.
      3. Create a policy to govern access to the S3 bucket and path used in your ingest workflow.
      4. Add the machine user that you have created for your ingest workflow to the policy you just created.

      For more information, see Creating Ranger policy to use in RAZ-enabled AWS environment.

  4. Note the workload user name and password, and cloud storage location to use in the Deployment Wizard.

Once you have added the NAR files to a cloud storage location, you are ready to launch the Deployment Wizard and deploy a flow.