Prerequisites

For your data ingest source

• You have a Slack sandbox account.

  If you already have a Slack environment, you may skip this step. However you are highly recommended to make changes in dev/test environments before moving them to production. Slack offers a sandbox environment where such experiments can be carried out.

• You have created a Slack workspace.
  1. In Slack Workspace Directory click Manage Organization.

     

  2. Click Create Workspace.

     

  3. Provide Workspace Name and Workspace Domain. You may leave Workspace Description empty.

     

• You have created, configured, and installed a Slack App.
  1. Go to https://api.slack.com/apps, and click on Create New App.

     
     
     

  2. Select From Scratch.
  3. Provide an App Name.
  4. Under Pick a workspace to develop your app in: select the workspace you created in the previous step.

     

  5. Go to https://api.slack.com/apps/ and select the application you have created.
  6. Select Event Subscription and click on the toggle to enable events.

     After enabling event subscription an input box appears.

  7. Provide a request URL.

     The request URL consists of the Cloudera DataFlow Inbound Connection Endpoint Hostname, the value of the Allowed Paths parameter (if specified), and the Listening Port in the following format https://[***Endpoint Hostname***]:[***Listening Port***][***/Allowed Path***]. For example, https://cdev-test.inbound.dfx.nnfjwxkq.xcu2-8y8x.dev.cldr.work:9876/events

  8. Select the events you want to subscribe to. Only the events configured here will be pushed to Cloudera DataFlow.

     

     

  9. Install the Slack App by clicking Install App in the left pane then Install to Workspace in the main pane.

     

For more information on getting started with Slack Enterprise Grid sandboxes, see the Slack documentation.

For Cloudera DataFlow

• Your environment has public subnets and public connectivity enabled.

• You have enabled Cloudera DataFlow for an environment.

  For information on how to enable Cloudera DataFlow for an environment, see Enabling Cloudera DataFlow for an Environment.

• You have created a Machine User to use as the Cloudera Workload User.

• You have given the Cloudera Workload User the EnvironmentUser role.
  1. From the Management Console, go to the environment for which Cloudera DataFlow is enabled.
  2. From the Actions drop down, click Manage Access.
  3. Identify the user you want to use as a Workload User.
  4. Give that user EnvironmentUser role.

• You have synchronized your user to the Cloudera Public Cloud environment that you enabled for Cloudera DataFlow.

  For information on how to synchronize your user to FreeIPA, see Performing User Sync.

• You have granted your Cloudera user the DFCatalogAdmin and DFFlowAdmin roles to enable your user to add the ReadyFlow to the Catalog and deploy the flow definition.
  1. Give a user permission to add the ReadyFlow to the Catalog.
     1. From the Management Console, click User Management.
     2. Enter the name of the user or group you wish to authorize in the Search field.
     3. Select the user or group from the list that displays.
     4. Click Roles > Update Roles.
     5. From Update Roles, select DFCatalogAdmin and click Update.
  2. Give your user or group permission to deploy flow definitions.
     1. From the Management Console, click Environments to display the Environment List page.
     2. Select the environment to which you want your user or group to deploy flow definitions.
     3. Click Actions > Manage Access to display the Environment Access page.
     4. Enter the name of your user or group you wish to authorize in the Search field.
     5. Select your user or group and click Update Roles.
     6. Select DFFlowAdmin from the list of roles.
     7. Click Update Roles.
  3. Give your user or group access to the Project where the ReadyFlow will be deployed.
     1. Go to DataFlow > Projects.
     2. Select the project where you want to manage access rights and click More > Manage Access.
  4. Start typing the name of the user or group you want to add and select them from the list.
  5. Select the Resource Roles you want to grant.
  6. Click Update Roles.
  7. Click Synchronize Users.
• You have enabled Inbound Connection Support for your Cloudera DataFlow during flow deployment.

  For more information, see Create an Inbound Connection Endpoint during flow deployment .

For your ADLS data ingest target

• You have your ADLS container and path into which you want to ingest data.

• You have performed one of the following to configure access to your ADLS folder:

  • You have configured access to the ADLS folders with a RAZ enabled environment.

    It is a best practice to enable RAZ to control access to your object store folders. This allows you to use your Cloudera Public Cloud credentials to access ADLS folders, increases auditability, and makes object store data ingest workflows portable across cloud providers.

    1. Ensure that Fine-grained access control is enabled for your Cloudera DataFlow environment.
    2. From the Ranger UI, navigate to the ADLS repository.
    3. Create a policy to govern access to the ADLS container and path used in your ingest workflow. For example: adls-to-adls-avro-ingest
    4. Add the machine user that you have created for your ingest workflow to ingest the policy you just created.

    For more information, see Ranger policies for RAZ-enabled Azure environment.

  • You have configured access to ADLS folders using ID Broker mapping.

    If your environment is not RAZ-enabled, you can configure access to ADLS folders using ID Broker mapping.

    1. Access IDBroker mappings.
       1. To access IDBroker mappings in your environment, click Actions > Manage Access.
       2. Choose the IDBroker Mappings tab where you can provide mappings for users or groups and click Edit.
    2. Add your Cloudera Workload User and the corresponding Azure role that provides write access to your folder in ADLS to the Current Mappings section by clicking the blue + sign.
    3. Click Save and Sync.

For your S3 data ingest target

• You have your source S3 path and bucket.

• Perform one of the following to configure access to S3 buckets:

  • You have configured access to S3 buckets with a RAZ enabled environment.

    It is a best practice to enable RAZ to control access to your object store buckets. This allows you to use your Cloudera credentials to access S3 buckets, increases auditability, and makes object store data ingest workflows portable across cloud providers.

    1. Ensure that Fine-grained access control is enabled for your Cloudera DataFlow environment.
    2. From the Ranger UI, navigate to the S3 repository.
    3. Create a policy to govern access to the S3 bucket and path used in your ingest workflow.
    4. Add the machine user that you have created for your ingest workflow to the policy you just created.

    For more information, see Creating Ranger policy to use in RAZ-enabled AWS environment.

  • You have configured access to S3 buckets using ID Broker mapping.

    If your environment is not RAZ-enabled, you can configure access to S3 buckets using ID Broker mapping.

    1. Access IDBroker mappings.
       1. To access IDBroker mappings in your environment, click Actions > Manage Access.
       2. Choose the IDBroker Mappings tab where you can provide mappings for users or groups and click Edit.
    2. Add your Cloudera Workload User and the corresponding AWS role that provides write access to your folder in your S3 bucket to the Current Mappings section by clicking the blue + sign.
    3. Click Save and Sync.