List of required configuration parameters for the Non-CDP S3 to S3/ADLS ReadyFlow

When deploying the Non-CDP S3 to S3/ADLS ReadyFlow, you have to provide the following parameters. Use the information you collected in Prerequisites.

Table 1. Non-CDP S3 to S3/ADLS ReadyFlow configuration parameters
Parameter Name Description
CDP Workload User Specify the CDP machine user or workload username that you want to use to authenticate to the object stores. Ensure this user has the appropriate access rights to the object store locations in Ranger or IDBroker.
CDP Workload User Password Specify the password of the CDP machine user or workload user you are using to authenticate against the object stores (via IDBroker).
Destination S3 or ADLS Path Specify the name of the destination S3 or ADLS path you want to write to. Make sure that the path starts with "/".
Destination S3 or ADLS Storage Location Specify the name of the destination S3 bucket or ADLS container you want to write to.
  • For S3, enter a value in the form:

    ns3a://[***Destination S3 Bucket***]

  • For ADLS, enter a value in the form:

    nabfs://[***Destination ADLS File System***]@[***Destination ADLS Storage Account***].dfs.core.windows.net

Preserve S3 Prefix Specify whether to preserve or ignore the folder hierarchy of the source files.
  • Set to "TRUE" to preserve folder hierarchy in the destination path.
  • Set to "FALSE" to flatten the hierarchy and only keep the file in the destination path.
Source S3 Access Key ID Specify the source (external) S3 access key ID.
Source S3 Bucket Specify the name of the source (external) S3 bucket you want to read from.
Source S3 Path Specify the name of the S3 path you want to read from. Make sure that the path does not start with "/".
Source S3 Region Specify the AWS region in which your source (external) bucket was created.
Source S3 Secret Access Key Specify the source (external) S3 secret access key.