ReadyFlow: Non-CDP S3 to S3/ADLS
You can use the Non-CDP S3 to S3/ADLS ReadyFlow to move your data from a non-CDP Amazon S3 to a destination S3 or ADLS.
This ReadyFlow consumes files from an external source non-CDP S3 location and writes them to a CDP managed destination S3 or ADLS location. For the source, specify the S3 bucket region, name, path and access keys. For the destination, specify the S3 or ADLS storage location and path. The ReadyFlow polls the source bucket for new files (it performs a listing periodically). You can pick whether to preserve or ignore the folder hierarchy of the source files when writing to the destination path. Failed S3 or ADLS write operations are retried automatically to handle transient issues. Define a KPI on the failure_WriteToS3/ADLS connection to monitor failed write operations.
This ReadyFlow leverages CDP's centralized access control for cloud storage access. Make sure to either set up an IDBroker mapping or Ranger policies when using fine-grained object store access allowing your workload user access to the destination S3 or ADLS location.
|Source||External Amazon S3|
|Destination||CDP managed Amazon S3 or ADLS|
|Destination Format||Same as source|
Moving data with Non-CDP S3 to S3/ADLS flow
You can use a Non-CDP S3 to S3/ADLS data flow when you want to move data from a non-CDP managed location in Amazon S3 to a CDP managed Amazon S3 or ADLS location. You need to specify the source and target locations.