Cloudera Docs

List of required configuration parameters for the S3 to S3 Avro ReadyFlow

When deploying the S3 to S3 Avro ReadyFlow, you have to provide the following parameters. Use the information you collected in Prerequisites.

Table 1. S3 to S3 Avro ReadyFlow configuration parameters
Parameter Name Description
CDP Workload User Specify the Cloudera machine user or workload username that you want to use to authenticate Schema Registry. Ensure this user has the appropriate access rights in Ranger.
CDP Workload User Password Specify the password of the Cloudera machine user or workload user you are using to authenticate against Schema Registry.
CDP Environment

Cloudera DataFlow uses this parameter to auto-populate the Flow Deployment with Hadoop configuration files required to interact with S3.

Cloudera DataFlow automatically adds all required configuration files to interact with Data Lake services. Unnecessary files that are added will not impact the deployment process.
Data Input Format

Specify the format of your input data.

If your data input format is CSV, define a CSV delimiter for the data in the CSV Delimiter text box. If you use AVRO or JSON format, the delimiter is ignored.

  • CSV
  • JSON
  • AVRO
Data Output Format

Specify the format of your output data.

As you are using AVRO format, the delimiter is ignored.
CSV Delimiter If your source data is CSV, specify the delimiter here.
S3 Bucket Specify the name of the S3 bucket that you want to read from.

The full path is constructed from:

s3a://#{S3 Bucket}/#{S3 Path}/${S3 Bucket}
S3 Path Specify the path within the bucket where you want to read from without any leading characters.

The full path is constructed from:

s3a://#{S3 Bucket}/#{S3 Path}/${S3 Bucket}
S3 Bucket Specify the name of the S3 bucket that you want to write to.

The full path is constructed from:

s3a://#{S3 Bucket}/#{S3 Path}/${S3 Bucket}
S3 Path Specify the path within the bucket where you want to write to without any leading characters.

The full path is constructed from:

s3a://#{S3 Bucket}/#{S3 Path}/${S3 Bucket}
S3 Bucket Region

Specify the AWS region in which your bucket was created.

Supported values are:

  • us-gov-west-1

  • us-gov-east-1

  • us-east-1

  • us-east-2

  • us-west-1

  • us-west-2

  • eu-west-1

  • eu-west-2

  • eu-west-3

  • eu-central-1

  • eu-north-1

  • eu-south-1

  • ap-east-1

  • ap-south-1

  • ap-southeast-1

  • ap-southeast-2

  • ap-northeast-1

  • ap-northeast-2

  • ap-northeast-3

  • sa-east-1

  • cn-north-1

  • cn-northwest-1

  • ca-central-1

  • me-south-1

  • af-south-1

  • us-iso-east-1

  • us-isob-east-1

  • us-iso-west-1
S3 Path Specify the path within the bucket where you want to write to without any leading characters.

The full path will be constructed from:

s3a://#{S3 Bucket}/#{S3 Path}/${Kafka.topic}
Schema Name

Identify the schema that you want to use in your data flow.

Cloudera DataFlow looks up this schema in the Schema Registry you define with the Schema Registry Hostname. See the Appendix for an example schema.
Schema Registry Hostname

Specify the hostname of the Schema Registry running on the master node in the Streams Messaging cluster that you want to connect to.

This must be the direct hostname of the Schema Registry itself, not the Knox Endpoint.