Cloudera Docs

List of required configuration parameters for the Kafka to ADLS Avro ReadyFlow

When deploying the Kafka to ADLS Avro ReadyFlow, you have to provide the following parameters. Use the information you collected in Prerequisites.

Table 1. Kafka to ADLS Avro ReadyFlow configuration parameters
Parameter Name Description
ADLS File System Specify the name of the ADLS data container you want to write to.

The full path will be constructed from: abfs://{ADLS File System}@#{ADLS Storage Account}.dfs.core.windows.net/#{ADLS Path}/${Kafka.topic}
ADLS Path Specify the path within the ADLS data container where you want to write to without any leading characters.

The full path will be constructed from: abfs://{ADLS File System}@#{ADLS Storage Account}.dfs.core.windows.net/#{ADLS Path}/${Kafka.topic}
ADLS Storage Account Specify the storage account name you want to write to.

The full ADLS data container path will be constructed from: abfs://{ADLS File System}@#{ADLS Storage Account}.dfs.core.windows.net/#{ADLS Path}/${Kafka.topic}
CDP Workload User Specify the Cloudera machine user or workload username that you want to use to authenticate to Kafka and the object store. Ensure this user has the appropriate access rights in Ranger for the Kafka topic and Ranger or IDBroker for object store access.
CDP Workload User Password Specify the password of the Cloudera machine user or workload user you are using to authenticate against Kafka and the object store.
CSV Delimiter If your source data is CSV, specify the delimiter here.
Data Input Format Specify the format of your input data.
  • CSV
  • JSON
  • AVRO
Kafka Broker Endpoint Specify the Kafka bootstrap servers string as a comma separated list.
Kafka Consumer Group ID The name of the consumer group used for the source topic you are consuming from.
Kafka Source Topic Specify a topic name that you want to read from.
Schema Name Specify the schema name to be looked up in the Schema Registry for the source Kafka topic.
Schema Registry Hostname Specify the hostname of the Schema Registry you want to connect to. This must be the direct hostname of the Schema Registry itself, not the Knox Endpoint.