List of required configuration parameters for the Azure Event Hub to ADLS ReadyFlow

For the Azure Event Hub to ADLS ReadyFlow, the following parameters are required. Use the information you collected in Prerequisites.

Azure Event Hub to ADLS ReadyFlow configuration parameters
Parameter Name Description
ADLS File System

Specify the name of the ADLS data container you want to write to.

The full path will be constructed from: abfs://{ADLS File System}@#{ADLS Storage Account}.dfs.core.windows.net/#{ADLS Path}/${Kafka.topic}
ADLS Path

Specify the path within the ADLS data container where you want to write to without any leading characters.

The full path will be constructed from: abfs://{ADLS File System}@#{ADLS Storage Account}.dfs.core.windows.net/#{ADLS Path}/${Kafka.topic}
ADLS Storage Account

Specify the storage account name you want to write to.

The full ADLS data container path will be constructed from: abfs://{ADLS File System}@#{ADLS Storage Account}.dfs.core.windows.net/#{ADLS Path}/${Kafka.topic}
CDP Workload User Specify the CDP machine user or workload username that you want to use to authenticate to Kafka and the object store. Ensure this user has the appropriate access rights in Ranger for the Kafka topic and Ranger or IDBroker for object store access.
CDP Workload User Password Specify the password of the CDP machine user or workload user you are using to authenticate against Kafka and the object store.
CSV Delimiter If your source data is CSV, specify the delimiter here.
Data Output Format Specify the desired format for your output data. You can select from
  • CSV
  • JSON
  • AVRO
with this ReadyFlow.
Event Hub Access Policy Name

Specify the Access Policy Name that this flow should use.

The full path for the event hub endpoint will be constructed from sb://#{Event Hub Namespace}.#{Event Hub Service Bus Endpoint}/;SharedAccessKeyName=#{Event Hub Access Policy Name};SharedAccessKey=#{Event Hub Access Primary Key}
Event Hub Access Primary Key

Specify the Primary Key that allows clients to use the Access Policy that you provided earlier.

The full path for the event hub endpoint will be constructed from sb://#{Event Hub Namespace}.#{Event Hub Service Bus Endpoint}/;SharedAccessKeyName=#{Event Hub Access Policy Name};SharedAccessKey=#{Event Hub Access Primary Key}
Event Hub Consumer Group Specify the Event Hub Consumer Group you want to use with this flow. Any consumer group other than $Default needs to be created in Event Hub first.
Event Hub Instance Name Specify the Event Hub Instance Name inside the Event Hub Namespace you want to use.
Event Hub Namespace

Specify the Event Hub Namespace which contains the Event Hub instance you want to use.

The full path for the event hub endpoint will be constructed from sb://#{Event Hub Namespace}.#{Event Hub Service Bus Endpoint}/;SharedAccessKeyName=#{Event Hub Access Policy Name};SharedAccessKey=#{Event Hub Access Primary Key}
Event Hub Partitions Count Specify the number of partitions that the Event Hub has. Only this number of partitions will be used, so it is important to ensure that if the number of partitions changes that this value be updated. Otherwise, some messages may not be consumed.
Event Hub Service Bus Endpoint

Specify the Event Hub Service Bus Endpoint.

The default value is .servicebus.windows.net

The full path for the event hub endpoint will be constructed from sb://#{Event Hub Namespace}.#{Event Hub Service Bus Endpoint}/;SharedAccessKeyName=#{Event Hub Access Policy Name};SharedAccessKey=#{Event Hub Access Primary Key}

Filter Rule Specify the filter rule expressed in SQL to filter streaming events for the destination object store. Records matching the filter will be written to the destination object store. The default value forwards all records.
Schema Text Specify the Avro-formatted schema to be used for the source event hub data.