ReadyFlow Overview: Kafka to ADLS Avro

You can use the Kafka to ADLS Avro ReadyFlow to move your data from a Kafka topic to an ADLS directory.

This ReadyFlow consumes JSON, CSV, or Avro data from a source Kafka topic and merges the events into Avro files before writing the data to Azure Data Lake Storage (ADLS). The flow writes out a file every time its size has either reached 100 MB or five minutes have passed. Files can reach a maximum size of 1 GB. You can specify the topic you want to read from as well as the target ADLS data container and path. Failed ADLS write operations are retried automatically to handle transient issues. Define a KPI on the 'failure_WriteToADLS' connection to monitor failed write operations.

Kafka to ADLS Avro ReadyFlow details
Source	Kafka topic
Source Format	JSON, CSV, Avro
Destination	ADLS
Destination Format	Avro

Moving data to object stores

Cloud environments offer numerous deployment options and services. There are many ways to store data in the cloud, but the easiest option is to use object stores. Object stores are extremely robust and cost-effective storage solutions with multiple levels of durability and availability. You can include them in your data pipeline, both as an intermediate step and as an end state. Object stores are accessible to many tools and connecting systems, and you have a variety of options to control access.