ReadyFlow: Confluent Cloud to S3/ADLS

You can use the Confluent Cloud to S3/ADLS ReadyFlow to ingest JSON, CSV or Avro data from a source Kafka topic in Confluent Cloud to a destination S3 or ADLS location, while filtering events using a SQL query.

This ReadyFlow consumes JSON, CSV or Avro data from a source Kafka topic in Confluent Cloud and parses the schema by looking up the schema name in the Confluent Schema Registry. You can filter events by specifying a SQL query in the 'Filter Rule' parameter. The filtered records are then converted to the specified output data format. The flow then writes out a file every time its size has either reached 100MB or five minutes have passed. Files can reach a maximum size of 1GB. The filtered events are then written to the destination Amazon S3 or Azure Data Lake Service (ADLS) location. Failed S3 or ADLS write operations are retried automatically to handle transient issues. Define a KPI on the failure_WriteToS3/ADLS connection to monitor failed write operations.

ReadyFlow details
Source	Confluent Cloud Kafka Topic
Source Format	JSON, CSV, Avro
Destination	Amazon S3 or ADLS
Destination Format	JSON, CSV, Avro