ReadyFlow: Confluent Cloud to S3/ADLS

You can use the Confluent Cloud to Confluent Cloud to S3/ADLS ReadyFlow to ingest JSON, CSV or Avro data from a source Kafka topic in Confluent Cloud to a destination S3 or ADLS location, while filtering events using a SQL query.

This ReadyFlow consumes JSON, CSV or Avro data from a source Kafka topic in Confluent Cloud and parses the schema by looking up the schema name in the Confluent Schema Registry. You can filter events by specifying a SQL query in the 'Filter Rule' parameter. The filtered records are then converted to the specified output data format. The flow then writes out a file every time its size has either reached 100MB or five minutes have passed. Files can reach a maximum size of 1GB. The filtered events are then written to the destination Amazon S3 or Azure Data Lake Service (ADLS) location. Failed S3 or ADLS write operations are retried automatically to handle transient issues. Define a KPI on the 'failure_WriteToS3/ADLS' connection to monitor failed write operations.

ReadyFlow details
Source Confluent Cloud Kafka Topic
Source Format JSON, CSV, Avro
Destination Amazon S3 or ADLS
Destination Format JSON, CSV, Avro

Moving data with the Confluent Cloud to S3/ADLS ReadyFlow

This ReadyFlow leverages CDP's centralized access control for cloud storage access. Make sure to either set up an IDBroker mapping or Ranger policies when using fine-grained object store access allowing your workload user access to the target S3 or ADLS locations.