ReadyFlow: ListenTCP filter to S3/ADLS
You can use the ListenTCP filter to S3/ADLS ReadyFlow to listen to TCP events on a specified port, filter them, and write them as JSON, CSV or Avro files to S3 or ADLS.
This ReadyFlow listens to a JSON, CSV or Avro data stream on a specified port and parses the data based on a specified Avro-formatted schema. You can filter events by specifying a SQL query in the 'Filter Rule' parameter. The filtered events are then converted to the specified output data format and written to the target S3 or ADLS destination. The flow writes out a file every time its size has either reached 100MB or five minutes have passed. Files can reach a maximum size of 1GB. Failed S3 or ADLS write operations are retried automatically to handle transient issues. Define a KPI on the 'failure_WriteToS3/ADLS' connection to monitor failed write operations.
|Source Format||JSON, CSV, Avro|
|Destination||CDP managed Amazon S3 or ADLS|
|Destination Format||JSON, CSV, Avro|
Moving data with the ListenTCP filter to S3/ADLS flow
You can use a Non-CDP ADLS to S3/ADLS data flow when you want to move data from a non-CDP managed location in ADLS to a CDP managed Amazon S3 or ADLS location. You need to specify the source and target locations.