ReadyFlow: Kafka filter to Kafka

Learn about the Kafka filter to Kafka ReadyFlow.

Kafka filter to Kafka ReadyFlow description

This ReadyFlow consumes JSON, CSV, or Avro data from a source Kafka topic and parses the schema by looking up the schema name in the CDP Schema Registry. You can filter events by specifying a SQL query in the Filter Rule parameter. The filtered events are then converted to the specified output data format and written to the destination Kafka topic. Failed Kafka write operations are retried automatically to handle transient issues. Define a KPI on the failure_WriteToKafka connection to monitor failed write operations.

Kafka filter to Kafka ReadyFlow details

ReadyFlow details
Source Kafka Topic
Source Format JSON, CSV, or Avro
Destination Kafka Topic
Destination Format JSON, CSV, or Avro
Table 1. Kafka filter to Kafka ReadyFlow configuration parameters
Parameter Name Description Example
CDP Workload User Specify the CDP machine user or workload user name that you want to use to authenticate to Kafka. Ensure that this user has the appropriate access rights to the source and destination Kafka topics.
CDP Workload User Password Specify the password of the CDP machine user or workload user you are using to authenticate against Kafka.
CSV Delimiter If your source data is CSV, specify the delimiter here.
Data Input Format Specify the format of your input data.
  • CSV
  • JSON
  • AVRO
Data Output Format Specify the desired format for your output data.
  • CSV
  • JSON
  • AVRO
Filter Rule Specify the filter rule expressed in SQL to filter streaming events for the destination topic. Records matching the filter will be written to the destination topic. The default value forwards all records.
Kafka Broker Endpoint Specify the Kafka bootstrap servers string as a comma separated list.
Kafka Consumer Group ID Specify the ID for the consumer group used for the source topic you are consuming from.
Kafka Destination Topic Specify the destination topic name.
Kafka Producer ID Specify the id for the Kafka producer writing into the destination topic.
Kafka Source Topic Specify the topic you want to read from.
Schema Name Specify the schema name to be looked up in the Schema Registry.
Schema Registry Hostname Specify the hostname of the Schema Registry you want to connect to. This must be the direct hostname of the Schema Registry itself, not the Knox Endpoint.