Quick Flow: Consume Kafka to Kafka
Learn about this Quick Flow that consumes from a Kafka topic and writes it to another Kafka topic while converting the message format from CSV to JSON. Review flow details, download the template, and understand the template parameters.
About this Quick Flow
This flow definition consumes data from a source Kafka topic and writes it to another Kafka topic while converting the message format from CSV to JSON. The user must configure a Schema Registry connection and specify a schema for the source Kafka topic which allows the user to provide SQL to filter the stream. If the events from the source Kafka topic do not adhere to the schema, the event is logged and not sent to the destination Kafka topic.
Quick Flow definition details and download
|Quick Flow definition details|
|Flow Artifact||Download the file:|
Quick Flow parameters
|CDP Env Truststore File||Provide the Truststore file that you generated for the environment where your Kafka cluster is running in||N|
|CDP Env Trustore Password||Provide the password for the Truststore you created for the environment where your Kafka cluster is running in||Y|
|CDP Schema Registry URL||Specify the Schema Registry URL you want to connect to in the format https://hostname:7790/api/v1||N||
|CSV Delimiter||Specify the delimiter for the source CSV data||N||, ; ||
|Kafka Broker Endpoint||Specify the Kafka bootstrap servers string as a comma separated list||N||
|Kafka Consumer Group ID||Specify the id for the consumer group used for the the source topic you are consuming from||N|
|Kafka Destination Topic||Specify the destination topic name||N|
|Kafka Producer ID||Specify the id for the Kafka producer writing into the destination topic||N|
|Kafka Service Account||
Specify the CDP machine user name that you want to use to authenticate to Kafka. Ensure this user has the appropriate access rights to the Kafka topics.
|Kafka Service Account Password||Specify the password of the CDP machine user you're using to authenticate against Kafka||Y|
|Kafka Source Topic||Specify a topic name or list of topics with the same schema that you want to read from||N|
|Schema Name||Specify the schema name to be looked up in the Schema Registry||N|