Ingesting Data into Kafka in CDP Public CloudPDF version

Verify data flow operation

Learn how you can verify the operation of your Kafka ingest data flow.

There are a number of ways to check that data is running through the flow you have built and it actually appears in Kafka.

  • You can verify that NiFi processors are not producing errors.
  • You can look at the processors in the UI to see the amount of data that has gone through them. You can also right-click on the processors, or on connections to view status history.
  • You can verify in Streams Messaging Manager that messages appear in the defined Kafka topic.
    To check this:
    1. Navigate to your Streams Messaging Data Hub cluster.
    2. Click Streams Messaging Manager in the Services section of the Cluster overview page.
    3. Search for your previously created topic and verify that your NiFi flow is visible, and it shows the producer and consumer as well.
    4. To check the messages written to the Kafka topic, click the blue Profile icon at the end of the Topic row and choose the Data Explorer tab.
    5. Since we are using an Avro schema in this example, the messages are encoded and not entirely readable. You can select the Avro Deserializer (Avro in both Key and Value fields) and make the data readable.