Producing data to Kafka topic

To produce data to Kafka topics, you create a property file, ensure that its permissions are set restrictively, list all the topics that the Machine User has access to, and produce a text message to the Kafka topic.

Here, you produce data to a Kafka topic on your SSH session.
You must have created a Kafka cluster, created a Machine User, granted environment access to the Machine User, created Kafka topics, and created Ranger policies.
  1. Create a new file with the following content:
    ssl.truststore.location=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks required username="srv_kafka-client" password="SuperSecretPassword";
  2. The file contains sensitive information. Ensure its permissions are set restrictively.
    chmod 400
    With this configuration you can start running Kafka client applications.
  3. Run the following command to list all the topics that the Machine User has access to.
    BROKER_HOST=$(hostname -f)
    kafka-topics \
      --bootstrap-server $BROKER_HOST:9093 \
      --command-config \

    The output include a number of INFO lines, with the two topics you created at the end of the output:


    If you do not see them, go back and review the previous steps.

  4. Run the following command to produce a simple text message to the Kafka topic by using the kafka-console-producer command.
    echo 'Hello, Kafka!' | kafka-console-producer \
      --broker-list $BROKER_HOST:9093 \
      --producer.config \
      --topic machine-data

If the output of the command above contains only INFO lines and no ERROR lines, it indicates that data was produced correctly to the Kafka topic.

In the next section, you use the kafka-console-consumer command to verify the contents of the Kafka topic.