Cloudera Docs

Producing data in Avro format

You can run the sample producer application to produce data to a Kafka topic in Avro format. You can send data to a Kafka topic in Avro format. Learn how to store a schema in Cloudera Schema Registry and use a simple Java Kafka client to send and read data using that schema.

You must have defined access policies for Schema Registry.
  1. On your computer, clone this repository using git and change to the kafka-client-avro directory. 
    git clone https://github.com/asdaraujo/cdp-examples
cd cdp-examples/kafka-client-avro
  2. Build the client binaries. 
    mvn clean package
  3. Create a copy of the producer properties template file. 
    cp src/main/resources/producer.properties.template producer.properties
  4. Edit the producer.properties file and replace the following placeholders with the corresponding values:

    • [***MACHINE_USER_NAME***]

      The Machine User name (prefixed with srv_). Note that this needs to be replaced at two locations in the template.

    • [***MACHINE_USER_PASSWORD***]

      The Machine User’s Workload Password. Note that this needs to be replaced at two locations in the template.

    • [***BROKER1***], [***BROKER2***], [***BROKER3***]

      The hostnames of three cluster brokers. Also, ensure that the broker port numbers match the numbers seen in the Streams Messaging Manager Brokers page.

    • [***TRUSTSTORE_PATH***]

      The path of the truststore.jks file created previously.

    • [***SCHEMA_REGISTRY_ENDPOINT***]

      The Schema Registry endpoint found previously.

  5. Run the producer application with the following command: 
    java \
  -cp ./target/kafka-client-avro-1.0-SNAPSHOT.jar \
  com.cloudera.examples.MachineDataProducer \
  ./producer.properties \
  ./src/main/avro/MachineData.v1.avsc
    The producer application (MachineDataProducer class) takes the following arguments:
    • The producer.properties file.
    • An Avro schema definition file that the producer uses to produce messages.
    If the producer is working correctly, you see several messages similar to the following:
    ... INFO  c.c.examples.MachineDataProducer - Successfully produced message to partition [machine-data-avro-8], offset [0]
  6. Leave the SSH session open and the producer running for now.