kafka-console-producer

Learn how you can use the kafka-console-producer tool to produce messages to a topic.

This tool is used to write messages to a topic in a text based format.

The following examples demonstrate the basic usage of the tool. In addition to reviewing these examples, you can also use the --help option to see a list of all available options.

Produce messages to a topic

To start producing message to a topic, you need to run the tool and specify a server as well as a topic.

kafka-console-producer --bootstrap-server [HOST1:PORT1] --topic [TOPIC]

Start typing messages once the tool is running.

>my first message
>my second message

Alternatively, you can also produce the contents of a file to a topic.

cat [FILE] | kafka-console-producer --bootstrap-server [HOST1:PORT1] --topic [TOPIC]

Produce messages to a topic in a secure cluster

In order to use the tool on a secure cluster, additional client configuration is required. You do this by creating a .properties file that contains the necessary configurations to run the tool on a secure cluster. Which properties are configured in this file depends on the security configuration of your cluster. See the client configuration topics in Securing Apache Kafka to learn more about which exact properties you need to use for your security configuration. Once the file is created, you can use --producer.config to pass the file to the tool.

Example .properties file:

security.protocol = SASL_SSL
sasl.mechanism=GSSAPI
sasl.kerberos.service.name = kafka
ssl.truststore.location = /var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password = test1234
sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true;

This example shows what properties you have to set when both Kerberos and TLS/SSL are configured.

Do the following to run the tool in a secure cluster:

Create a .properties file.
Use the example above. Make changes as necessary.

Run the tool with the --producer.config option.

kafka-console-producer --bootstrap-server [HOST1:PORT1] --topic [TOPIC] --producer.config client.properties

Define a key-value delimiter

It is possible to define a key-value delimiter for the given producer instance. The delimiter can vary each time you run the tool. For example, if you run the tool with the delimiter set to - and then a second time using :, Kafka will know how to store the data. The first term is always stored as the key, the second as the value. As a result, when the data written this way is consumed, the keys and values are consumed irrespective of what delimiter was used when the records were written to the topic.

To produce messages with key-value delimiters, you need to set two properties, parse.key and key.seperator. Both properties can be set with the --properties option.

kafka-console-producer --bootstrap-server [HOST1:PORT1] --topic [TOPIC] --property parse.key=true --property key.separator=":"

Configure retry backoff

Before each retry, the producer refreshes the metadata of the relevant topics. You can configure the amount of time the producer waits before refreshing the metadata with the --retry-backoff-ms option.

kafka-console-producer --bootstrap-server [HOST1:PORT1] --topic [TOPIC] --retry-backoff-ms 1000