kafka-console-producer
Learn how you can use the kafka-console-producer
tool to produce
messages to a topic.
This tool is used to write messages to a topic in a text based format.
The following examples demonstrate the basic usage of the tool. In addition to reviewing these
examples, you can also use the --help
option to see a list of all available
options.
Produce messages to a topic
To start producing message to a topic, you need to run the tool and specify a server as well as a topic.
kafka-console-producer --bootstrap-server [HOST1:PORT1] --topic [TOPIC]
>my first message
>my second message
cat [FILE] | kafka-console-producer --bootstrap-server [HOST1:PORT1] --topic [TOPIC]
Produce messages to a topic in a secure cluster
In order to use the tool on a secure cluster, additional client configuration is required.
You do this by creating a .properties
file that contains the necessary
configurations to run the tool on a secure cluster. Which properties are configured in this
file depends on the security configuration of your cluster. See the client configuration
topics in Securing Apache Kafka to learn more about which exact properties you need
to use for your security configuration. Once the file is created, you can use
--producer.config
to pass the file to the tool.
.properties
file:security.protocol = SASL_SSL
sasl.mechanism=GSSAPI
sasl.kerberos.service.name = kafka
ssl.truststore.location = /var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password = test1234
sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true;
This example shows what properties you have to set when both Kerberos and TLS/SSL are configured.
- Create a
.properties
file.Use the example above. Make changes as necessary.
- Run the tool with the
--producer.config
option.kafka-console-producer --bootstrap-server [HOST1:PORT1] --topic [TOPIC] --producer.config client.properties
Define a key-value delimiter
It is possible to define a key-value delimiter for the given producer instance. The
delimiter can vary each time you run the tool. For example, if you run the tool with the
delimiter set to -
and then a second time using :
, Kafka
will know how to store the data. The first term is always stored as the key, the second as
the value. As a result, when the data written this way is consumed, the keys and values are
consumed irrespective of what delimiter was used when the records were written to the topic.
To produce messages with key-value delimiters, you need to set two properties,
parse.key
and key.seperator
. Both properties can be set
with the --properties
option.
kafka-console-producer --bootstrap-server [HOST1:PORT1] --topic [TOPIC] --property parse.key=true --property key.separator=":"
Configure retry backoff
--retry-backoff-ms
option.
kafka-console-producer --bootstrap-server [HOST1:PORT1] --topic [TOPIC] --retry-backoff-ms 1000