Hadoop Security Guide
Also available as:
loading table of contents...

Enable SSL for Kafka Clients

Kafka allows clients to connect over SSL. By default SSL is disabled, but it can be enabled as needed.

Before you begin, be sure to generate the key, SSL certificate, keystore, and truststore that will be used by Kafka.

Configuring the Kafka Broker

The Kafka Broker supports listening on multiple ports and IP addresses. To enable this feature, specify one or more comma-separated values in the listeners property in server.properties.

Both PLAINTEXT and SSL ports are required if SSL is not enabled for inter-broker communication (see the following subsection for information about enabling inter-broker communication):


The following SSL configuration settings are needed on the broker side:

ssl.keystore.location = /var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password = test1234
ssl.key.password = test1234
ssl.truststore.location = /var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password = test1234

The following optional settings are available:


Specify whether client authentication is required, requested, or not required.

none: no client authentication.

required: client authentication is required.

requested: client authentication is requested, but a client without certs can still connect.

Note: If you set ssl.client.auth to requested or required, then you must provide a truststore for the Kafka broker. The truststore should contain all CA certificates that are used to sign clients' keys.

ssl.cipher.suitesSpecify one or more cipher suites: named combinations of authentication, encryption, MAC and key exchange algorithms used to negotiate the security settings for a network connection using the TLS or SSL network protocol. 
ssl.enabled.protocolsSpecify the SSL protocols that you will accept from clients. Note: SSL is deprecated; its use in production is not recommended.TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.typeSpecify the SSL keystore type.JKS
ssl.truststore.typeSpecify the SSL truststore type.JKS

Enabling SSL for Inter-Broker Communication

To enable SSL for inter-broker communication, add the following setting to the broker properties file (default is PLAINTEXT):

security.inter.broker.protocol = SSL

Enabling Additional Cipher Suites

To enable any cipher suites other than the defaults that come with JVM (see Java Cryptography documentation), you will need to install JCE Unlimited Strength Policy files (download link).

Validating the Configuration

After you start the broker, you should see the following information in the server.log file:

with addresses: PLAINTEXT -> EndPoint(,9092,PLAINTEXT),SSL -> EndPoint(,9093,SSL)

To make sure that the server keystore and truststore are set up properly, run the following command:

openssl s_client -debug -connect localhost:9093 -tls1

(Note: TLSv1 should be listed under ssl.enabled.protocols)

In the openssl output you should see the server certificate; for example:

Server certificate
subject=/C=US/ST=CA/L=Santa Clara/O=org/OU=org/CN=JBrown
issuer=/C=US/ST=CA/L=Santa Clara/O=org/OU=org/CN=kafak/emailAddress=test@test.com

If the certificate does not display, or if there are any other error messages, then your keystore is not set up properly.

Configuring Kafka Producer and Kafka Consumer

SSL is supported for new Kafka Producers and Consumer processes; the older API is not supported. Configuration settings for SSL are the same for producers and consumers.

If client authentication is not needed in the broker, then the following is a minimal configuration example:

security.protocol = SSL
ssl.truststore.location = /var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password = test1234

If client authentication is required, first create a keystore (described earlier in this chapter). Next, specify the following settings:

ssl.keystore.location = /var/private/ssl/kafka.client.keystore.jks
ssl.keystore.password = test1234
ssl.key.password = test1234

One or more of the following optional settings might also be needed, depending on your requirements and the broker configuration:

ssl.provider The name of the security provider used for SSL connections. Default value is the default security provider of the JVM. 
ssl.cipher.suitesSpecify one or more cipher suites: named combinations of authentication, encryption, MAC and key exchange algorithms used to negotiate the security settings for a network connection using the TLS or SSL network protocol. 
ssl.enabled.protocolsList at least one of the protocols configured on the broker side.TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.typeSpecify the SSL keystore type. JKS
ssl.truststore.typeSpecify the SSL truststore type.JKS

The following two examples launch console-producer and console-consumer processes:

kafka-console-producer.sh --broker-list localhost:9093 --topic test --producer.config client-ssl.properties

kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic test --new-consumer --consumer.config client-ssl.properties