Cluster discovery using load balancers

In case of more complex Kafka cluster setups, you might need a cluster discovery solution more sophisticated than a DNS server. In such cases, you should consider using a load balancer.

Cluster discovery using load balancers is less lightweight than using DNS servers, but a viable solution for more complex cases. With a load balancer, it is possible to poll the nodes, check their health status and exclude stopped nodes from targets, automatically redirect requests to living nodes. When using a load balancer the requests are forwarded to a broker as shown in the figure:

Because the client connects to the load balancer and is then forwarded to a broker, SSL handshake and SASL authentication errors can occur (this is a defending mechanism to avoid man-in-the-middle attacks), therefore, additional setup is needed.

The configuration depends on your security protocol:

• No security in the cluster (security protocol is PLAINTEXT)

  Setup steps are not required before connecting to the Kafka cluster, the load balancers should work out of the box with Kafka.

• SASL with Kerberos enabled

  Perform the setup described in section Setup for SASL with Kerberos.

• TLS/SSL encryption is enabled

  Perform the setup described in section Setup for TLS/SSL encryption.

• SASL with Kerberos and TLS/SSL are both enabled

  Perform the setup described in Setup for SASL with Kerberos and Setup for TLS/SSL encryption.