Cluster discovery using load balancers
In case of more complex Kafka cluster setups, you might need a cluster discovery solution more sophisticated than a DNS server. In such cases, you should consider using a load balancer.
Cluster discovery using load balancers is less lightweight than using DNS servers, but a viable solution for more complex cases. With a load balancer, it is possible to poll the nodes, check their health status and exclude stopped nodes from targets, automatically redirect requests to living nodes. When using a load balancer the requests are forwarded to a broker as shown in the figure:
Because the client connects to the load balancer and is then forwarded to a broker, SSL handshake and SASL authentication errors can occur (this is a defending mechanism to avoid man-in-the-middle attacks), therefore, additional setup is needed.
The configuration depends on your security protocol:
No security in the cluster (security protocol is PLAINTEXT)
Setup steps are not required before connecting to the Kafka cluster, the load balancers should work out of the box with Kafka.
SASL with Kerberos enabled
Perform the setup described in section Setup for SASL with Kerberos.
TLS/SSL encryption is enabled
Perform the setup described in section Setup for TLS/SSL encryption.
SASL with Kerberos and TLS/SSL are both enabled
Perform the setup described in Setup for SASL with Kerberos and Setup for TLS/SSL encryption.