Client and broker compatibility across Kafka versions

An overview on client and broker version compatibility.

Maintaining compatibility across different Kafka clients and brokers is a common issue. Mismatches among client and broker versions can occur as part of any of the following scenarios:

Upgrading your Kafka cluster without upgrading your Kafka clients.
Using a third-party application that produces to or consumes from your Kafka cluster.
Having a client program communicating with two or more Kafka clusters (such as consuming from one cluster and producing to a different cluster).
Using Flume or Spark as a Kafka consumer.

In these cases, it is important to understand client/broker compatibility across Kafka versions. Here are general rules that apply:

Newer Kafka brokers can talk to older Kafka clients. The reverse is not true: older Kafka brokers cannot talk to newer Kafka clients.
Changes in either the major part or the minor part of the upstream version major.minor determines whether the client and broker are compatible. Differences among the maintenance versions are not considered when determining compatibility.

As a result, the general pattern for upgrading Kafka from version A to version B is:

Change Kafka server.properties to refer to version A.
Upgrade the brokers to version B and restart the cluster.
Upgrade the clients to version B.
After all the clients are upgraded, change the properties from the first step to version B and restart the cluster.