Migrating Kafka clusters from ZooKeeper to KRaft
Learn about migrating an existing Zookeeper-based Kafka cluster to KRaft. Migration is semi-automatic and requires minimal configuration changes. Additionally, migration is done in a rolling fashion and requires no downtime.
Apache Kafka Raft (KRaft) is a consensus protocol used for metadata management that was developed as a replacement for Apache ZooKeeper. Using KRaft for managing Kafka metadata instead of ZooKeeper offers various benefits including a simplified architecture and a reduced operational footprint.
Starting with Cloudera Streams Messaging - Kubernetes Operator 1.3, ZooKeeper is deprecated and will be removed in a future release. Cloudera encourages you to migrate your existing clusters to KRaft.
Migration at glance
Migration is semi-automatic. You manage a migration (start, finalize, or roll back) using the
strimzi.io/kraft
annotation. You add the annotation to the
Kafka resource of the cluster.
The majority of configuration changes needed in your cluster are handled by the Strimzi
Cluster Operator when you add the strimzi.io/kraft
annotation. Other than
adding annotation, you are only required to deploy KRaft controllers before starting migration,
and clean up unused configuration properties after migration. All other steps are automated.
The annotation has the following valid values.
-
migration
- Starts the migration process and migrates a cluster up to a state where rollbacks are still possible. -
enabled
- Finalizes a migration. Once finalization is triggered, rollbacks to ZooKeeper are no longer possible. -
rollback
- Reverts the cluster from Kraft to ZooKeeper. Rollbacks are only possible for clusters where migration is not finalized. -
disabled
- The default value of the annotation. Indicates that the cluster uses ZooKeeper. Also used to finalize a rollback.
Cluster metadata states during migration
When you apply strimzi.io/kraft
annotation to a Kafka resource, the Strimzi
Cluster Operator performs various migration-related actions on the cluster. During a migration,
the cluster goes through a number of metadata states. You can monitor these metadata states to
see the progress of migration. The metadata state of the cluster is also an indicator that shows
whether a cluster can be rolled back.
The metadata states that the cluster goes through during a migration and a rollback are as follows.
- Migration
-
-
ZooKeeper
- Indicates that the cluster uses ZooKeeper. This is the initial state. Migration is started and the cluster is moved to the next state when thestrimzi.io/kraft="migration"
annotation is added to the Kafka resource. -
KRaftMigration
- KRaft controllers are running, metadata and brokers are getting migrated to KRaft. In this state some brokers are already running in KRaft mode while others are still using ZooKeeper. -
KRaftDualWriting
- Metadata migration is finished. However, both brokers and controllers are still connected to ZooKeeper. Metadata is being written by KRaft controllers to ZooKeeper. -
KRaftPostMigration
- Brokers are running in KRaft mode and have been disconnected from ZooKeeper. KRaft controllers are connected to ZooKeeper and continue to write metadata to ZooKeeper, but are ready to disconnect.This metadata state marks the end of the first migration phase. The Strimzi Cluster Operator only moves the cluster to the next state when you update the
strimzi.io/kraft
annotation toenabled
. This is the last state where rollback is still possible. -
PreKRaft
- KRaft controllers are disconnected from ZooKeeper. Zookeeper is ready to be removed. -
KRaft
- Migration is finalized. Both Kafka brokers and controllers are running in KRaft mode. ZooKeeper is disconnected from the cluster. ZooKeeper instances are removed.
-
- Rollback
-
-
KRaftPostMigration
- Brokers are running in KRaft mode and have been disconnected from ZooKeeper. KRaft controllers are connected to ZooKeeper and continue to write metadata to ZooKeeper, but are ready to disconnect. This is the initial state when you begin a rollback. -
KRaftDualWriting
- Both brokers and controllers are connected to ZooKeeper. Metadata is being written by KRaft controllers to ZooKeeper as well. In case of a rollback, the Strimzi Cluster Operator stops at this metadata state and waits until you start finalization by setting thestrimzi.io/kraft
annotation todisabled
. -
ZooKeeper
- Indicates that the cluster uses ZooKeeper. This is the final state during a rollback.
-
Migrating a Kafka cluster to KRaft
You migrate an existing ZooKeeper-based Kafka cluster to KRaft by creating KRaft controllers and then using the strimzi.io/kraft annotation on the Kafka resource of the cluster to initiate and finalize the migration. After finalization, you remove configuration related to ZooKeeper.
Ensure the following:
-
You are on Cloudera Streaming Kubernetes Operator 1.3 or higher.
-
The Kafka cluster version is 3.9.0.1.3 or higher.
Migration is not supported from lower versions. If you are on a lower version, upgrade to a supported version.
Rolling back a Kafka cluster to ZooKeeper
You roll back a KRaft-based cluster to Zookeeper using the strimzi.io/kraft="annotation" on the Kafka resource of the cluster. Additionally, you remove KRaft controllers.
Rollbacks to ZooKeeper are only possible if the migration to KRaft is not finalized.
A migration is not finalized if the Kafka resource has the
strimzi.io/kraft="migration"
annotation and is in
KRaftPostMigration
metadata state. Clusters that have the
strimzi.io/kraft="enabled"
annotation and are in the
PreKraft
or KRaft
metadata state are finalized
and cannot be rolled back.