Migrating Kafka clusters from ZooKeeper to KRaft

Learn about migrating an existing Zookeeper-based Kafka cluster to KRaft. Migration is semi-automatic and requires minimal configuration changes. Additionally, migration is done in a rolling fashion and requires no downtime.

Apache Kafka Raft (KRaft) is a consensus protocol used for metadata management that was developed as a replacement for Apache ZooKeeper. Using KRaft for managing Kafka metadata instead of ZooKeeper offers various benefits including a simplified architecture and a reduced operational footprint.

Starting with Cloudera Streams Messaging - Kubernetes Operator 1.3, ZooKeeper is deprecated and will be removed in a future release. Cloudera encourages you to migrate your existing clusters to KRaft.

Migration at glance

Migration is semi-automatic. You manage a migration (start, finalize, or roll back) using the strimzi.io/kraft annotation. You add the annotation to the Kafka resource of the cluster.

The majority of configuration changes needed in your cluster are handled by the Strimzi Cluster Operator when you add the strimzi.io/kraft annotation. Other than adding annotation, you are only required to deploy KRaft controllers before starting migration, and clean up unused configuration properties after migration. All other steps are automated.

The annotation has the following valid values.

migration - Starts the migration process and migrates a cluster up to a state where rollbacks are still possible.
enabled - Finalizes a migration. Once finalization is triggered, rollbacks to ZooKeeper are no longer possible.
rollback - Reverts the cluster from Kraft to ZooKeeper. Rollbacks are only possible for clusters where migration is not finalized.
disabled - The default value of the annotation. Indicates that the cluster uses ZooKeeper. Also used to finalize a rollback.

Cluster metadata states during migration

When you apply strimzi.io/kraft annotation to a Kafka resource, the Strimzi Cluster Operator performs various migration-related actions on the cluster. During a migration, the cluster goes through a number of metadata states. You can monitor these metadata states to see the progress of migration. The metadata state of the cluster is also an indicator that shows whether a cluster can be rolled back.

The metadata states that the cluster goes through during a migration and a rollback are as follows.

Migration

ZooKeeper - Indicates that the cluster uses ZooKeeper. This is the initial state. Migration is started and the cluster is moved to the next state when the strimzi.io/kraft="migration" annotation is added to the Kafka resource.
KRaftMigration - KRaft controllers are running, metadata and brokers are getting migrated to KRaft. In this state some brokers are already running in KRaft mode while others are still using ZooKeeper.
KRaftDualWriting - Metadata migration is finished. However, both brokers and controllers are still connected to ZooKeeper. Metadata is being written by KRaft controllers to ZooKeeper.
KRaftPostMigration - Brokers are running in KRaft mode and have been disconnected from ZooKeeper. KRaft controllers are connected to ZooKeeper and continue to write metadata to ZooKeeper, but are ready to disconnect.

This metadata state marks the end of the first migration phase. The Strimzi Cluster Operator only moves the cluster to the next state when you update the strimzi.io/kraft annotation to enabled. This is the last state where rollback is still possible.
PreKRaft - KRaft controllers are disconnected from ZooKeeper. Zookeeper is ready to be removed.
KRaft - Migration is finalized. Both Kafka brokers and controllers are running in KRaft mode. ZooKeeper is disconnected from the cluster. ZooKeeper instances are removed.

Rollback

KRaftPostMigration - Brokers are running in KRaft mode and have been disconnected from ZooKeeper. KRaft controllers are connected to ZooKeeper and continue to write metadata to ZooKeeper, but are ready to disconnect. This is the initial state when you begin a rollback.
KRaftDualWriting - Both brokers and controllers are connected to ZooKeeper. Metadata is being written by KRaft controllers to ZooKeeper as well. In case of a rollback, the Strimzi Cluster Operator stops at this metadata state and waits until you start finalization by setting the strimzi.io/kraft annotation to disabled.
ZooKeeper - Indicates that the cluster uses ZooKeeper. This is the final state during a rollback.

Migrating a Kafka cluster to KRaft

You migrate an existing ZooKeeper-based Kafka cluster to KRaft by creating KRaft controllers and then using the strimzi.io/kraft annotation on the Kafka resource of the cluster to initiate and finalize the migration. After finalization, you remove configuration related to ZooKeeper.

Ensure the following:

You are on Cloudera Streaming Kubernetes Operator 1.3 or higher.
The Kafka cluster version is 3.9.0.1.3 or higher.

Migration is not supported from lower versions. If you are on a lower version, upgrade to a supported version.

Create a YAML configuration containing your KafkaNodePool resource manifest for KRaft controllers.
The KafkaNodePool configuration must have spec.roles set to controller. Migrating with a Kafka node in combined mode (has both broker and controller roles) is not supported. Ensure that you only set controller as the role.
Deploy as many controllers as you have ZooKeeper nodes. Additionally, Cloudera strongly recommends that you deploy an uneven number of controllers.
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
```

Deploy the KafkaNodePool resource.

This deploys the KRaft controllers.

kubectl apply \
  --filename [***NODE POOL YAML***] \
  --namespace [***NAMESPACE***]

Start migration by annotating your Kafka resource with strimzi.io/kraft="migration".

kubectl annotate kafka [***CLUSTER NAME***] \
  --namespace [***NAMESPACE***] \
  --overwrite \
  strimzi.io/kraft="migration"

Monitor the migration.
```
kubectl get kafka [***CLUSTER NAME***] --namespace [***NAMESPACE***] --watch
```
Monitor the METADATA STATE column. The cluster goes through multiple metadata states during migration. This phase of the migration is complete when the cluster is in the KRaftPostMigration state.
```
NAME         #...   READY   METADATA STATE       WARNINGS
my-cluster   #...   True    KRaftPostMigration   True
```
note
If the Kafka resource includes the inter.broker.protocol.version or log.message.format.version properties, the Strimzi Cluster Operator reports warnings (WARNINGS column is true). This is because neither properties are used or supported by Kafka when it is running in KRaft mode. This is normal and expected behavior. You will be removing these properties as when you finalize migration.
Finalize the migration.

important
Ensure that your cluster is in a healthy and working state before continuing. Once you finalize migration, you cannot rollback your cluster to use ZooKeeper. For rollback instructions skip to Rolling back a cluster to ZooKeeper.
1. Annotate the Kafka resource with strimzi.io/kraft="enabled".
  This annotation starts the finalization process and removes ZooKeeper pods.
```
kubectl annotate kafka [***KAFKA NAME***] \
  --namespace [***NAMESPACE***] \
  --overwrite \
  strimzi.io/kraft="enabled"
```
2. Monitor finalization.
```
kubectl get kafka [***CLUSTER NAME***] --namespace [***NAMESPACE***] --watch
```
  Monitor the METADATA STATE column. The cluster goes through multiple metadata states during finalization. Migration is finalized when the cluster is in the KRaft state.
```
NAME         #...   READY   METADATA STATE   WARNINGS
my-cluster   #...   True    KRaft            True
```
3. Remove inter.broker.protocol.version and log.message.format.version properties from your Kafka resource.
4. Remove spec.zookeeper from your Kafka resource.
  ZooKeeper is no longer in use for the cluster, keeping its configuration is not necessary.

Rolling back a Kafka cluster to ZooKeeper

You roll back a KRaft-based cluster to Zookeeper using the strimzi.io/kraft="annotation" on the Kafka resource of the cluster. Additionally, you remove KRaft controllers.

Rollbacks to ZooKeeper are only possible if the migration to KRaft is not finalized. A migration is not finalized if the Kafka resource has the strimzi.io/kraft="migration" annotation and is in KRaftPostMigration metadata state. Clusters that have the strimzi.io/kraft="enabled" annotation and are in the PreKraft or KRaft metadata state are finalized and cannot be rolled back.

Start the rollback by annotating your Kafka resource with strimzi.io/kraft="rollback".

kubectl annotate kafka [***CLUSTER NAME***] \
  --namespace [***NAMESPACE***] \]
  --overwrite \
  strimzi.io/kraft="rollback"

Monitor the rollback.
```
kubectl get kafka [***KAFKA NAME***] --namespace [***NAMESPACE***] --watch
```
Monitor the METADATA STATE column. This stage of the rollback is finished once the cluster reverts to the KRaftDualWriting metadata state.
```
NAME         #...   READY   METADATA STATE       WARNINGS
my-cluster   #...   True    KRaftDualWriting     True
```
note
The Strimzi Cluster Operator logs a warning notifying you that the strimzi.io/kraft annotation cannot be set to rollback if the cluster metadata state is KRaftDualWriting. This is normal and expected behavior. You update this annotation as part of the finalization process.
Finalize the rollback.
1. Annotate your Kafka resource with strimzi.io/kraft="disabled".
```
kubectl annotate kafka [***CLUSTER NAME***] \
  --namespace [***NAMESPACE***] \
  --overwrite \
  strimzi.io/kraft="disabled"
```
  note
  The state of the Kafka resource will not change to ready. This is because a corresponding KafkaNodePool resource has spec.roles set to controller, which is invalid for a ZooKeeper-based cluster. This is normal and expected behavior. You remove invalid node pools in a following step.
2. Delete the KafkaNodePool resource you created for your KRaft controllers.
```
kubectl delete kafkanodepool [***NODE POOL NAME***] --namespace [***NAMESPACE***]
```
3. Monitor rollback finalization.
```
kubectl get kafka [***CLUSTER NAME***] --namespace [***NAMESPACE***] --watch
```
  The rollback is finalized once the Kafka cluster is in the ZooKeeper metadata state.
```
NAME         #...   READY   METADATA STATE    WARNINGS
my-cluster   #...   True    ZooKeeper         True
```