Upgrade & MigrationPDF version

Migrating Kafka clusters from ZooKeeper to KRaft

Learn about migrating an existing Zookeeper-based Kafka cluster to KRaft. Migration is semi-automatic and requires minimal configuration changes. Additionally, migration is done in a rolling fashion and requires no downtime.

Apache Kafka Raft (KRaft) is a consensus protocol used for metadata management that was developed as a replacement for Apache ZooKeeper. Using KRaft for managing Kafka metadata instead of ZooKeeper offers various benefits including a simplified architecture and a reduced operational footprint.

Starting with Cloudera Streams Messaging - Kubernetes Operator 1.3, ZooKeeper is deprecated and will be removed in a future release. Cloudera encourages you to migrate your existing clusters to KRaft.

Migration is semi-automatic. You manage a migration (start, finalize, or roll back) using the strimzi.io/kraft annotation. You add the annotation to the Kafka resource of the cluster.

The majority of configuration changes needed in your cluster are handled by the Strimzi Cluster Operator when you add the strimzi.io/kraft annotation. Other than adding annotation, you are only required to deploy KRaft controllers before starting migration, and clean up unused configuration properties after migration. All other steps are automated.

The annotation has the following valid values.

  • migration - Starts the migration process and migrates a cluster up to a state where rollbacks are still possible.

  • enabled - Finalizes a migration. Once finalization is triggered, rollbacks to ZooKeeper are no longer possible.

  • rollback - Reverts the cluster from Kraft to ZooKeeper. Rollbacks are only possible for clusters where migration is not finalized.

  • disabled - The default value of the annotation. Indicates that the cluster uses ZooKeeper. Also used to finalize a rollback.

When you apply strimzi.io/kraft annotation to a Kafka resource, the Strimzi Cluster Operator performs various migration-related actions on the cluster. During a migration, the cluster goes through a number of metadata states. You can monitor these metadata states to see the progress of migration. The metadata state of the cluster is also an indicator that shows whether a cluster can be rolled back.

The metadata states that the cluster goes through during a migration and a rollback are as follows.

Migration
  • ZooKeeper - Indicates that the cluster uses ZooKeeper. This is the initial state. Migration is started and the cluster is moved to the next state when the strimzi.io/kraft="migration" annotation is added to the Kafka resource.

  • KRaftMigration - KRaft controllers are running, metadata and brokers are getting migrated to KRaft. In this state some brokers are already running in KRaft mode while others are still using ZooKeeper.

  • KRaftDualWriting - Metadata migration is finished. However, both brokers and controllers are still connected to ZooKeeper. Metadata is being written by KRaft controllers to ZooKeeper.

  • KRaftPostMigration - Brokers are running in KRaft mode and have been disconnected from ZooKeeper. KRaft controllers are connected to ZooKeeper and continue to write metadata to ZooKeeper, but are ready to disconnect.

    This metadata state marks the end of the first migration phase. The Strimzi Cluster Operator only moves the cluster to the next state when you update the strimzi.io/kraft annotation to enabled. This is the last state where rollback is still possible.

  • PreKRaft - KRaft controllers are disconnected from ZooKeeper. Zookeeper is ready to be removed.

  • KRaft - Migration is finalized. Both Kafka brokers and controllers are running in KRaft mode. ZooKeeper is disconnected from the cluster. ZooKeeper instances are removed.

Rollback
  • KRaftPostMigration - Brokers are running in KRaft mode and have been disconnected from ZooKeeper. KRaft controllers are connected to ZooKeeper and continue to write metadata to ZooKeeper, but are ready to disconnect. This is the initial state when you begin a rollback.

  • KRaftDualWriting - Both brokers and controllers are connected to ZooKeeper. Metadata is being written by KRaft controllers to ZooKeeper as well. In case of a rollback, the Strimzi Cluster Operator stops at this metadata state and waits until you start finalization by setting the strimzi.io/kraft annotation to disabled.

  • ZooKeeper - Indicates that the cluster uses ZooKeeper. This is the final state during a rollback.

You migrate an existing ZooKeeper-based Kafka cluster to KRaft by creating KRaft controllers and then using the strimzi.io/kraft annotation on the Kafka resource of the cluster to initiate and finalize the migration. After finalization, you remove configuration related to ZooKeeper.

Ensure the following:

  • You are on Cloudera Streaming Kubernetes Operator 1.3 or higher.

  • The Kafka cluster version is 3.9.0.1.3 or higher.

Migration is not supported from lower versions. If you are on a lower version, upgrade to a supported version.

  1. Create a YAML configuration containing your KafkaNodePool resource manifest for KRaft controllers.
    The KafkaNodePool configuration must have spec.roles set to controller. Migrating with a Kafka node in combined mode (has both broker and controller roles) is not supported. Ensure that you only set controller as the role.

    Deploy as many controllers as you have ZooKeeper nodes. Additionally, Cloudera strongly recommends that you deploy an uneven number of controllers.

    apiVersion: kafka.strimzi.io/v1beta2
    kind: KafkaNodePool
    metadata:
      name: controller
      labels:
        strimzi.io/cluster: my-cluster
    spec:
      replicas: 3
      roles:
        - controller
      storage:
        type: jbod
        volumes:
          - id: 0
            type: persistent-claim
            size: 100Gi
            deleteClaim: false
  2. Deploy the KafkaNodePool resource.
    This deploys the KRaft controllers.
    kubectl apply \
      --filename [***NODE POOL YAML***] \
      --namespace [***NAMESPACE***]
  3. Start migration by annotating your Kafka resource with strimzi.io/kraft="migration".
    kubectl annotate kafka [***CLUSTER NAME***] \
      --namespace [***NAMESPACE***] \
      --overwrite \
      strimzi.io/kraft="migration"
  4. Monitor the migration.
    kubectl get kafka [***CLUSTER NAME***] --namespace [***NAMESPACE***] --watch
    

    Monitor the METADATA STATE column. The cluster goes through multiple metadata states during migration. This phase of the migration is complete when the cluster is in the KRaftPostMigration state.

    NAME         #...   READY   METADATA STATE       WARNINGS
    my-cluster   #...   True    KRaftPostMigration   True
  5. Finalize the migration.
    1. Annotate the Kafka resource with strimzi.io/kraft="enabled".
      This annotation starts the finalization process and removes ZooKeeper pods.
      kubectl annotate kafka [***KAFKA NAME***] \
        --namespace [***NAMESPACE***] \
        --overwrite \
        strimzi.io/kraft="enabled"
    2. Monitor finalization.
      kubectl get kafka [***CLUSTER NAME***] --namespace [***NAMESPACE***] --watch

      Monitor the METADATA STATE column. The cluster goes through multiple metadata states during finalization. Migration is finalized when the cluster is in the KRaft state.

      NAME         #...   READY   METADATA STATE   WARNINGS
      my-cluster   #...   True    KRaft            True
      
    3. Remove inter.broker.protocol.version and log.message.format.version properties from your Kafka resource.
    4. Remove spec.zookeeper from your Kafka resource.
      ZooKeeper is no longer in use for the cluster, keeping its configuration is not necessary.

You roll back a KRaft-based cluster to Zookeeper using the strimzi.io/kraft="annotation" on the Kafka resource of the cluster. Additionally, you remove KRaft controllers.

Rollbacks to ZooKeeper are only possible if the migration to KRaft is not finalized. A migration is not finalized if the Kafka resource has the strimzi.io/kraft="migration" annotation and is in KRaftPostMigration metadata state. Clusters that have the strimzi.io/kraft="enabled" annotation and are in the PreKraft or KRaft metadata state are finalized and cannot be rolled back.

  1. Start the rollback by annotating your Kafka resource with strimzi.io/kraft="rollback".
    kubectl annotate kafka [***CLUSTER NAME***] \
      --namespace [***NAMESPACE***] \]
      --overwrite \
      strimzi.io/kraft="rollback" 
  2. Monitor the rollback.
    kubectl get kafka [***KAFKA NAME***] --namespace [***NAMESPACE***] --watch

    Monitor the METADATA STATE column. This stage of the rollback is finished once the cluster reverts to the KRaftDualWriting metadata state.

    NAME         #...   READY   METADATA STATE       WARNINGS
    my-cluster   #...   True    KRaftDualWriting     True
  3. Finalize the rollback.
    1. Annotate your Kafka resource with strimzi.io/kraft="disabled".
      kubectl annotate kafka [***CLUSTER NAME***] \
        --namespace [***NAMESPACE***] \
        --overwrite \
        strimzi.io/kraft="disabled"
      
    2. Delete the KafkaNodePool resource you created for your KRaft controllers.
      kubectl delete kafkanodepool [***NODE POOL NAME***] --namespace [***NAMESPACE***]
    3. Monitor rollback finalization.
      kubectl get kafka [***CLUSTER NAME***] --namespace [***NAMESPACE***] --watch

      The rollback is finalized once the Kafka cluster is in the ZooKeeper metadata state.

      NAME         #...   READY   METADATA STATE    WARNINGS
      my-cluster   #...   True    ZooKeeper         True

We want your opinion

How can we improve this page?

What kind of feedback do you have?