Cloudera Docs

Migrating Kafka from ZooKeeper to KRaft

Learn about migrating an existing Zookeeper-based Kafka cluster to KRaft. Migration is performed in Cloudera Management Console, using the migration action available under the Cloudera Data Hub cluster management. Additionally, migration is done in a rolling fashion requiring no downtime.

Apache Kafka Raft (KRaft) is a consensus protocol used for metadata management that was developed as a replacement for Apache ZooKeeper. Using KRaft for managing Kafka metadata instead of ZooKeeper offers various benefits including a simplified architecture and a reduced operational footprint.

Migration at glance

Migration is completed in two steps, start and finalize, with the option to roll back at any time before finalization. You manage a migration (start, finalize, or rollback) using the migration action available under the Cloudera Data Hub cluster management in Cloudera Management Console. Configuration changes needed for the migration are automated by the migration actions.

The migration actions in Cloudera Data Hub are as follows:

  • Migrate Kafka to KRaft – Starts migration of Kafka brokers from ZooKeeper to KRaft for the specified Kafka service. Migrates the cluster up to a sate where rollback is still possible.

    When this action finishes, brokers will run in KRaft mode and will be disconnected from ZooKeeper. KRaft controllers will still be connected to ZooKeeper and will continue to write metadata to ZooKeeper, but are ready to disconnect (dual-write mode).

  • Finalize KRaft Migration – Finalizes migration by disconnecting KRaft controllers from ZooKeeper. Reverting to ZooKeeper is not possible once this action starts.

  • Rolling back KRaft Migration – Reverts migration of Kafka brokers from ZooKeeper to KRaft for the specified Kafka service using rolling restarts.

You can only start these actions if your cluster meets the required prerequisites. If prerequisites are not met, the actions will be disabled.

Migrating a Cloudera Streams Messaging cluster to KRaft

You migrate an existing ZooKeeper-based Cloudera Streams Messaging cluster to KRaft by using the migration feature available in Cloudera Management Console.

  • Migrating an existing ZooKeeper-based Cloudera Streams Messaging cluster KRaft is only available if you are on Cloudera Manager 7.13.2 or later and Cloudera Runtime 7.3.2.

    Migration is only possible with this combination of versions. This is because:

    • Earlier Cloudera Manager versions do not include the necessary Kafka service actions.

    • Cloudera Runtime 7.3.2 is the only version where migration is possible. Neither previous or future major, minor, and maintenance versions support migration.

  • The Kafka service must run with inter-broker protocol version 3.9.

    Verify the version by checking the value of the Kafka Inter-Broker Protocol Version property in Cloudera Manager > Kafka service > Configuration. The value of the property must be 3.9 or empty. An empty value means that version is set to the default, which is 3.9 in Cloudera Runtime 7.3.2.

  1. Navigate to your Streams Messaging cluster in Cloudera Management Console.
  2. Select Upgrade on the Data Hub details page.

    If you are on Cloudera Runtime 7.3.2, the migration to KRaft will appear.

  3. Click Start to confirm that you want to start the migration from ZooKeeper to KRaft.
    The migration remains in progress until a success message is displayed.When the migration is completed, you can either finalize the migration or roll back the Kafka service to the previous state.
  4. Finalize or Roll back the migration.
    1. Ensure that all applications and services connecting to Kafka operate as expected.
    2. Click Finalize.
    3. Click Finalize to confirm that you want to continue the migration. This action cannot be undone and rolling back to ZooKeeper will no longer be possible after the finalization.
    4. Check the status of the migration and if it shows Migration completed, the migration from ZooKeeper to KRaft was successful. Kafka is running in KRaft mode in your Cloudera Data Hub cluster.

      In case the finalization has failed, use the Retry Finalization button to start the process again.

    1. Click Roll back.
    2. Click Roll back to confirm that you want to revert back to ZooKeeper.You can reinitiate the migration to KRaft again after rolling back to ZooKeeper.
    3. Check the status of the migration and if the Start Migration button appears again, the rollback to ZooKeeper was successful. Kafka is running in ZooKeeper mode in your Cloudera Data Hub cluster.

      In case the rollback has failed, use the Retry Rollback button to start the process again.

ZooKeeper to KRaft migration for your Cloudera Streams Messaging cluster is completed.