Cloudera Docs

Migrate Kafka Using the DefaultReplicationPolicy

How to migrate Kafka with Streams Replication Manager (SRM) using the DefaultReplicationPolicy.

  • Be aware that remote topics on the source cluster are renamed during the migration.

    As a result, you must make significant changes to all Kafka producers and consumers. Otherwise, they will not be able to connect to the correct topics in the CDP Private Cloud Base cluster. If this behavior is not suitable, review Migrate Kafka Using Streams Replication Manager and use the IdentityReplicationPolicy or the MigratingReplicationPolicy instead.

  • Setup and Configure SRM in the CDP Private Cloud Base cluster.

    For more information, see Add and Configure SRM and the SRM Configuration Examples. Setup instructions might differ in different versions, ensure that you view the version of the documentation that matches your Runtime version.

  1. Use the srm-control tool to include every topic and every consumer group.

    Including consumer groups is required for checkpointing.

    srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"

    srm-control groups --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"
  2. Validate that data is being migrated.

    Use the Cluster Replications page on the Streams Messaging Manager (SMM) UI to monitor and validate the status of the migration. Alternatively, you can use the various endpoints available on the SRM Service REST API.

  3. Wait until replication is caught up.
  4. Stop producers.
  5. Stop consumers.
  6. Reconfigure all consumers to read from CDP Private Cloud Base Kafka and apply offset translation using SRM.
  7. Start consumers.
  8. Reconfigure all producers to write to CDP Private Cloud Base Kafka.

    The HDF instances of Kafka and SMM are no longer required.

  9. Start producers.

Kafka is migrated. Kafka clients produce and consume from the CDP Private Cloud Base cluster. Migration is complete.