Migrate Kafka Using the Default Replication Policy

How to Migrate Kafka with Streams Replication Manager using the default replication policy.

You can use Streams Replication Manager (SRM) out of the box with its default replication policy to migrate Kafka data from a HDF cluster to a CDP Private Cloud Base cluster. With this method, the remote topics in the target cluster will have different names. As a result, the replicated environment will differ from the source environment. You will need to make significant changes to all Kafka producers and consumers otherwise they will not be able to connect to the correct topics in the CDP Private Cloud Base cluster.

Setup and Configure Streams Replication Manager in the CDP Private Cloud Base cluster. For more information, see Add and Configure SRM and the SRM Configuration Examples in the SRM documentation for Runtime.

  1. Use the srm-control tool to whitelist every topic and every consumer group.

    Whitelisting consumer groups is required for offset translation.

    srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"
    
    srm-control groups --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"
  2. Validate that data is being migrated.

    Use the Cluster Replications page on the Streams Messaging Manager (SMM) UI to monitor and validate the status of the migration.

  3. Stop producers.
  4. Stop consumer.
  5. Reconfigure all consumers to read from CDP Private Cloud Base Kafka and apply offset translation using SRM.
  6. Start consumers.
  7. Reconfigure all producers to write to CDP Private Cloud Base Kafka.

    The HDF instances of Kafka and SMM are no longer required.

  8. Start producers.

Kafka is migrated. Kafka clients produce and consume from the CDP Private Cloud Base cluster. Migration is complete.