Cloudera Docs

Migrate Kafka Using the IdentityReplicationPolicy

How to migrate Kafka with Streams Replication Manager (SRM) using the IdentityReplicationPolicy.

  • Ensure that you have reviewed Migrate Kafka Using Streams Replication Manager and understand the limitations and use cases for this policy.
  • Setup and configure Streams Replication Manager in the CDP Private Cloud Base cluster. For more information, see Add and Configure SRM. Setup instructions might differ in different versions. Ensure that you view the version of the documentation that matches your Runtime version.
  • Ensure that SRM is set up to use the IdentityReplicationPolicy. This is done differently depending on the Cloudera Runtime version.
    • 7.1.8: In Cloudera Manager, add the following entry to the Streams Replication Manager's Replication Configs property.
      replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy
    • 7.1.9 or higher: In Cloudera Manager, select the Enable Prefixless Replication property. This property configures SRM to use the IdentityReplicationPolicy.
  • If you are on Cloudera Runtime 7.1.8, monitoring features provided by the SRM Service are not supported when this policy is in use. This means that you cannot use the SMM UI or SRM Service REST API to validate that data is being migrated.
  1. Use the srm-control tool to include every topic and every consumer group in the allowlist.
    Including consumer groups in the allowlist is required for checkpointing.
    srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"
    srm-control groups --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"
  2. Validate that data is being migrated.
    Use the Cluster Replications page on the SMM UI to monitor and validate the status of the migration. Alternatively, you can use the various endpoints available on the SRM Service REST API. Doing this is only possible if you are on Cloudera Runtime 7.1.9 or higher.
  3. Stop producers.
  4. Stop consumers.
  5. Reconfigure all consumers to read from CDP Private Cloud Base Kafka and apply offset translation using SRM.
  6. Start consumers.
  7. Reconfigure all producers to write to CDP Private Cloud Base Kafka.

    The HDF instances of Kafka and SMM are no longer required.

  8. Start producers.
Kafka is migrated. Kafka clients produce and consume from the CDP Private Cloud Base cluster. Migration is complete.