Migrating Streaming Workloads to Cloudera Private CloudPDF version

Migrate Kafka Using the IdentityReplicationPolicy

How to migrate Kafka with Streams Replication Manager using the IdentityReplicationPolicy.

  • Ensure that you have reviewed Migrate Kafka Using Streams Replication Manager and understand the limitations and use cases for this policy.
  • Setup and configure Streams Replication Manager in the Cloudera Private Cloud Base cluster. For more information, see Add and Configure Streams Replication Manager. Setup instructions might differ in different versions. Ensure that you view the version of the documentation that matches your Runtime version.
  • Ensure that Streams Replication Manager is set up to use the IdentityReplicationPolicy. This is done differently depending on the Cloudera Runtime version.
    • 7.1.8: In Cloudera Manager, add the following entry to the Streams Replication Manager's Replication Configs property.
      replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy
    • 7.1.9 or higher: In Cloudera Manager, select the Enable Prefixless Replication property. This property configures Streams Replication Manager to use the IdentityReplicationPolicy.
  • If you are on Cloudera Runtime 7.1.8, monitoring features provided by the Streams Replication Manager Service are not supported when this policy is in use. This means that you cannot use the Streams Messaging Manager UI or Streams Replication Manager Service REST API to validate that data is being migrated.
  1. Use the srm-control tool to include every topic and every consumer group in the allowlist.
    Including consumer groups in the allowlist is required for checkpointing.
    srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"
    srm-control groups --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"
  2. Validate that data is being migrated.
    Use the Cluster Replications page on the Streams Messaging Manager UI to monitor and validate the status of the migration. Alternatively, you can use the various endpoints available on the Streams Replication Manager Service REST API. Doing this is only possible if you are on Cloudera Runtime 7.1.9 or higher.
  3. Stop producers.
  4. Stop consumers.
  5. Reconfigure all consumers to read from Cloudera Private Cloud Base Kafka and apply offset translation using Streams Replication Manager.
  6. Start consumers.
  7. Reconfigure all producers to write to Cloudera Private Cloud Base Kafka.

    The HDF instances of Kafka and Streams Messaging Manager are no longer required.

  8. Start producers.
Kafka is migrated. Kafka clients produce and consume from the Cloudera Private Cloud Base cluster. Migration is complete.