Migrate Kafka Using Streams Replication Manager

Overview of the methods that can be used to migrate Kafka from HDF to CDP Private Cloud Base using Streams Replication Manager.

There are two approaches to using Streams Replication Manager (SRM) for data migration. You can either use SRM out of the box with its default replication policy, or configure it to use a custom one.

SRM replicates source topics to remote topics (replicated topics) on the target cluster. Because of how SRM’s default replication policy works, remote topics on the target cluster will have different names than the topics on the source cluster. The name references the source cluster that the topic was replicated from in the form of a prefix. For example, the topic1 topic from the us-west source cluster creates the us-west.topic1 remote topic on the target cluster. As a result, the replicated environment differs from the source environment.

Kafka does not support renaming topics. Therefore, reconfiguring existing Kafka clients to use the remote topic names is required once all topics are replicated.

However, in many cases this approach may not be suitable, therefore, an alternative method is also provided. This involves configuring SRM to use a custom replication policy, which does not change the remote topic names.

Review the following notes and decide which approach is better suited for your environment.

Migration with the default replication policy
  • SRM can be used out of the box.
  • Remote topics will have a different name in the target cluster.
  • Reconfiguration of all Kafka clients is required.
Migration with the custom replication policy
  • Advanced configuration of SRM to use an alternate replication policy is required.
  • Source and remote topics will have identical names.
  • Reconfiguration of all Kafka clients is not required.
  • Using an SRM service configured with the custom replication policy for any other scenario than data migration is not supported. Once migration is complete, the SRM instance you set up has to be reverted to use the default replication policy or deleted from the cluster.