Add Streams Replication Manager to an existing cluster
Streams Replication Manager can be installed on an existing cluster managed by Cloudera Manager. To do this, you need to add Streams Replication Manager to the cluster and configure a number of mandatory properties related to clusters, replications, and role targets.
- Streams Replication Manager Driver role: This role is responsible for connecting to the specified clusters and performing replication between them. The driver can be installed on one or more hosts.
- Streams Replication Manager Service role: This role consist of a REST API and a Kafka Streams application to aggregate and expose cluster, topic and consumer group metrics. The service can be installed on one host only.
The following steps walk you through the process of adding Streams Replication Manager to your cluster. The configuration examples on this page are simple examples that are meant to demonstrate the type of information that you have to enter. For comprehensive deployment and configuration examples, see Deployment recommendations and Configuration examples.
-
If you are planning on replicating data to or from a Kafka service running in either a CDH 5.x or 6.x cluster and you are using Sentry for authorization, make sure that the
streamsrepmgr
user is added to the Kafka Super users property. You can find the Super users property by going to . Do this on all CDH 5.x or 6.x clusters where data replication will happen. - If you are planning on replicating data to or from a Kafka service running in Runtime
7.x and you are using Ranger for authorization, make sure that the
streamsrepmgr
user has all required permissions assigned to it in Ranger. Do this on all Runtime 7.x clusters where data replication will happen.
- Replicating data to or from the specified clusters is now possible.
- The SRM service REST API Swagger UI is available at one of the following addresses:
-
http://<srm-service-host>:<srm-service-port/swagger
-
https://<srm-service-host>:<srm-service-port/swagger
-
- Enable Kerberos and TLS/SSL for SRM.
- Use the
srm-control
tool to kick off replication by adding topics or groups to the allowlist.
If you plan to use Streams Messaging Manager (SMM) to monitor Kafka cluster replications, configure SMM to communicate with Streams Replication Manager (SRM). For information, see Configuring SMM for Monitoring SRM Replications.