Known Issues in Streams Replication Manager
Learn about the known issues in Streams Replication Manager, the impact or changes to the functionality, and the workaround.
- CDPD-22089: SRM does not sync re-created source topics until the offsets have caught up with target topic
- Messages written to topics that were deleted and re-created are not replicated until the source topic reaches the same offset as the target topic. For example, if at the time of deletion and re-creation there are a 100 messages on the source and target clusters, new messages will only get replicated once the re-created source topic has 100 messages. This leads to messages being lost.
- CDPD-14019: SRM may automatically re-create deleted topics
auto.create.topics.enableis enabled, deleted topics are automatically recreated on source clusters.
- Prior to deletion, remove the topic from the topic
allowlist with the
srm-controltool. This prevents topics from being re-created.
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --remove [TOPIC1][TOPIC2]
- CDPD-11079: Blacklisted topics appear in the list of replicated topics
- If a topic was originally replicated but was later disallowed
(blacklisted), it will still appear as a replicated topic under the
/remote-topicsREST API endpoint. As a result, if a call is made to this endpoint, the disallowed topic will be included in the response. Additionally, the disallowed topic will also be visible in the SMM UI. However, it's Partitions and Consumer Groups will be 0, its Throughput, Replication Latency and Checkpoint Latency will show N/A.
- CDPD-60823: Configuring the SRM Client's secure storage is mandatory for unsecured environments
- In an unsecured environment the
srm-controltool should not need any additional configuration to run. However, due to an issue with the automatic generation of the default configuration, configuring the SRM Client's secure storage is mandatory for the
srm-controltool. This is true even if none of the clusters that the tool connects to are secured.If a secure storage is not configured, the tool will fail with the following NullPointerException:
java.lang.NullPointerException at com.cloudera.dim.mirror.SecureConfigProvider.retrievePassword(SecureConfigProvider.java:99) at com.cloudera.dim.mirror.SecureConfigProvider.configure(SecureConfigProvider.java:113) at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:533) at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:477) at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:107) at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142) at org.apache.kafka.connect.mirror.MirrorMakerConfig.<init>(MirrorMakerConfig.java:88) at com.cloudera.dim.mirror.MirrorControlCommand$SourceTargetCommand.init(MirrorControlCommand.java:97) at com.cloudera.dim.mirror.MirrorControlCommand.issueCommand(MirrorControlCommand.java:369) at com.cloudera.dim.mirror.MirrorControlCommand.main(MirrorControlCommand.java:346)
- Configure a secure storage password and set it as an
environment variable in your CLI session before running the
- In Cloudera Manager, select the Streams Replication Manager service.
- Go to Configuration.
- Find and configure the SRM Client's Secure Storage Password
Take note of the password that you configure.
Click Save changes.
- Restart the SRM service
- SSH into one of the SRM hosts in your cluster.
- Set the secure storage password as an environment
Replace [***SECURE STORAGE ENV VAR***] with the name of the environment variable you specified in Environment Variable Holding SRM Client's Secure Storage Password. Replace [***SRM SECURE STORAGE PASSWORD***] with the password you specified in SRM Client's Secure Storage Password. For example:
export [***SECURE STORAGE ENV VAR***]=”[***SECURE STORAGE PASSWORD***]”
- OPSAPS-61001: Saving configuration changes for SRM is not possible
- Cloudera Manager incorrectly labels the SRM Client's
Secure Storage Password property as mandatory. Moreover, it does not offer
this property for configuration when SRM is installed with the Add Service
As a result, it is possible to install and start SRM without configuring this property. However, in a case like this, making changes to SRM's configuration is not possible until the SRM Client's Secure Storage Password property is set.
- Configure the SRM Client's Secure Storage Password property.
- SRM cannot replicate Ranger authorization policies to or from Kafka clusters
- Due to a limitation in the Kafka-Ranger plugin, SRM cannot
replicate Ranger policies to or from clusters that are configured to use Ranger for
authorization. If you are using SRM to replicate data to or from a cluster that uses
Ranger, disable authorization policy synchronization in SRM. This can be achieved by
clearing the Sync Topic Acls Enabled