Known Issues in Streams Replication Manager
Learn about the known issues in Streams Replication Manager, the impact or changes to the functionality, and the workaround.
- CDPD-22089: SRM does not sync re-created source topics until the offsets have caught up with target topic
- Messages written to topics that were deleted and re-created are not replicated until the source topic reaches the same offset as the target topic. For example, if at the time of deletion and re-creation there are a 100 messages on the source and target clusters, new messages will only get replicated once the re-created source topic has 100 messages. This leads to messages being lost.
- CDPD-14019: SRM may automatically re-create deleted topics
auto.create.topics.enableis enabled, deleted topics are automatically recreated on source clusters.
- Prior to deletion, remove the topic from the topic allowlist
srm-controltool. This prevents topics from being re-created.
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --remove [TOPIC1][TOPIC2]
- CDPD-13864 and CDPD-15327: Replication stops after the network configuration of a source or target cluster is changed
- If the network configuration of a cluster which is taking part in a replication flow is changed, for example, port numbers are changed as a result of enabling or disabling TLS, SRM will not update its internal configuration even if SRM is reconfigured and restarted. From SRM’s perspective, it is the cluster identity that has changed. SRM cannot determine whether the new identity corresponds to the same cluster or not, only the owner or administrator of that cluster can know. In this case, SRM tries to use the last known configuration of that cluster which might not be valid, resulting in the halt of replication.
- There are three workarounds for this issue. Choose one of the following:
- Increase the driver rebalance timeout
Increasing the rebalance timeout to 5 minutes (300000 ms) or longer can resolve the issue. In general a 5 minute timeout should be sufficient for most deployments. However, depending on your scenario, an even longer period might be required. Increasing the rebalance timeout might lead to increased latency when the SRM drivers stop. The cluster will be slower when it rebalances the load of the removed driver.The rebalance timeout can be configured on a per cluster (alias) basis by adding the following to the Streams Replication Manager’s Replication Configs Cloudera Manager property:
Replace [***ALIAS***] with a cluster alias specified in Streams Replication Manager Cluster alias. Do this for all clusters that are taking part in the replication process. When correctly configured, your configuration will have a
[***ALIAS***].rebalance.timeout.ms = [***VALUE***]
rebalance.timeout.msentry corresponding to each cluster (alias). For example:
After the new broker configuration is applied by SRM, the rebalance timeout can be reverted back to its original value, or removed from the configuration altogether.
primary.rebalance.timeout.ms = 30000 secondary.rebalance.timeout.ms = 30000 tertiary.rebalance.timeout.ms = 30000
- Decrease replication admin timeout
Decreasing the replication admin timeout to 15 seconds (15000 ms) can resolve the issue. With higher loads, this might cause WARN messages to appear in the SRM driver log.The admin timeout can be configured on a per replication basis by adding the following to the Streams Replication Manager’s Replication Configs Cloudera Manager property:
Replace [***REPLICATION***] with a replication specified in Streams Replication Manager’s Replication Configs. Do this for all affected replications. When correctly configured, your configuration will have an
[***REPLICATION***].admin.timeout.ms = [***VALUE***]
admin.timeout.msentry corresponding to each affected replication. For example:
After the new broker configuration is applied by SRM, the admin timeout can be reverted back to its original value, or removed from the configuration altogether.
primary->secondary.admin.timeout.ms = 15000 secondary->primary.admin.timeout.ms = 15000
- Upgrade the brokers incrementally
- Instead of switching over to the new configuration, open two separate listeners on the broker. One for the old configuration, and one for the new configuration. After updating SRM's configuration and restarting SRM, the old listener can be turned off. Non–inter-broker listeners can be configured with the dynamic configuration API of Kafka, this way not every listener change has to be followed by a restart.
- CDPD-11079: Blacklisted topics appear in the list of replicated topics
- If a topic was originally replicated but was later disallowed
(blacklisted), it will still appear as a replicated topic under the
/remote-topicsREST API endpoint. As a result, if a call is made to this endpoint, the disallowed topic will be included in the response. Additionally, the disallowed topic will also be visible in the SMM UI. However, it's Partitions and Consumer Groups will be 0, its Throughput, Replication Latency and Checkpoint Latency will show N/A.
- SRM cannot replicate Ranger authorization policies to or from Kafka clusters
- Due to a limitation in the Kafka-Ranger plugin, SRM cannot
replicate Ranger policies to or from clusters that are configured to use Ranger for
authorization. If you are using SRM to replicate data to or from a cluster that uses
Ranger, disable authorization policy synchronization in SRM. This can be achieved by
clearing the Sync Topic Acls Enabled