Known Issues in Streams Replication Manager

Learn about the known issues in Streams Replication Manager, the impact or changes to the functionality, and the workaround.

Known Issues

MM2-163: SRM does not sync re-created source topics until the offsets have caught up with target topic: Messages written to topics that were deleted and re-created are not replicated until the source topic reaches the same offset as the target topic. For example, if at the time of deletion and re-creation there are a 100 messages on the source and target clusters, new messages will only get replicated once the re-created source topic has 100 messages. This leads to messages being lost.; None

CDPD-14019: SRM may automatically re-create deleted topics: If auto.create.topics.enable is enabled, deleted topics are automatically recreated on source clusters.; Prior to deletion, remove the topic from the topic allowlist with the srm-control tool. This prevents topics from being re-created.
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --remove [TOPIC1][TOPIC2]

CSP-462: Replication failing when SRM driver is present on multiple nodes: Kafka replication fails when the SRM driver is installed on more than one node.; None

CDPD-11074: The srm-control tool can be called without --target: The srm-control tool can be initialized without specifying the --target option. If the tool is called this way it will fail to run correctly.; Do not use the tool without specifying the --target option. Always specify both --source and --target options. For example:
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --list

CDPD-13864 and CDPD-15327: Replication stops after the network configuration of a source or target cluster is changed

If the network configuration of a cluster which is taking part in a replication is changed, for example, port numbers are changed as a result of enabling or disabling TLS, SRM will not update its internal configuration even if SRM is reconfigured and restarted. From SRM’s perspective, it is the cluster identity that has changed. SRM cannot determine whether the new identity corresponds to the same cluster or not, only the owner or administrator of that cluster can know. In this case, SRM tries to use the last known configuration of that cluster which might not be valid, resulting in the halt of replication.

There are three workarounds for this issue. Choose one of the following:

Increase the driver rebalance timeout

Increasing the rebalance timeout to 5 minutes (300000 ms) or longer can resolve the issue. In general a 5 minute timeout should be sufficient for most deployments. However, depending on your scenario, an even longer period might be required. Increasing the rebalance timeout might lead to increased latency when the SRM drivers stop. The cluster will be slower when it rebalances the load of the removed driver.

The rebalance timeout can be configured on a per cluster (alias) basis by adding the following to the Streams Replication Manager’s Replication Configs Cloudera Manager property:

[***ALIAS***].rebalance.timeout.ms = [***VALUE***]

Replace [***ALIAS***] with a cluster alias specified in Streams Replication Manager Cluster alias. Do this for all clusters that are taking part in the replication process. When correctly configured, your configuration will have a rebalance.timeout.ms entry corresponding to each cluster (alias). For example:

primary.rebalance.timeout.ms = 30000
secondary.rebalance.timeout.ms = 30000
tertiary.rebalance.timeout.ms = 30000

After the new broker configuration is applied by SRM, the rebalance timeout can be reverted back to its original value, or removed from the configuration altogether.

Decrease replication admin timeout

Decreasing the replication admin timeout to 15 seconds (15000 ms) can resolve the issue. With higher loads, this might cause WARN messages to appear in the SRM driver log.

The admin timeout can be configured on a per replication basis by adding the following to the Streams Replication Manager’s Replication Configs Cloudera Manager property:

[***REPLICATION***].admin.timeout.ms = [***VALUE***]

Replace [***REPLICATION***] with a replication specified in Streams Replication Manager’s Replication Configs. Do this for all affected replications. When correctly configured, your configuration will have an admin.timeout.ms entry corresponding to each affected replication. For example:

primary->secondary.admin.timeout.ms = 15000
secondary->primary.admin.timeout.ms = 15000

After the new broker configuration is applied by SRM, the admin timeout can be reverted back to its original value, or removed from the configuration altogether.

Upgrade the brokers incrementally

Instead of switching over to the new configuration, open two separate listeners on the broker. One for the old configuration, and one for the new configuration. After updating SRM's configuration and restarting SRM, the old listener can be turned off. Non–inter-broker listeners can be configured with the dynamic configuration API of Kafka, this way not every listener change has to be followed by a restart.

CDPD-11709: Blacklisted topics appear in the list of replicated topics: If a topic was originally replicated but was later excluded for replication, it will still appear as a replicated topic under the /remote-topics REST API endpoint. As a result, if a call is made to this endpoint, this topic will be included in the response. Additionally, the excluded topic will also be visible in the SMM UI. However, it's Partitions and Consumer Groups will be 0, its Throughput, Replication Latency and Checkpoint Latency will show N/A.; None

CDPD-18300: SRM resolves configuration provider references in its internal configuration topic: SRM saves its internal configuration topic with fully resolved properties. This means that even configuration provider references are resolved. Sensitive information can be emitted into the configuration topic this way.; None

CDPD-22094: The SRM service role displays as healthy, but no metrics are processed: The SRM service role might encounter errors that make metrics processing impossible. An example of this is when the target Kafka cluster is not reachable. The SRM service role does not automatically stop or recover if such an error is encountered. It continues to run and displays as healthy in Cloudera Manager. Metrics, however, are not processed. In addition, no new data is displayed in SMM for the replications.; Ensure that all clusters are available and are in a healthy state.

Restart SRM.

CDPD-22389: The SRM driver role displays as healthy, but replication fails: During startup, the SRM driver role might encounter errors that make data replication impossible. An example of this is when one of the clusters added for replication is not reachable. The SRM driver role does not automatically stop or recover if such an error is encountered. It will start up, continue to run, and display as healthy in Cloudera Manager. Replication, however, will not happen.; Ensure that all clusters are available and are in a healthy state.

Restart SRM.

CDPD-23683: The replication status reported by the SRM service role for healthy replications is flaky: The replication status reported by the SRM service role is flaky. The replication status might change between active and inactive frequently even if the replication is healthy. This status is also reflected in SMM on the replications tab.; None
OPSAPS-59124: Kafka, SMM, and SRM fail to start when there are multiple Ranger Admin roles running: If there are multiple Ranger Admin roles configured in a cluster, Kafka cannot create the Kafka resource based services in Ranger, which are required for authorization. If the resource based services are missing, the Kafka, SMM, and SRM services will fail to start.; In Cloudera Manager, select the Kafka service.

Go to Configuration.

Find the Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) property and add the following:
RANGER_REST_URL=[***RANGER ADMIN HOST***]:[***RANGER ADMIN PORT***]

Replace [***RANGER ADMIN HOST***] with the hostname where a Ranger Admin service role is deployed. You can find the hostname by going to Ranger > Instances. The hostname is displayed in the Hostname column next to Ranger Admin. Choose one of the available Ranger Admin instances.

Replace [***RANGER ADMIN PORT***] with the port used by the Ranger Admin service role. The port is specified in the Admin HTTP Port or Admin HTTPS port Ranger property. Which port is used depends on whether SSL is enabled for Ranger Admin.

Click Save Changes.

Restart Kafka.

Restart SMM and SRM.
CDPD-31745: SRM Control fails to configure internal topic when target is earlier than Kafka 2.3: When the target Kafka cluster of a replication is earlier than version 2.3, the srm-control internal topic is created with an incorrect configuration (cleanup.policy=compact). This causes the srm-control topic to lose the replication filter records, causing issues in the replication.; After a replication is enabled where the target Kafka cluster is earlier than 2.3, manually configure all srm-control.[***SOURCE CLUSTER ALIAS***].internal topics in the target cluster to use cleanup.policy=compact.

CDPD-31235: Negative consumer group lag when replicating groups through SRM

SRM checkpointing reads the offset-syncs topic to create offset mappings for committed consumer group offsets. In some corner cases, it is possible that a mapping is not available in offset-syncs. In a case like this SRM simply copies the source offset, which might not be a valid offset in the replica topic.

One possible situation is if there is an empty topic in the source cluster with a non-zero end offset (for example, retention already removed the records), and a consumer group which has a committed offset set to the end offset. If replication is configured to start replicating this topic, it will not have an offset mapping available in offset-syncs (as the topic is empty), causing SRM to copy the source offset.

This can cause issues when automatic offset synchronization is enabled, as the consumer group offset can be potentially set to a high number. SRM never rewinds these offsets, so even when there is a correct offset mapping available, the offset will not be updated correctly.

After offset mappings are created, stop the consumers of the group and set the committed offsets of the group to the end of the topic on the target cluster with this command:

kafka-consumer-groups --bootstrap-server [***HOST***]:[***PORT***] --group [***GROUP***] --topic [***SOURCE CLUSTER ALIAS***].[***TOPIC***] --reset-offsets --to-latest --execute

Alternatively, set it to the beginning of the topic with this command:

kafka-consumer-groups --bootstrap-server [***HOST***]:[***PORT***] --group <group> --topic [***SOURCE CLUSTER ALIAS***].[***TOPIC***] --reset-offsets --to-earliest --execute

Limitations

SRM cannot replicate Ranger authorization policies to or from Kafka clusters: Due to a limitation in the Kafka-Ranger plugin, SRM cannot replicate Ranger policies to or from clusters that are configured to use Ranger for authorization. If you are using SRM to replicate data to or from a cluster that uses Ranger, disable authorization policy synchronization in SRM. This can be achieved by clearing the Sync Topic Acls Enabled (sync.topic.acls.enabled) checkbox.
SRM cannot ensure the exactly-once semantics of transactional source topics: SRM data replication uses at-least-once guarantees, and as a result cannot ensure the exactly-once semantics (EOS) of transactional topics in the backup/target cluster.
note
Even though EOS is not guaranteed, you can still replicate the data of a transactional source, but you must set isolation.level to read_committed for SRM's internal consumers. This can be done by adding [***SOURCE CLUSTER ALIAS***]->[***TARGET CLUSTER ALIAS***].consumer.isolation.level=read_committed to the Streams Replication Manager's Replication Configs SRM service property in Cloudera Manger.
SRM checkpointing is not supported for transactional source topics: SRM does not correctly translate checkpoints (committed consumer group offsets) for transactional topics. Checkpointing assumes that the offset mapping function is always increasing, but with transactional source topics this is violated. Transactional topics have control messages in them, which take up an offset in the log, but they are never returned on the consumer API. This causes the mappings to decrease, causing issues in the checkpointing feature. As a result of this limitation, consumer failover operations for transactional topics is not possible.