Known Issues in Streams Replication Manager
Learn about the known issues in Streams Replication Manager, the impact or changes to the functionality, and the workaround.
Known Issues
- CDPD-22089: SRM does not sync re-created source topics until the offsets have caught up with target topic
- Messages written to topics that were deleted and re-created are not replicated until the source topic reaches the same offset as the target topic. For example, if at the time of deletion and re-creation there are a 100 messages on the source and target clusters, new messages will only get replicated once the re-created source topic has 100 messages. This leads to messages being lost.
- CDPD-14019: SRM may automatically re-create deleted topics
- If
auto.create.topics.enable
is enabled, deleted topics are automatically recreated on source clusters.
- CDPD-11079: Blacklisted topics appear in the list of replicated topics
- If a topic was originally replicated but was later excluded for
replication, it will still appear as a replicated topic under the
/remote-topics
REST API endpoint. As a result, if a call is made to this endpoint, this topic will be included in the response. Additionally, the excluded topic will also be visible in the SMM UI. However, it's Partitions and Consumer Groups will be 0, its Throughput, Replication Latency and Checkpoint Latency will show N/A.
- CDPD-60823: Configuring the SRM Client's secure storage is mandatory for unsecured environments
- In an unsecured environment the
srm-control
tool should not need any additional configuration to run. However, due to an issue with the automatic generation of the default configuration, configuring the SRM Client's secure storage is mandatory for thesrm-control
tool. This is true even if none of the clusters that the tool connects to are secured.If a secure storage is not configured, the tool will fail with the following NullPointerException:java.lang.NullPointerException at com.cloudera.dim.mirror.SecureConfigProvider.retrievePassword(SecureConfigProvider.java:99) at com.cloudera.dim.mirror.SecureConfigProvider.configure(SecureConfigProvider.java:113) at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:533) at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:477) at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:107) at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142) at org.apache.kafka.connect.mirror.MirrorMakerConfig.<init>(MirrorMakerConfig.java:88) at com.cloudera.dim.mirror.MirrorControlCommand$SourceTargetCommand.init(MirrorControlCommand.java:97) at com.cloudera.dim.mirror.MirrorControlCommand.issueCommand(MirrorControlCommand.java:369) at com.cloudera.dim.mirror.MirrorControlCommand.main(MirrorControlCommand.java:346)
- OPSAPS-61001: Saving configuration changes for SRM is not possible
- Cloudera Manager incorrectly labels the SRM Client's
Secure Storage Password property as mandatory. Moreover, it does not offer
this property for configuration when SRM is installed with the Add Service
Wizard.
As a result, it is possible to install and start SRM without configuring this property. However, in a case like this, making changes to SRM's configuration is not possible until the SRM Client's Secure Storage Password property is set.
- OPSAPS-61278: The SRM Client’s secure storage fails to generate correctly in FIPS-enabled clusters
- In a FIPS enabled cluster, the SRM Client’s secure storage fails
to generate correctly. As a result, the automatically generated configuration used by
the
srm-control
tool will contain unresolvable references, making it unusable. - OPSAPS-61814: Using the service dependency method to configure Kerberos enabled co-located clusters is not supported
- Using the Streams Replication Manager Co-located Kafka Cluster Alias property to auto-configure the connection to a Kerberos enabled co-located Kafka cluster is not supported. In a case like this, the generated JAAS configuration contains host-specific configuration. This causes SRM to fail to connect to the co-located Kafka cluster on other hosts.
- OPSAPS-63992: Rolling restart unavailable for SRM
- Initiating a rolling restart for the SRM service is not possible. Consequently, performing a rolling upgrade of the SRM service is also not possible.
- CDPD-31745: SRM Control fails to configure internal topic when target is earlier than Kafka 2.3
- When the target Kafka cluster of a replication is earlier than
version 2.3, the
srm-control
internal topic is created with an incorrect configuration (cleanup.policy=compact
). This causes thesrm-control
topic to lose the replication filter records, causing issues in the replication. - CDPD-31235: Negative consumer group lag when replicating groups through SRM
-
SRM checkpointing reads the offset-syncs topic to create offset mappings for committed consumer group offsets. In some corner cases, it is possible that a mapping is not available in offset-syncs. In a case like this SRM simply copies the source offset, which might not be a valid offset in the replica topic.
One possible situation is if there is an empty topic in the source cluster with a non-zero end offset (for example, retention already removed the records), and a consumer group which has a committed offset set to the end offset. If replication is configured to start replicating this topic, it will not have an offset mapping available in offset-syncs (as the topic is empty), causing SRM to copy the source offset.
This can cause issues when automatic offset synchronization is enabled, as the consumer group offset can be potentially set to a high number. SRM never rewinds these offsets, so even when there is a correct offset mapping available, the offset will not be updated correctly.
- OPSAPS-67772: SRM Service metrics processing fails when the noexec option is enabled for /tmp
- The SRM Service role uses /tmp to extract
RocksDB .so files, which are required for metrics processing to
function. If the
noexec
option is enabled for the /tmp directory, the SRM Service role is not able load the required RocksDB files. This results in metrics processing failing. - OPSAPS-67738: SRM Service role's Remote Querying feature does not work when the noexec option is enabled for /tmp
-
The SRM Service role puts the Netty native libraries into the /tmp directory. As a result, If the
noexec
option is enabled for the /tmp directory, the Remote Querying feature will fail to function. - OPSAPS-62546: Kafka External Account SSL keypassword configuration is used incorrectly by SRM
- When a Kafka External Account specifies a keystore that uses an
SSL key password, SRM uses it as the
ssl.keystore.key
configuration. Due to using the incorrectssl.keystore.key
configuration, SRM will fail to load the keystore in certain cases. - CDPD-60426: Configuration changes are lost following a rolling restart of the service
- In certain cases, SRM might fail to apply configuration updates if the service is restarted with a rolling restart. In a case like this, configuration changes are ignored without any warning or indication. This issue also affects rolling upgrades.
Limitations
- SRM cannot replicate Ranger authorization policies to or from Kafka clusters
- Due to a limitation in the Kafka-Ranger plugin, SRM cannot
replicate Ranger policies to or from clusters that are configured to use Ranger for
authorization. If you are using SRM to replicate data to or from a cluster that uses
Ranger, disable authorization policy synchronization in SRM. This can be achieved by
clearing the Sync Topic Acls Enabled
(
sync.topic.acls.enabled
) checkbox. - SRM cannot ensure the exactly-once semantics of transactional source topics
- SRM data replication uses at-least-once guarantees, and as a result cannot ensure the exactly-once semantics (EOS) of transactional topics in the backup/target cluster.
- SRM checkpointing is not supported for transactional source topics
- SRM does not correctly translate checkpoints (committed consumer group offsets) for transactional topics. Checkpointing assumes that the offset mapping function is always increasing, but with transactional source topics this is violated. Transactional topics have control messages in them, which take up an offset in the log, but they are never returned on the consumer API. This causes the mappings to decrease, causing issues in the checkpointing feature. As a result of this limitation, consumer failover operations for transactional topics is not possible.