Known Issues in Streams Replication Manager

Learn about the known issues in Streams Replication Manager, the impact or changes to the functionality, and the workaround.

CDPD-22089: SRM does not sync re-created source topics until the offsets have caught up with target topic

Messages written to topics that were deleted and re-created are not replicated until the source topic reaches the same offset as the target topic. For example, if at the time of deletion and re-creation there are a 100 messages on the source and target clusters, new messages will only get replicated once the re-created source topic has 100 messages. This leads to messages being lost.

None

CDPD-14019: SRM may automatically re-create deleted topics

If auto.create.topics.enable is enabled, deleted topics are automatically recreated on source clusters.

Prior to deletion, remove the topic from the topic allowlist with the srm-control tool. This prevents topics from being re-created.

srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --remove [TOPIC1][TOPIC2]

CDPD-13864 and CDPD-15327: Replication stops after the network configuration of a source or target cluster is changed

If the network configuration of a cluster which is taking part in a replication flow is changed, for example, port numbers are changed as a result of enabling or disabling TLS, SRM will not update its internal configuration even if SRM is reconfigured and restarted. From SRM’s perspective, it is the cluster identity that has changed. SRM cannot determine whether the new identity corresponds to the same cluster or not, only the owner or administrator of that cluster can know. In this case, SRM tries to use the last known configuration of that cluster which might not be valid, resulting in the halt of replication.

There are three workarounds for this issue. Choose one of the following:

Increase the driver rebalance timeout

Increasing the rebalance timeout to 5 minutes (300000 ms) or longer can resolve the issue. In general a 5 minute timeout should be sufficient for most deployments. However, depending on your scenario, an even longer period might be required. Increasing the rebalance timeout might lead to increased latency when the SRM drivers stop. The cluster will be slower when it rebalances the load of the removed driver.

The rebalance timeout can be configured on a per cluster (alias) basis by adding the following to the Streams Replication Manager’s Replication Configs Cloudera Manager property:

[***ALIAS***].rebalance.timeout.ms = [***VALUE***]

Replace [***ALIAS***] with a cluster alias specified in Streams Replication Manager Cluster alias. Do this for all clusters that are taking part in the replication process. When correctly configured, your configuration will have a rebalance.timeout.ms entry corresponding to each cluster (alias). For example:

primary.rebalance.timeout.ms = 30000
secondary.rebalance.timeout.ms = 30000
tertiary.rebalance.timeout.ms = 30000

After the new broker configuration is applied by SRM, the rebalance timeout can be reverted back to its original value, or removed from the configuration altogether.

Decrease replication admin timeout

Decreasing the replication admin timeout to 15 seconds (15000 ms) can resolve the issue. With higher loads, this might cause WARN messages to appear in the SRM driver log.

The admin timeout can be configured on a per replication basis by adding the following to the Streams Replication Manager’s Replication Configs Cloudera Manager property:

[***REPLICATION***].admin.timeout.ms = [***VALUE***]

Replace [***REPLICATION***] with a replication specified in Streams Replication Manager’s Replication Configs. Do this for all affected replications. When correctly configured, your configuration will have an admin.timeout.ms entry corresponding to each affected replication. For example:

primary->secondary.admin.timeout.ms = 15000
secondary->primary.admin.timeout.ms = 15000

After the new broker configuration is applied by SRM, the admin timeout can be reverted back to its original value, or removed from the configuration altogether.

Upgrade the brokers incrementally

Instead of switching over to the new configuration, open two separate listeners on the broker. One for the old configuration, and one for the new configuration. After updating SRM's configuration and restarting SRM, the old listener can be turned off. Non–inter-broker listeners can be configured with the dynamic configuration API of Kafka, this way not every listener change has to be followed by a restart.

CDPD-11079: Blacklisted topics appear in the list of replicated topics

If a topic was originally replicated but was later disallowed (blacklisted), it will still appear as a replicated topic under the /remote-topics REST API endpoint. As a result, if a call is made to this endpoint, the disallowed topic will be included in the response. Additionally, the disallowed topic will also be visible in the SMM UI. However, it's Partitions and Consumer Groups will be 0, its Throughput, Replication Latency and Checkpoint Latency will show N/A.

None

CDPD-60823: Configuring the SRM Client's secure storage is mandatory for unsecured environments

In an unsecured environment the srm-control tool should not need any additional configuration to run. However, due to an issue with the automatic generation of the default configuration, configuring the SRM Client's secure storage is mandatory for the srm-control tool. This is true even if none of the clusters that the tool connects to are secured.

If a secure storage is not configured, the tool will fail with the following NullPointerException:

java.lang.NullPointerException
at com.cloudera.dim.mirror.SecureConfigProvider.retrievePassword(SecureConfigProvider.java:99)
at com.cloudera.dim.mirror.SecureConfigProvider.configure(SecureConfigProvider.java:113)
at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:533)
at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:477)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:107)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
at org.apache.kafka.connect.mirror.MirrorMakerConfig.<init>(MirrorMakerConfig.java:88)
at com.cloudera.dim.mirror.MirrorControlCommand$SourceTargetCommand.init(MirrorControlCommand.java:97)
at com.cloudera.dim.mirror.MirrorControlCommand.issueCommand(MirrorControlCommand.java:369)
at com.cloudera.dim.mirror.MirrorControlCommand.main(MirrorControlCommand.java:346)

Configure a secure storage password and set it as an environment variable in your CLI session before running the srm-control tool.

1. In Cloudera Manager, select the Streams Replication Manager service.
2. Go to Configuration.
3. Find and configure the SRM Client's Secure Storage Password property.

   Take note of the password that you configure.

4. Click Save changes.

5. Restart the SRM service
6. SSH into one of the SRM hosts in your cluster.
7. Set the secure storage password as an environment variable.

   export [***SECURE STORAGE ENV VAR***]=”[***SECURE STORAGE PASSWORD***]”
   

   Replace [***SECURE STORAGE ENV VAR***] with the name of the environment variable you specified in Environment Variable Holding SRM Client's Secure Storage Password. Replace [***SRM SECURE STORAGE PASSWORD***] with the password you specified in SRM Client's Secure Storage Password. For example:

   export SECURESTOREPASS=”mypassword”

OPSAPS-61001: Saving configuration changes for SRM is not possible

Cloudera Manager incorrectly labels the SRM Client's Secure Storage Password property as mandatory. Moreover, it does not offer this property for configuration when SRM is installed with the Add Service Wizard.

As a result, it is possible to install and start SRM without configuring this property. However, in a case like this, making changes to SRM's configuration is not possible until the SRM Client's Secure Storage Password property is set.

Configure the SRM Client's Secure Storage Password property.

OPSAPS-60601: The SRM client's secure storage might become corrupted if the JAAS Secret properties are used

Cloudera Manager generates a secure storage for SRM clients that stores the sensitive data (security related properties) needed to access the clusters that SRM replicates. The sensitive data that the secure storage contains is sourced from the Kafka credentials created by the user in the Administration > External Accounts > Kafka Credentials > Add Kafka credentials modal window in Cloudera Manager. If the JAAS Secret properties available in this modal window are used, the generated secure storage can become corrupted. In a case like this, the JAAS configuration is only partially saved to the configuration.

Specify the JAAS configuration using the Streams Replication Manager’s Replication Configs Cloudera Manager property.

This is done by adding the sasl.jaas.config property to Streams Replication Manager’s Replication Configs with an appropriate prefix. For example:

[***ALIAS***].sasl.jaas.config=[***JAAS CONFIG***]

Replace [***ALIAS***] with a cluster alias specified in Streams Replication Manager Cluster alias. Replace [***JAAS CONFIG***] with a valid JAAS configuration. Repeat this process for all clusters that require a JAAS configuration.

OPSAPS-60601: Replication does not start when the target cluster of the replication is unsecured

When replicating data into an unsecured cluster, the configuration generated for SRM will contain references to defined, but otherwise empty environment variables related to TLS/SSL properties (keystore or truststore locations). The values of these variables cannot be processed by SRM. As a result, replication does not start.

Create and use a placeholder truststore file for the unsecured cluster:

1. Create a placeholder truststore with the keytool utility. For example:

   keytool -genkeypair -alias placeholder -storepass secret -keypass secret -keystore placeholder.jks -dname "CN=Placeholder, OU=Department, O=Company, L=City, ST=State, C=CA"

2. Copy the resulting placeholder.jks file to the same location on all SRM driver hosts.
3. Configure SRM to use the keystore for the unsecured cluster.
   This can be done by adding the ssl.truststore.location and ssl.truststore.password properties to the Streams Replication Manager’s Replication Configs Cloudera Manager property with an appropriate prefix. For example:

   [***ALIAS***].ssl.truststore.location=[***TRUSTSTORE LOCATION***]
   [***ALIAS***].ssl.truststore.password=[***TRUSTSTORE PASSWORD***]

   Replace [***ALIAS***] with the unsecured cluster's alias. You can find the alias in Streams Replication Manager Cluster alias. Replace [***TRUSTSTORE LOCATION**] with the location you copied the placeholder.jks file to in Step 2. Replace [***TRUSTSTORE PASSWORD***] with the password you specified when creating the keystore. Repeat this step for all unsecured clusters.

OPSAPS-60775: Kafka External Accounts configurations are not generated for the SRM Service

Kafka External Account configurations are not generated for SRM Service, making it unable to target clusters defined through External Accounts.

Use the co-located cluster auto-configuration, or the legacy array configuration (Streams Replication Manager’s Replication Configs) to configure the target cluster of SRM Service.

OPSAPS-61814: Using the service dependency method to configure Kerberos enabled co-located clusters is not supported

Using the Streams Replication Manager Co-located Kafka Cluster Alias property to auto-configure the connection to a Kerberos enabled co-located Kafka cluster is not supported. In a case like this, the generated JAAS configuration contains host-specific configuration. This causes SRM to fail to connect to the co-located Kafka cluster on other hosts.

Define your co-located Kafka clusters using Kafka credentials. For more information, see Defining co-located Kafka clusters using Kafka credentials. Alternatively, use the Streams Replication Manager's Replication Configs property to configure the connection to the co-located Kafka clusters.

OPSAPS-63992: Rolling restart unavailable for SRM

Initiating a rolling restart for the SRM service is not possible. Consequently, performing a rolling upgrade of the SRM service is also not possible.

None

CDPD-31745: SRM Control fails to configure internal topic when target is earlier than Kafka 2.3

When the target Kafka cluster of a replication is earlier than version 2.3, the srm-control internal topic is created with an incorrect configuration (cleanup.policy=compact). This causes the srm-control topic to lose the replication filter records, causing issues in the replication.

After a replication is enabled where the target Kafka cluster is earlier than 2.3, manually configure all srm-control.[***SOURCE CLUSTER ALIAS***].internal topics in the target cluster to use cleanup.policy=compact.

CDPD-31235: Negative consumer group lag when replicating groups through SRM

SRM checkpointing reads the offset-syncs topic to create offset mappings for committed consumer group offsets. In some corner cases, it is possible that a mapping is not available in offset-syncs. In a case like this SRM simply copies the source offset, which might not be a valid offset in the replica topic.

One possible situation is if there is an empty topic in the source cluster with a non-zero end offset (for example, retention already removed the records), and a consumer group which has a committed offset set to the end offset. If replication is configured to start replicating this topic, it will not have an offset mapping available in offset-syncs (as the topic is empty), causing SRM to copy the source offset.

This can cause issues when automatic offset synchronization is enabled, as the consumer group offset can be potentially set to a high number. SRM never rewinds these offsets, so even when there is a correct offset mapping available, the offset will not be updated correctly.

After offset mappings are created, stop the consumers of the group and set the committed offsets of the group to the end of the topic on the target cluster with this command:

kafka-consumer-groups --bootstrap-server [***HOST***]:[***PORT***] --group [***GROUP***] --topic [***SOURCE CLUSTER ALIAS***].[***TOPIC***] --reset-offsets --to-latest --execute

Alternatively, set it to the beginning of the topic with this command:

kafka-consumer-groups --bootstrap-server [***HOST***]:[***PORT***] --group <group> --topic [***SOURCE CLUSTER ALIAS***].[***TOPIC***] --reset-offsets --to-earliest --execute

OPSAPS-62546: Kafka External Account SSL keypassword configuration is used incorrectly by SRM

When a Kafka External Account specifies a keystore that uses an SSL key password, SRM uses it as the ssl.keystore.key configuration. Due to using the incorrect ssl.keystore.key configuration, SRM will fail to load the keystore in certain cases.

Workaround: For the keystores used by the Kafka External Accounts, the SSL key password should match the SSL keystore password, and the SSL keystore key password should not be provided. Alternatively, you can use the legacy connection configurations based on the streams.replication.manager.configs to specify the SSL key password.