Known Issues in Streams Replication Manager

Known Issues

CDPD-22089: SRM does not sync re-created source topics until the offsets have caught up with target topic

Messages written to topics that were deleted and re-created are not replicated until the source topic reaches the same offset as the target topic. For example, if at the time of deletion and re-creation there are a 100 messages on the source and target clusters, new messages will only get replicated once the re-created source topic has 100 messages. This leads to messages being lost.

None

CDPD-14019: SRM may automatically re-create deleted topics

If auto.create.topics.enable is enabled, deleted topics are automatically recreated on source clusters.

Prior to deletion, remove the topic from the topic allowlist with the srm-control tool. This prevents topics from being re-created.

srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --remove [TOPIC1][TOPIC2]

CDPD-11079: Blacklisted topics appear in the list of replicated topics

If a topic was originally replicated but was later excluded for replication, it will still appear as a replicated topic under the /remote-topics REST API endpoint. As a result, if a call is made to this endpoint, this topic will be included in the response. Additionally, the excluded topic will also be visible in the SMM UI. However, it's Partitions and Consumer Groups will be 0, its Throughput, Replication Latency and Checkpoint Latency will show N/A.

None

CDPD-60823: Configuring the SRM Client's secure storage is mandatory for unsecured environments

In an unsecured environment the srm-control tool should not need any additional configuration to run. However, due to an issue with the automatic generation of the default configuration, configuring the SRM Client's secure storage is mandatory for the srm-control tool. This is true even if none of the clusters that the tool connects to are secured.

If a secure storage is not configured, the tool will fail with the following NullPointerException:

java.lang.NullPointerException
at com.cloudera.dim.mirror.SecureConfigProvider.retrievePassword(SecureConfigProvider.java:99)
at com.cloudera.dim.mirror.SecureConfigProvider.configure(SecureConfigProvider.java:113)
at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:533)
at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:477)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:107)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
at org.apache.kafka.connect.mirror.MirrorMakerConfig.<init>(MirrorMakerConfig.java:88)
at com.cloudera.dim.mirror.MirrorControlCommand$SourceTargetCommand.init(MirrorControlCommand.java:97)
at com.cloudera.dim.mirror.MirrorControlCommand.issueCommand(MirrorControlCommand.java:369)
at com.cloudera.dim.mirror.MirrorControlCommand.main(MirrorControlCommand.java:346)

Configure a secure storage password and set it as an environment variable in your CLI session before running the srm-control tool.

1. In Cloudera Manager, select the Streams Replication Manager service.
2. Go to Configuration.
3. Find and configure the SRM Client's Secure Storage Password property.

   Take note of the password that you configure.

4. Click Save changes.

5. Restart the SRM service
6. SSH into one of the SRM hosts in your cluster.
7. Set the secure storage password as an environment variable.

   export [***SECURE STORAGE ENV VAR***]=”[***SECURE STORAGE PASSWORD***]”
   

   Replace [***SECURE STORAGE ENV VAR***] with the name of the environment variable you specified in Environment Variable Holding SRM Client's Secure Storage Password. Replace [***SRM SECURE STORAGE PASSWORD***] with the password you specified in SRM Client's Secure Storage Password. For example:

   export SECURESTOREPASS=”mypassword”

OPSAPS-61001: Saving configuration changes for SRM is not possible

Cloudera Manager incorrectly labels the SRM Client's Secure Storage Password property as mandatory. Moreover, it does not offer this property for configuration when SRM is installed with the Add Service Wizard.

As a result, it is possible to install and start SRM without configuring this property. However, in a case like this, making changes to SRM's configuration is not possible until the SRM Client's Secure Storage Password property is set.

Configure the SRM Client's Secure Storage Password property.

OPSAPS-61278: The SRM Client’s secure storage fails to generate correctly in FIPS-enabled clusters

In a FIPS enabled cluster, the SRM Client’s secure storage fails to generate correctly. As a result, the automatically generated configuration used by the srm-control tool will contain unresolvable references, making it unusable.

There are two workarounds for this issue. Choose one of the following:

• Configure SRM using the Streams Replication Manager’s Replication Configs Cloudera Manager property.

  Ensure that all cluster connection related properties (cluster aliases, bootstrap servers, security-related properties) are added, with the appropriate prefixes, for all clusters taking part in the replication process.

• Manually create a custom configuration file and use it with the srm-control tool.
  1. Create a copy of the default configuration located at /etc/streams_replication_manager/conf/srm.properties.
  2. Update all security related properties in the copy with the appropriate values.
     The values that you must update are similar to the following example:

     ${secure:/var/lib/streams_replication_manager/client.store:ext_ssl_truststore_password}

     Ensure that you replace all ${...} references with actual values such as the key and truststore locations, passwords, and so on.
  3. Run the srm-control tool using the --config option.
     For example:

     srm-control topics --config [***PATH TO CUSTOM CONFIGURATION FILE***] --source [***SOURCE_CLUSTER***] --target [***TARGET_CLUSTER***] --add [***TOPIC1***],[***TOPIC2***]

Using the service dependency method to configure Kerberos enabled co-located clusters is not supported

Using the Streams Replication Manager Co-located Kafka Cluster Alias property to auto-configure the connection to a Kerberos enabled co-located Kafka cluster is not supported. In a case like this, the generated JAAS configuration contains host-specific configuration. This causes SRM to fail to connect to the co-located Kafka cluster on other hosts.

Define your co-located Kafka clusters using Kafka credentials. For more information, see Defining co-located Kafka clusters using Kafka credentials. Alternatively, use the Streams Replication Manager's Replication Configs property to configure the connection to the co-located Kafka clusters.

OPSAPS-63992: Rolling restart unavailable for SRM

Initiating a rolling restart for the SRM service is not possible. Consequently, performing a rolling upgrade of the SRM service is also not possible.

Upgrade to Cloudera Manager 7.7.1 or later. Upgrading Cloudera Runtime is not required to fix this issue.

CDPD-31745: SRM Control fails to configure internal topic when target is earlier than Kafka 2.3

When the target Kafka cluster of a replication is earlier than version 2.3, the srm-control internal topic is created with an incorrect configuration (cleanup.policy=compact). This causes the srm-control topic to lose the replication filter records, causing issues in the replication.

After a replication is enabled where the target Kafka cluster is earlier than 2.3, manually configure all srm-control.[***SOURCE CLUSTER ALIAS***].internal topics in the target cluster to use cleanup.policy=compact.

OPSAPS-62546: Kafka External Account SSL keypassword configuration is used incorrectly by SRM

When a Kafka External Account specifies a keystore that uses an SSL key password, SRM uses it as the ssl.keystore.key configuration. Due to using the incorrect ssl.keystore.key configuration, SRM will fail to load the keystore in certain cases.

Workaround: For the keystores used by the Kafka External Accounts, the SSL key password should match the SSL keystore password, and the SSL keystore key password should not be provided. Alternatively, you can use the legacy connection configurations based on the streams.replication.manager.configs to specify the SSL key password.

Limitations