Cloudera Docs

Topics and Groups Subcommand

Learn how to use the topics and groups subcommand of the srm-control command line tool.

The topics and groups subcommands are used to manipulate the topic or group allowlist (whitelist) and denylist (blacklist). Both subcommands support the same set of command options.

If you are modifying allow and denylists which target newly created topics or consumer groups, changes made with the srm-control tool may not be instantaneous A topic or group needs to be discovered by Streams Replication Manager before it can be added to or removed from allow or denylists. New topic and group discovery happens every 10 minutes by default. As a result, you may need to wait up to 10 minutes until you can see the changes made.

Add topics or groups to an allowlist:
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add [TOPIC1],[TOPIC2]
srm-control groups --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add [GROUP1],[GROUP2]
Remove topics or groups from an allowlist:
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --remove [TOPIC1],[TOPIC2]
srm-control groups --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --remove [GROUP1],[GROUP2]
Add topics or groups to a denylist (blacklist):
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add-blacklist [TOPIC1],[TOPIC2]
srm-control groups --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add-blacklist [GROUP1],[GROUP2]
Remove topics or groups from a denylist:
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --remove-blacklist [TOPIC1],[TOPIC2]
srm-control groups --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --remove-blacklist [GROUP1],[GROUP2]

In addition to adding or removing items, you can also use the tool to look at the contents of a deny or allowlist.

srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --list

Using Regular Expressions

Specifying topics or groups is also possible with regular expressions. This is done by specifying a regular expression instead of topic names in the --add, add-blackilist, or remove-blacklist options.

For example, assume that you want to replicate all topics from the source cluster to the target cluster. In a case like this, you must add all topics to the allowlist. This can be done with a single srm-control command using the .* regular expression. 
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"
You can also use regular expressions to specify a set of topics. For example, assume that you have many topics in your Kafka cluster, but only want to add the following topics to the allowlist.
testTopic1
testTopic2
testTopic3
In a case like this, you can use the testTopic.* or ^test.* regular expressions to add the topics.
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add "testTopic.*"
srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add "^test.*"

Client Override Options

The topics and groups subcommands support a number of client override options. Client override options allow users to temporarily specify or override configuration properties used for replication. These options also enable users to issue srm-control commands even if the configuration of Streams Replication Manager is not available on the host where the command is being issued. While it is possible to specify a range of properties with the client override options, and they can prove to be a powerful tool in certain scenarios, Cloudera recommends that you use Cloudera Manager to manage client configuration options.

The following client override options are available:
  • --bootstrap-servers: Specifies the bootstraps servers.
  • --producer-props: Specifies producer configuration properties.
  • --consumer-props: Specifies consumer configuration properties.
  • --props: Specifies client configuration properties.

A simple example of using client override options is when you want to change the bootstrap server. This can be done in two ways.

You can specify the bootstrap server with the --bootstrap-servers option. 
srm-control --bootstrap-servers localhost:9092 topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --list
Alternatively, you can also use the --props option together with the bootstrap.servers Kafka property to define the bootstrap server.
srm-control --props bootstrap.servers=localhost:9092 topics --source [SOURCE_CLUSTER] --list