Cloudera Docs

Defining external Kafka clusters

Before you can start replicating data with Streams Replication Manager (SRM), you must define the external Kafka clusters that take part in the replication process. This is done using Cloudera Manager by creating Kafka credentials.

The following list of steps walk you through how you can define the external Kafka clusters that SRM connects to.

  • Ensure that you have reviewed Defining and adding clusters for replication and understand that the following list of steps are only one part of the full configuration workflow.

  • When creating and configuring Kafka Credentials you will be presented with various configuration properties. A comprehensive list of these properties is provided in Kafka credentials property reference. Cloudera recommends having this resource open during configuration.

  1. In Cloudera Manager, go to Administration > External Accounts.
  2. Go to the Kafka Credentials tab.

    The Kafka Credentials tab enables you to create, configure, and manage Kafka credentials. On this tab you will create a credential for each external cluster taking part in the replication process.

  3. Create and configure Kafka credentials for the external clusters:

    Repeat this step for all external clusters.

    1. Click Add Kafka credentials.
    2. Configure the Kafka credentials:
      The security configuration of the external cluster determines which of the available properties you must set. To see which exact properties are required for your scenario, click one of the following tabs matching the security protocol of the cluster you are defining:
      • Name
      • Bootstrap servers
      • Security protocol
      Encryption only
      • Name
      • Bootstrap servers
      • Security protocol
      • Truststore Password
      • Truststore Path
      • Truststore Type
      Encryption and authentication
      • Name
      • Bootstrap servers
      • Security protocol
      • Truststore Password
      • Truststore Path
      • Truststore Type
      • Key Password
      • Keystore Password
      • Keystore Path
      • Keystore Type
      • Name
      • Bootstrap servers
      • Security protocol
      • JAAS Secret [1-3]
      • JAAS Template
      • Kerberos Service Name
      • SASL Mechanism
      • Name
      • Bootstrap servers
      • Security protocol
      • Truststore Password
      • Truststore Path
      • Truststore Type
      • JAAS Secret [1-3]
      • JAAS Template
      • Kerberos Service Name
      • SASL Mechanism
    3. Click Add.
      If credential creation is successful, a new entry corresponding to the Kafka credential you specified appears on the page.
    The following tabs collect examples of different Kafka credentials as well as some notes regarding configuration. Review these examples to better understand how to correctly configure a Kafka credential.
    name=uswest
Bootstrap servers=uswest-node1.cluster.com:9092,uswest-node2.cluster.com:9092,uswest-node3.cluster.com:9092
Security protocol=PLAINTEXT
    Encryption only
    Name=uswest
Bootstrap servers=uswest-node1.cluster.com:9093,uswest-node2.cluster.com:9093,uswest-node3.cluster.com:9093
Security protocol=SSL
Truststore Password=password
Truststore Path=/opt/srm/us-west-truststore.jks
Truststore Type=JKS
    Encryption and authentication
    Name=uswest
Bootstrap servers=uswest-node1.cluster.com:9093,uswest-node2.cluster.com:9093,uswest-node3.cluster.com:9093
Security protocol=SSL
Truststore Password=password
Truststore Path=/opt/srm/us-west-truststore.jks
Truststore Type=JKS
Keystore Password=password
Keystore Path=/opt/srm/us-west-keystore.jks
Keystore Type=JKS
Key Password=password
    Kerberos
    Name=uswest
Bootstrap servers=uswest-node1.cluster.com:9092,uswest-node2.cluster.com:9092,uswest-node3.cluster.com:9092
Security protocol=SASL_PLAINTEXT
JAAS Template=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/opt/srm/streamsrepmgr.keytab" principal="streamsrepmgr@REALM.COM";
Kerberos Service Name=kafka
SASL Mechanism=GSSAPI
    PLAIN
    Name=uswest
Bootstrap servers=uswest-node1.cluster.com:9092,uswest-node2.cluster.com:9092,uswest-node3.cluster.com:9092
Security protocol=SASL_PLAINTEXT
JAAS Secret 1=password
JAAS Template=org.apache.kafka.common.security.plain.PlainLoginModule required username="username" password="##JAAS_SECRET_1##";
SASL Mechanism=PLAIN
    Kerberos
    Name=uswest
Bootstrap servers=uswest-node1.cluster.com:9093,uswest-node2.cluster.com:9093,uswest-node3.cluster.com:9093
Security protocol=SASL_SSL
Truststore Password=password
Truststore Path=/opt/srm/us-west-truststore.jks
Truststore Type=JKS
JAAS Template=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/opt/srm/streamsrepmgr.keytab" principal="streamsrepmgr@REALM.COM";
Kerberos Service Name=kafka
SASL Mechanism=GSSAPI
    PLAIN
    Name=uswest
Bootstrap servers=uswest-node1.cluster.com:9093,uswest-node2.cluster.com:9093,uswest-node3.cluster.com:9093
Security protocol=SASL_SSL
Truststore Password=password
Truststore Path=/opt/srm/us-west-truststore.jks
Truststore Type=JKS
JAAS Secret 1=password
JAAS Template=org.apache.kafka.common.security.plain.PlainLoginModule required username="username" password="##JAAS_SECRET_1##";
SASL Mechanism=PLAIN
The external Kafka clusters are defined using Kafka credentials.
Define the co-located Kafka cluster either with a Kafka credential or through a service dependency.