Registering Kafka clusters in Cloudera Surveyor for Apache Kafka

Bootstrap servers for Kafka deployed in Kubernetes

If Cloudera Surveyor for Apache Kafka and the Kafka cluster you want to register are deployed in the same Kubernetes cluster, you can use the DNS name of the Kubernetes Service that provides access to the Kafka cluster as the bootstrap server.

For example, the DNS name of the default ClusterIP Service for a Kafka cluster that was deployed with the Strimzi Cluster Operator is similar to the following example.

my-cluster-kafka-bootstrap.my-namespace.svc.cluster.local

Where my-cluster is the name of the Kafka cluster, kafka-bootstrap is a fixed affix, my-namespace is the namespace of the cluster, and svc.cluster.local is the domain name used internally by the Kubernetes cluster.

Mounting Secrets

Use the secretsToMount property to specify which Secrets and keys from a Secret you want to mount. The Secret must be in the same namespace where Cloudera Surveyor is installed. The following example mounts a single key from a single Secret.

#...
secretsToMount:
  - create: false
    secretRef: "[***SECRET NAME***]"
    items:
      - key: "[***SECRET KEY***]"
        path: "[***PATH TO MOUNT DATA***]"

Each item inside secretsToMount must have a secretRef property, which specifies the name of the Secret to mount. Optionally, you can include an items array, which maps Secret keys to paths. If not present, all keys from the Secret are mounted as is.

Data is mounted to /opt/secrets/[***SECRET NAME***]/ in the Cloudera Surveyor Container. This means that the value you specify in secretsToMount[*].items[*].path is relative to /opt/secrets/[***SECRET NAME***]/.

Referencing mounted data

All Kafka clients used by Cloudera Surveyor use the Kafka DirectoryConfigProvider. Use the following syntax to reference mounted data as the value for a client property.

${dir:[***FULL DIR PATH***]:[***FILENAME***]}

For example, assume you have a Secret named my-kafka-jaas-secret. This Secret contains a single key named jaas-key. The key contains a JAAS configuration. To mount the Secret and reference its data, you pass the following configuration.

#...
clusterConfigs:
  clusters:
    - clusterName: "my-kafka"
      commonClientConfig:
        sasl.jaas.config: "\\${dir:/opt/secrets/my-kafka-jaas-secret/jaas.conf}"
secretsToMount:
  - create: false
    secretRef: "my-kafka-jaas-secret"
    items:
      - key: "jaas-key"
        path: "jaas.conf"

In this example, the value of the jaas-key key in the Secret is mounted to /opt/secrets/my-kafka-jaas-secret/jaas.conf. This file is referenced as the value for the sasl.jaas.config Kafka client property using DirectoryConfigProvider syntax. As a result, the Kafka client in Cloudera Surveyor that connects to the my-kafka cluster uses the specified JAAS configuration for authentication.

Creating Secrets

If a Secret that you want to mount does not exist, you can create it by setting the secretsToMount[*].create property to true. In this case, the specified Secret is created and managed by Helm. The content for each item in the items array is set through the secretsToMount[*].items[*].content property.

Because the content property must include your actual data in plaintext, Cloudera recommends that you do not set this property directly in your custom values file (values.yaml). Instead, pass the contents of your Secrets with the --set-file option when you run a helm install or helm upgrade command. This allows you to reference the contents of the Secrets from a file. This way, sensitive data is not made available in shell history or stored directly in your values file.

For example, assume you specify the following in your values file.

#...
secretsToMount:
  - create: true
    secretRef: "my-kafka-jaas-secret"
    items:
      - key: "jaas-key"
        path: "jaas.conf"

Notice that content is not specified. When applying this configuration with helm, you pass the contents of jaas-key in my-kafka-jaas-secret using --set-file.

helm upgrade cloudera-surveyor [***CHART***] \
  --namespace [***NAMESPACE***] \
  --values [***MY-VALUES.YAML***] \
  --set-file secretsToMount[0].items[0].content=[***FILEPATH***] \
  --reuse-values

• Ensure that you use correct numeric indices ([0]) to target specific list items. secretsToMount can have multiple Secrets, and each Secret can have multiple keys.

• [***FILEPATH***] in this case points to a file containing the JAAS configuration.

Configuration levels and hierarchy

Kafka client configuration is applied in a hierarchical order, from the least specific to most specific. Configurations are also merged. If duplicate keys exist, the value from the more specific configuration overrides the value from the less specific one.

This configuration hierarchy provides granular control over each client that Cloudera Surveyor uses. In addition, it enables you to specify common properties at higher levels, reducing redundancy and simplifying management in configuration.

The order from least to most specific levels is as follows.

• Global common – Client configuration applied to all Kafka clients that Cloudera Surveyor uses. Configured with surveyorConfig.surveyor.commonClientConfig.

• Global pool – Client configuration applied to all Kafka clients belonging to the specified pool. Configured with surveyorConfig.surveyor.snapshotClientPool.clientConfig and surveyorConfig.surveyor.adminClientPool.clientConfig.

• Cluster common – Client configuration applied to Kafka clients used for the specified cluster. Configured with clusterConfigs.clusters[*].commonClientConfig.

• Cluster pool – Client configuration applied to all Kafka clients belonging to the specified pool that connect to the specified cluster. Configured with clusterConfigs.clusters[*].snapshotClientPool.clientConfig and clusterConfigs.clusters[*].adminClientPool.clientConfig

For example, assume you pass the following configuration to Cloudera Surveyor.

#...
surveyorConfig:
  surveyor:
    commonClientConfig:
      security.protocol: SASL_SSL
      sasl.mechanism: PLAIN
  snapshotClientPool:
    clientConfig:
      request.timeout.ms: 30000
  adminClientPool:
    clientConfig:
      retries: 5

clusterConfigs:
  clusters:
    - clusterName: "[***CLUSTER NAME***]"
      tags: 
        - "[***TAG1***]"
        - "[***TAG2***]"
      bootstrapServers: "[***BOOTSTRAP SERVERS***]"
      commonClientConfig:
        security.protocol: "PLAINTEXT"
      snapshotClientPool:
        clientConfig:
          request.timeout.ms: 10000

The resulting client configuration used by the clients is as follows.

• For the snapshot clients connecting to the [***CLUSTER NAME***] cluster:

  • security.protocol is PLAINTEXT (cluster common override).

  • request.timeout.ms is 10000 (cluster pool override).

• For the admin clients connecting to the [***CLUSTER NAME***] cluster:

  • security.protocol is PLAINTEXT (cluster common override).

  • retries is 5 (no override).