Cloudera Docs

Installing Cloudera Surveyor for Apache Kafka with Helm

Learn how to install Cloudera Surveyor in Cloudera Streams Messaging - Kubernetes Operator with Helm. Cloudera Surveyor is a UI application that enables you to manage and monitor Kafka clusters.

Cloudera Surveyor Is installed in your Kubernetes cluster with the Cloudera Surveyor Helm chart using the helm install command. When you install the chart, Helm deploys an instance of Cloudera Surveyor, which enables you to manage and monitor your Kafka clusters through a UI interface.

During installation, you configure Cloudera Surveyor using a custom values file (values.yaml) passed to the Helm chart with the --values (-f) option. This file contains properties for configuring Cloudera Surveyor itself as well as Kafka cluster-specific settings that define which Kafka clusters Cloudera Surveyor connects to. Additionally, some properties are configured with --set options.

Installation instructions are provided for the following scenarios.

  • Installing in an internet environment – Follow these steps to install a fully secure instance of Cloudera Surveyor in a Kubernetes cluster with internet access.

  • Installing in an air-gapped environment –Follow these steps to install a fully secure instance of Cloudera Surveyor in a Kubernetes cluster without internet access or if you want to install from a self-hosted registry.

  • Installing for evaluation – Follow these steps to install an unsecure instance of Cloudera Surveyor for development or proof of concept purposes.

Installing Cloudera Surveyor in an internet environment

Complete these steps to install Cloudera Surveyor if your Kubernetes cluster has internet access. These steps install a fully secure instance of Cloudera Surveyor that has authentication, authorization, and channel encryption configured. The example configuration also demonstrates how you can connect a secure Kafka cluster to Cloudera Surveyor.

  • General prerequisites:

    • Ensure that your Kubernetes environment meets requirements listed in System requirements.

    • Your Kubernetes cluster requires internet connectivity to complete these steps. It must be able to reach the Cloudera Docker registry.

    • Ensure that you have access to your Cloudera credentials (username and password). Credentials are required to access the Cloudera Archive and Cloudera Docker registry where installation artifacts are hosted.

    • Ensure that you have access to a valid Cloudera license.

    • Review the Helm chart reference before installation.

      The Helm chart accepts various configuration properties that you can set during installation. Using these properties you can customize your installation.

  • Prerequisites for channel encryption (TLS):

    • An Ingress controller is installed in your Kubernetes cluster. These steps use the Ingress-Nginx controller.

    • Optional: cert-manager is installed in your Kubernetes cluster.

  • Prerequisites for LDAP authentication:

    • An LDAP server is available that has TLS enabled.

    • The server is accessible from the Kubernetes cluster where Cloudera Surveyor is deployed.

    • Entries containing usernames and passwords are located under a common base in the directory information tree. Passwords must be stored in the userPassword attribute in the user entries.

  • Prerequisites for authorization:

    • Ensure that Kafka ACLs are set up for your Kafka cluster. Cloudera Surveyor uses Kafka ACLs to provide authorization.

  1. Create a namespace in your Kubernetes cluster. 
    kubectl create namespace [***NAMESPACE***]
    This is the namespace where you install Cloudera Surveyor. Use the namespace you create in all installation steps that follow.
  2. Prepare a Docker configuration JSON file that includes your Cloudera credentials. 
    {
  "auths": {
    "container.repository.cloudera.com": {
      "auth": "[***BASE64 ENCODED USERNAME:PASSWORD***]"
    }
  }
}

    The value of auth is the base64-encoded string of your credentials, with your username and password separated by a colon (:). For example, if your username is myuser and your password is pass, you must encode myuser:pass.

  3. Create a Kubernetes Secret containing your Cloudera credentials. 
    kubectl create secret docker-registry [***CLOUDERA CREDENTIALS SECRET***] \
  --from-file=.dockerconfigjson=[***PATH TO CLOUDERA CREDENTIALS JSON***] \
  --namespace=[***NAMESPACE***]
  4. Log in to the Cloudera Docker registry with helm. 
    helm registry login container.repository.cloudera.com

    Enter your Cloudera credentials when prompted.

  5. Create Secrets for sensitive Kafka client configuration values.
    Cloudera Surveyor connects to Kafka clusters as any other Kafka client and requires a client configuration. If the Kafka cluster is secured, the client configuration will include sensitive property values. Cloudera recommends that you store sensitive values in Secrets, mount the Secrets to the Cloudera Surveyor Container, and reference the values in your configuration instead of hard-coding them.
    Typically, you need to create two Secrets. One contains the Kafka cluster truststore and password, and the other contains a JAAS configuration. 
    kubectl create secret generic [***KAFKA TRUSTSTORE SECRET***] \
  --namespace [***NAMESPACE***] \
  --from-file=[***KAFKA TRUSTSTORE KEY***]=[***PATH TO TRUSTSTORE***] \
  --from-file=[***KAFKA TRUSTSTORE PASSWORD KEY***]=[***TRUSTSTORE PASSWORD FILE***]
    kubectl create secret generic [***KAFKA JAAS.CONF SECRET***] \
  --namesapce [***NAMESPACE***] \
  --from-file=[***KAFKA JAAS.CONF KEY***]=[***PATH TO JAAS.CONF***]
    Take note of the Secret names as well as the key names you configure. You will need to specify them in a later step.
  6. Prepare a certificate and private key for Cloudera Surveyor.

    • If you have cert-manager available, create a Certificate resource. Take note of the Secret name you configure in spec.secretname of the Certificate resource, you will need to specify it in a later step.

    • If you are managing keys manually, create a certificate and private key and save it to a Secret. Take note of the Secret name, you will need to specify it in a later step.

    This Secret is referred to as [***SURVEYOR TLS CERT SECRET***] in the following steps.
  7. Prepare a certificate and private key for Ingress.

    • If you have cert-manager available, the certificate and private key for Ingress are automatically requested by the Ingress. You only need to ensure that you have a valid Issuer available in cert-manager. You specify the name of the Issuer resource in a later step.

    • If you are managing keys manually, create a certificate and private key and save it to a Secret. Take note of the Secret name, you will need to specify it in a later step.

    This Secret referred to as [***INGRESS TLS CERT SECRET***] in the following steps.
  8. Set up resources for LDAP authentication.
    1. Generate an authentication key and save it as a file.
      The authentication key is a random sequence of at least 32 bytes that is saved to a file. To generate it, use any tool or method that is available to you. Take note of the location and name of the file that you create. You will need to specify the file in a later step.
    2. Generate a Java truststore (PKCS12 or JKS) containing the TLS certificate of the root Certificate Authority (CA) of the LDAP certificate chain. 
      keytool -import -trustcacerts -file [***LDAP ROOT CA***] \
  -keystore [***TRUSTSTORE NAME***] \
  -storepass [***TRUSTSTORE PASSWORD***] \
  -storetype PKCS12
    3. Create a Secret containing the truststore and the truststore password. 
      kubectl create secret generic [***LDAP TRUSTSTORE SECRET***] \
  --namespace [***NAMESPACE***] \
  --from-file=[***TRUSTSTORE SECRET KEY***]=[***TRUSTSTORE NAME***] \
  --from-file=[***TRUSTSTORE PW SECRET KEY***]=[***PATH TO TRUSTSTORE PW FILE***]
      Take note of [***LDAP TRUSTSTORE SECRET***], [***TRUSTSTORE SECRET KEY***], and [***TRUSTSTORE PW SECRET KEY***]. You will need to specify these names in a custom values file you create in a later step.
  9. Prepare a custom values file (values.yaml).

    The values file contains configuration for Cloudera Surveyor. This file specifies the Kafka clusters that Cloudera Surveyor connects to as well as various other configuration properties.

    clusterConfigs:
  clusters:
    - clusterName: [***CLUSTER NAME***]
      tags:
        - [***TAG1***]
        - [***TAG2***]
      bootstrapServers: [***BOOTSTRAP SERVERS***]
      commonClientConfig:
        security.protocol: "SASL_SSL"
        sasl.mechanism: PLAIN
        ssl.truststore.type: "pkcs12"
        ssl.truststore.location: "/opt/secrets/[***KAFKA TRUSTSTORE SECRET***]/[***KAFKA TRUSTSTORE FILE***]"
        ssl.truststore.password: "\\${dir:/opt/secrets/[***KAFKA TRUSTSTORE SECRET***]:[***KAFKA TRUSTSTORE PASSWORD FILE***]}"
        sasl.jaas.config: "\\${dir:/opt/secrets/[***KAFKA JAAS.CONF SECRET***]:[***KAFKA JAAS.CONF***]}"
      adminOperationTimeout: PT1M
      authorization:
        enabled: true
secretsToMount:
  - create: false
    secretRef: [***KAFKA TRUSTSTORE SECRET***]
    items:
      - key: [***KAFKA TRUSTSTORE KEY***] 
        path: [***KAFKA TRUSTSTORE FILE***] 
      - key: [***KAFKA TRUSTSTORE PASSWORD KEY***] 
        path: [***KAFKA TRUSTSTORE PASSWORD FILE***]
  - create: false
    secretRef: [***KAFKA JAAS.CONF SECRET***] 
    items:
      - key: [***KAFKA JAAS.CONF KEY***] 
        path: [***KAFKA JAAS.CONF***]
surveyorConfig:
  surveyor:
    authentication:
      enabled: true
  quarkus:
    security:
      ldap:
        dir-context:
          url: ldaps://openldap-chart.ldap:1390
          principal: cn=admin,dc=openldap-chart,dc=ldap
          password: adminpassword
        identity-mapping:
          rdn-identifier: uid
          search-base-dn: ou=users,dc=openldap-chart,dc=ldap
          attribute-mappings:
            "0":
              from: cn
              filter: (cn={0},ou=users,dc=openldap-chart,dc=ldap)
              filter-base-dn: ou=users,dc=openldap-chart,dc=ldap
tlsConfigs:
  enabled: true
  secretRef: [***SURVEYOR TLS CERT SECRET***]
ingress:
  enabled: true
  protocol: HTTPS
  className: nginx
  rules:
    path: "/"
    host: "my-domain.example.com"
    port: 8443
  tls:
    enabled: true
    secretRef: "[***INGRESS TLS CERT SECRET***]"
globalTruststore:
  secretRef:
    name: [***LDAP TRUSTSTORE SECRET***]
    key: [***TRUSTSTORE SECRET KEY***]
  type: PKCS12
  password:
    secretRef:
      name: [***LDAP TRUSTSTORE SECRET***]
      key: [***TRUSTSTORE PW SECRET KEY***]
    clusterConfigs specifies the Kafka clusters that Cloudera Surveyor connects to. Clusters specified here are the ones that will be available on the UI for monitoring and management.

    • clusterConfigs.clusters[*] – An array of Kafka clusters and their configuration. Each entry defines the configuration for a single Kafka cluster.

    • clusterConfigs.clusters[*].clustername – The name of the cluster. This name is displayed on the UI.

    • clusterConfigs.clusters[*].bootstrapServers – A comma-separated list of the bootstrap servers for the Kafka cluster that Cloudera Surveyor connects to. Specify multiple servers for highly available connections.

    • clusterConfigs.clusters[*].tags – User defined tags. Used for organization and filtering.

    • clusterConfigs.clusters[*].commonClientConfig – Kafka client configuration properties applied to all clients for this cluster. Must contain upstream Kafka client properties as a map. The exact properties that you specify here depend on the security configuration of the Kafka cluster that you want to connect. This example specifies a Kafka cluster that uses PLAIN authentication with TLS.

      Sensitive property values are referenced from Secrets instead of being hardcoded. Secrets containing sensitive properties are mounted using the secretsToMount property. References use Kafka DirectoryConfigProvider syntax. For more information, see Registering Kafka clusters.

    • clusterConfigs.clusters[*].authorization.enabled – Enables authorization for this cluster.

    secretsToMount specifies the Secrets to mount in the Cloudera Surveyor Container. You use secretsToMount to mount the sensitive values required for Kafka client configuration specified in clusterConfigs.

    • secretsToMount[*].create – Specifies whether to create the Secret. Set to false in this example as the Secrets are assumed to already exist.

    • secretsToMount[*].secretRef – The name of the Secret to mount.

    • secretsToMount[*].items[*].key – The key in the Secret to mount.

    • secretsToMount[*].items[*].path – The path where the item is mounted. The path is relative to /opt/secrets/[***SECRET NAME***]/ in the Cloudera Surveyor Container.

    surveyorConfig specifies global configuration for Cloudera Surveyor. This example sets various authentication properties.

    • surveyorConfig.surveyor.authentication.enabled - Enables or disables authentication. Set to true by default. Included in the example as a reference, you do not need to set the property explicitly to enable authentication.

    • surveyorconfig.surveyor.authentication.* - These properties control user session behavior and authentication token management. They define the session timeout, inactivity timeout, and the interval for renewing authentication tokens. All time intervals are specified in ISO 8601 format. For optimal security, Cloudera recommends that token renewal is shorter than or equal to half of the inactivity timeout.

    • surveyorconfig.quarkus.security.ldap.dircontext.* - These properties configure the LDAP server that Cloudera Surveyor connects to. They specify the server URL, the distinguished name (DN) of the bind user, and the password of the bind user. These are required for establishing a secure connection with the LDAP directory.

    • surveyorconfig.quarkus.security.ldap.identity-mapping.* - These properties configure how Cloudera Surveyor interacts with the LDAP directory to identify users and map their group memberships. They define the attributes and base DNs used to locate user entries and groups in the directory, as well as the filters applied to verify group membership.

    tlsConfigs enables TLS and specifies the Secret containing the certificate of Cloudera Surveyor.

    • tlsConfigs.enabled – Enables or disables TLS. Set to true by default. Included in the example as a reference, you do not need to set the property explicitly to enable TLS.

    • tlsConfigs.secretRef – Name of the Secret containing the Cloudera Surveyor certificate and key.

    ingress enables the creation of an Ingress. The Ingress provides secure external access to the Cloudera Surveyor UI.

    • ingress.enabled – Enables or disables Ingress.

    • ingress.protocol – Configures the Ingress protocol.

    • ingress.className – The class name of the Ingress controller. This example configures the Ingress-Nginx controller.

    • ingress.rules.host – Specifies the DNS hostname that the Ingress controller should match for incoming HTTP/HTTPS requests.

    • ingress.rules.port – The port of the Ingress rule. This is the port of the Kubernetes Service that the Ingress forwards requests to.

    • ingress.tls.enabled – Enables TLS for the Ingress.

    • ingress.tls.secretRef – The name of the Secret that contains Ingress TLS certificates.

    globalTruststore specifies the Secrets containing the truststore of the LDAP server and the password for the truststore.

    • globalTruststore.secretRef.name – The name of the Kubernetes Secret containing the truststore of the LDAP server.

    • globalTruststore.secretRef.key – The key in the Kubernetes Secret that contains the truststore.

    • globalTruststore.password.name – The name of the Kubernetes Secret containing the truststore password.

    • globalTruststore.password.key – The key in the Kubernetes Secret that contains the truststore password.

  10. Install Cloudera Surveyor with helm install. 
    helm install cloudera-surveyor \
  --namespace [***NAMESPACE***] \
  --values [***VALUES FILE***] \
  --set-file surveyorConfig.surveyor.authentication.keys.active=[***PATH TO AUTHENTICATION KEY FILE***] \
  --set 'image.imagePullSecrets=[***CLOUDERA CREDENTIALS SECRET***]' \
  --set-file clouderaLicense.fileContent=[***PATH TO LICENSE FILE***] \
  oci://container.repository.cloudera.com/cloudera-helm/csm-operator/surveyor \
  --version 1.4.0-b199

    • The string cloudera-surveyor is the Helm release name of the chart installation. This is an arbitrary, user defined name. Cloudera recommends that you use a unique and easily identifiable name.

    • [***VALUES FILE***] is the values file you prepared in Step 9.

    • surveyorConfig.surveyor.authentication.keys.active specifies the file containing the authentication key that you generated in Step 8. The key is required for LDAP authentication to function.

    • imagePullSecrets specifies what Secret is used to pull images from the Cloudera registry. Setting this property is mandatory, otherwise, Helm cannot pull the necessary images from the Cloudera Docker registry. Ensure that you replace [***CLOUDERA CREDENTIALS SECRET***] with the name of the Secret you created in Step 3.

    • clouderaLicense.fileContent is used to register your license. If this property is set, a Secret is generated that contains the license you specify. Setting this property is mandatory. Cloudera Surveyor will not function without a valid license. Ensure that you replace [***PATH TO LICENSE FILE***] with the full path to your Cloudera license file.

    • You can use --set to override properties that are defined in your values file, or add additional properties that are not present in your values file.

  11. Verify your installation.
    This is done by listing the Deployments and Pods in your namespace. If installation is successful, a Cloudera Surveyor Deployment and two Pods will be present in the cluster.
    kubectl get deployments --namespace [***NAMESPACE***]
    NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
#...
cloudera-surveyor   2/2     2            2           13m
    kubectl get pods --namespace [***NAMESPACE***]
    NAME                       READY   STATUS   RESTARTS   AGE
#...
cloudera-surveyor-649f755f6d-689gx   1/1     Running            0           13m
cloudera-surveyor-649f755f6d-xj2kp   1/1     Running            0           13m
  12. Access the Cloudera Surveyor UI.
    The UI is accessible by connecting to the Ingress.
    kubectl get ingress cloudera-surveyor-ingress --namespace [***NAMESPACE***]
    NAME                         CLASS   HOSTS                      ADDRESS     PORTS  
cloudera-surveyor-ingress   nginx   my-domain.example.com  10.14.91.1  80, 443

    Typically you will be able to access the UI through the host and port listed. However, the exact port that you have to use might be infrastructure dependent. If you are unable to connect, check the configuration of your Ingress controller. You can also try connecting directly to the Service of the Ingress.

Cloudera Surveyor is installed. You can now manage and monitor your Kafka clusters using the UI.

Installing Cloudera Surveyor in an air-gapped environment

Complete these steps to install Cloudera Surveyor if your Kubernetes cluster does not have internet access or if you want to install from a self-hosted registry. These steps install a fully secure instance of Cloudera Surveyor that has authentication, authorization, and channel encryption configured. The example configuration also demonstrates how you can connect a secure Kafka cluster to Cloudera Surveyor.

  • General prerequisites:

    • Ensure that your Kubernetes environment meets requirements listed in System requirements.

    • A self-hosted Docker registry is required. Your registry must be accessible by your Kubernetes cluster.

    • A machine with Internet connectivity is required. While the Kubernetes cluster does not need internet access, you will need a machine to pull the images from the Cloudera Docker registry.

    • Access to docker or equivalent utility that you can use to pull and push images is required. The following steps use docker. Replace commands where necessary.

    • Ensure that you have access to your Cloudera credentials (username and password). Credentials are required to access the Cloudera Archive and Cloudera Docker registry where installation artifacts are hosted.

    • Ensure that you have access to a valid Cloudera license.

    • Review the Helm chart reference before installation.

      The Helm chart accepts various configuration properties that you can set during installation. Using these properties you can customize your installation.

  • Prerequisites for channel encryption (TLS):

    • An Ingress controller is installed in your Kubernetes cluster. These steps use the Ingress-Nginx controller.

    • Optional: cert-manager is installed in your Kubernetes cluster.

  • Prerequisites for LDAP authentication:

    • An LDAP server is available that has TLS enabled.

    • The server is accessible from the Kubernetes cluster where Cloudera Surveyor is deployed.

    • Entries containing usernames and passwords are located under a common base in the directory information tree. Passwords must be stored in the userPassword attribute in the user entries.

  • Prerequisites for authorization:

    • Ensure that Kafka ACLs are set up for your Kafka cluster. Cloudera Surveyor uses Kafka ACLs to provide authorization.

  1. Copy the following installation artifacts to your self-hosted registry.
    Table 1. Cloudera Surveyor artifacts on the Cloudera Docker registry
    Artifact Location Description
    Cloudera Surveyor server Docker image container.repository.cloudera.com/cloudera/surveyor:0.1.0.1.4.0-b199 Docker image used for deploying Cloudera Surveyor.
    Cloudera Surveyor UI application Docker image container.repository.cloudera.com/cloudera/surveyor-app:0.1.0.1.4.0-b199 Docker image used for deploying Cloudera Surveyor.
    Cloudera Surveyor Helm chart oci://container.repository.cloudera.com/cloudera-helm/csm-operator/surveyor:1.4.0-b199 Helm chart used to install Cloudera Surveyor with helm-install.

    This step involves pulling the artifacts from the Cloudera Docker registry, retagging them, and then pushing them to your self-hosted registry. The exact steps you need to carry it out depend on your environment and how your registry is set up. The following substeps demonstrate the basic workflow using docker and helm.

    1. Log in to the Cloudera Docker registry with both docker and helm.
      Provide your Cloudera credentials when prompted.
      docker login container.repository.cloudera.com
      helm registry login container.repository.cloudera.com
    2. Pull the Docker images from the Cloudera Docker registry. 
      docker pull \
  container.repository.cloudera.com/cloudera/[***IMAGE NAME***]:[***VERSION***]
    3. Pull the Cloudera Surveyor Helm chart. 
      helm pull \
  oci://container.repository.cloudera.com/cloudera-helm/csm-operator/surveyor \
  --version 1.4.0-b199
    4. Retag the Docker images you pulled so that they contain the address of your registry. 
      docker tag \
  [***ORIGINAL IMAGE TAG***] \
  [***YOUR REGISTRY***]/cloudera/[***IMAGE NAME***]:[***VERSION***]
    5. Push the images and chart to your self-hosted registry. 
      docker push \
  [***YOUR REGISTRY***]/cloudera/[***IMAGE NAME***]:[***VERSION***]
      helm push \
  surveyor-1.4.0-b199.tgz \
  oci://[***REGISTRY HOSTNAME***]:[***PORT***]/cloudera-helm/csm-operator/surveyor
  2. Create a namespace in your Kubernetes cluster. 
    kubectl create namespace [***NAMESPACE***]
    This is the namespace where you install Cloudera Surveyor. Use the namespace you create in all installation steps that follow.
  3. Prepare a Docker configuration JSON file that includes credentials for your self-hosted registry. 
    {
  "auths": {
    "[****YOUR REGISTRY***]": {
      "auth": "[***BASE64 ENCODED USERNAME:PASSWORD***]"
    }
  }
}

    The value of auth is the base64-encoded string of your credentials, with your username and password separated by a colon (:). For example, if your username is myuser and your password is pass, you must encode myuser:pass.

  4. Create a Kubernetes Secret containing credentials for your self-hosted registry. 
    kubectl create secret docker-registry [***REGISTRY CREDENTIALS SECRET***] \
  --from-file=.dockerconfigjson=[***PATH TO REGISTRY CREDENTIALS JSON***] \
  --namespace=[***NAMESPACE***]
  5. Log in to your self-hosted registry with helm. 
    helm registry login [***REGISTRY HOSTNAME***]:[***PORT***]

    Enter your credentials when prompted.

  6. Create Secrets for sensitive Kafka client configuration values.
    Cloudera Surveyor connects to Kafka clusters as any other Kafka client and requires a client configuration. If the Kafka cluster is secured, the client configuration will include sensitive property values. Cloudera recommends that you store sensitive values in Secrets, mount the Secrets to the Cloudera Surveyor Container, and reference the values in your configuration instead of hard-coding them.
    Typically, you need to create two Secrets. One contains the Kafka cluster truststore and password, and the other contains a JAAS configuration. 
    kubectl create secret generic [***KAFKA TRUSTSTORE SECRET***] \
  --namespace [***NAMESPACE***] \
  --from-file=[***KAFKA TRUSTSTORE KEY***]=[***PATH TO TRUSTSTORE***] \
  --from-file=[***KAFKA TRUSTSTORE PASSWORD KEY***]=[***TRUSTSTORE PASSWORD FILE***]
    kubectl create secret generic [***KAFKA JAAS.CONF SECRET***] \
  --namesapce [***NAMESPACE***] \
  --from-file=[***KAFKA JAAS.CONF KEY***]=[***PATH TO JAAS.CONF***]
    Take note of the Secret names as well as the key names you configure. You will need to specify them in a later step.
  7. Prepare a certificate and private key for Cloudera Surveyor.

    • If you have cert-manager available, create a Certificate resource. Take note of the Secret name you configure in spec.secretname of the Certificate resource, you will need to specify it in a later step.

    • If you are managing keys manually, create a certificate and private key and save it to a Secret. Take note of the Secret name, you will need to specify it in a later step.

    This Secret is referred to as [***SURVEYOR TLS CERT SECRET***] in the following steps.
  8. Prepare a certificate and private key for Ingress.

    • If you have cert-manager available, the certificate and private key for Ingress are automatically requested by the Ingress. You only need to ensure that you have a valid Issuer available in cert-manager. You specify the name of the Issuer resource in a later step.

    • If you are managing keys manually, create a certificate and private key and save it to a Secret. Take note of the Secret name, you will need to specify it in a later step.

    This Secret referred to as [***INGRESS TLS CERT SECRET***] in the following steps.
  9. Set up resources for LDAP authentication.
    1. Generate an authentication key and save it as a file.
      The authentication key is a random sequence of at least 32 bytes that is saved to a file. To generate it, use any tool or method that is available to you. Take note of the location and name of the file that you create. You will need to specify the file in a later step.
    2. Generate a Java truststore (PKCS12 or JKS) containing the TLS certificate of the root Certificate Authority (CA) of the LDAP certificate chain. 
      keytool -import -trustcacerts -file [***LDAP ROOT CA***] \
  -keystore [***TRUSTSTORE NAME***] \
  -storepass [***TRUSTSTORE PASSWORD***] \
  -storetype PKCS12
    3. Create a Secret containing the truststore and the truststore password. 
      kubectl create secret generic [***LDAP TRUSTSTORE SECRET***] \
  --namespace [***NAMESPACE***] \
  --from-file=[***TRUSTSTORE SECRET KEY***]=[***TRUSTSTORE NAME***] \
  --from-file=[***TRUSTSTORE PW SECRET KEY***]=[***PATH TO TRUSTSTORE PW FILE***]
      Take note of [***LDAP TRUSTSTORE SECRET***], [***TRUSTSTORE SECRET KEY***], and [***TRUSTSTORE PW SECRET KEY***]. You will need to specify these names in a custom values file you create in a later step.
  10. Prepare a custom values file (values.yaml).

    The values file contains configuration for Cloudera Surveyor. This file specifies the Kafka clusters that Cloudera Surveyor connects to as well as various other configuration properties.

    clusterConfigs:
  clusters:
    - clusterName: [***CLUSTER NAME***]
      tags:
        - [***TAG1***]
        - [***TAG2***]
      bootstrapServers: [***BOOTSTRAP SERVERS***]
      commonClientConfig:
        security.protocol: "SASL_SSL"
        sasl.mechanism: PLAIN
        ssl.truststore.type: "pkcs12"
        ssl.truststore.location: "/opt/secrets/[***KAFKA TRUSTSTORE SECRET***]/[***KAFKA TRUSTSTORE FILE***]"
        ssl.truststore.password: "\\${dir:/opt/secrets/[***KAFKA TRUSTSTORE SECRET***]:[***KAFKA TRUSTSTORE PASSWORD FILE***]}"
        sasl.jaas.config: "\\${dir:/opt/secrets/[***KAFKA JAAS.CONF SECRET***]:[***KAFKA JAAS.CONF***]}"
      adminOperationTimeout: PT1M
      authorization:
        enabled: true
secretsToMount:
  - create: false
    secretRef: [***KAFKA TRUSTSTORE SECRET***]
    items:
      - key: [***KAFKA TRUSTSTORE KEY***] 
        path: [***KAFKA TRUSTSTORE FILE***] 
      - key: [***KAFKA TRUSTSTORE PASSWORD KEY***] 
        path: [***KAFKA TRUSTSTORE PASSWORD FILE***]
  - create: false
    secretRef: [***KAFKA JAAS.CONF SECRET***] 
    items:
      - key: [***KAFKA JAAS.CONF KEY***] 
        path: [***KAFKA JAAS.CONF***]
surveyorConfig:
  surveyor:
    authentication:
      enabled: true
  quarkus:
    security:
      ldap:
        dir-context:
          url: ldaps://openldap-chart.ldap:1390
          principal: cn=admin,dc=openldap-chart,dc=ldap
          password: adminpassword
        identity-mapping:
          rdn-identifier: uid
          search-base-dn: ou=users,dc=openldap-chart,dc=ldap
          attribute-mappings:
            "0":
              from: cn
              filter: (cn={0},ou=users,dc=openldap-chart,dc=ldap)
              filter-base-dn: ou=users,dc=openldap-chart,dc=ldap
tlsConfigs:
  enabled: true
  secretRef: [***SURVEYOR TLS CERT SECRET***]
ingress:
  enabled: true
  protocol: HTTPS
  className: nginx
  rules:
    path: "/"
    host: "my-domain.example.com"
    port: 8443
  tls:
    enabled: true
    secretRef: "[***INGRESS TLS CERT SECRET***]"
globalTruststore:
  secretRef:
    name: [***LDAP TRUSTSTORE SECRET***]
    key: [***TRUSTSTORE SECRET KEY***]
  type: PKCS12
  password:
    secretRef:
      name: [***LDAP TRUSTSTORE SECRET***]
      key: [***TRUSTSTORE PW SECRET KEY***]
    clusterConfigs specifies the Kafka clusters that Cloudera Surveyor connects to. Clusters specified here are the ones that will be available on the UI for monitoring and management.

    • clusterConfigs.clusters[*] – An array of Kafka clusters and their configuration. Each entry defines the configuration for a single Kafka cluster.

    • clusterConfigs.clusters[*].clustername – The name of the cluster. This name is displayed on the UI.

    • clusterConfigs.clusters[*].bootstrapServers – A comma-separated list of the bootstrap servers for the Kafka cluster that Cloudera Surveyor connects to. Specify multiple servers for highly available connections.

    • clusterConfigs.clusters[*].tags – User defined tags. Used for organization and filtering.

    • clusterConfigs.clusters[*].commonClientConfig – Kafka client configuration properties applied to all clients for this cluster. Must contain upstream Kafka client properties as a map. The exact properties that you specify here depend on the security configuration of the Kafka cluster that you want to connect. This example specifies a Kafka cluster that uses PLAIN authentication with TLS.

      Sensitive property values are referenced from Secrets instead of being hardcoded. Secrets containing sensitive properties are mounted using the secretsToMount property. References use Kafka DirectoryConfigProvider syntax. For more information, see Registering Kafka clusters.

    • clusterConfigs.clusters[*].authorization.enabled – Enables authorization for this cluster.

    secretsToMount specifies the Secrets to mount in the Cloudera Surveyor Container. You use secretsToMount to mount the sensitive values required for Kafka client configuration specified in clusterConfigs.

    • secretsToMount[*].create – Specifies whether to create the Secret. Set to false in this example as the Secrets are assumed to already exist.

    • secretsToMount[*].secretRef – The name of the Secret to mount.

    • secretsToMount[*].items[*].key – The key in the Secret to mount.

    • secretsToMount[*].items[*].path – The path where the item is mounted. The path is relative to /opt/secrets/[***SECRET NAME***]/ in the Cloudera Surveyor Container.

    surveyorConfig specifies global configuration for Cloudera Surveyor. This example sets various authentication properties.

    • surveyorConfig.surveyor.authentication.enabled - Enables or disables authentication. Set to true by default. Included in the example as a reference, you do not need to set the property explicitly to enable authentication.

    • surveyorconfig.surveyor.authentication.* - These properties control user session behavior and authentication token management. They define the session timeout, inactivity timeout, and the interval for renewing authentication tokens. All time intervals are specified in ISO 8601 format. For optimal security, Cloudera recommends that token renewal is shorter than or equal to half of the inactivity timeout.

    • surveyorconfig.quarkus.security.ldap.dircontext.* - These properties configure the LDAP server that Cloudera Surveyor connects to. They specify the server URL, the distinguished name (DN) of the bind user, and the password of the bind user. These are required for establishing a secure connection with the LDAP directory.

    • surveyorconfig.quarkus.security.ldap.identity-mapping.* - These properties configure how Cloudera Surveyor interacts with the LDAP directory to identify users and map their group memberships. They define the attributes and base DNs used to locate user entries and groups in the directory, as well as the filters applied to verify group membership.

    tlsConfigs enables TLS and specifies the Secret containing the certificate of Cloudera Surveyor.

    • tlsConfigs.enabled – Enables or disables TLS. Set to true by default. Included in the example as a reference, you do not need to set the property explicitly to enable TLS.

    • tlsConfigs.secretRef – Name of the Secret containing the Cloudera Surveyor certificate and key.

    ingress enables the creation of an Ingress. The Ingress provides secure external access to the Cloudera Surveyor UI.

    • ingress.enabled – Enables or disables Ingress.

    • ingress.protocol – Configures the Ingress protocol.

    • ingress.className – The class name of the Ingress controller. This example configures the Ingress-Nginx controller.

    • ingress.rules.host – Specifies the DNS hostname that the Ingress controller should match for incoming HTTP/HTTPS requests.

    • ingress.rules.port – The port of the Ingress rule. This is the port of the Kubernetes Service that the Ingress forwards requests to.

    • ingress.tls.enabled – Enables TLS for the Ingress.

    • ingress.tls.secretRef – The name of the Secret that contains Ingress TLS certificates.

    globalTruststore specifies the Secrets containing the truststore of the LDAP server and the password for the truststore.

    • globalTruststore.secretRef.name – The name of the Kubernetes Secret containing the truststore of the LDAP server.

    • globalTruststore.secretRef.key – The key in the Kubernetes Secret that contains the truststore.

    • globalTruststore.password.name – The name of the Kubernetes Secret containing the truststore password.

    • globalTruststore.password.key – The key in the Kubernetes Secret that contains the truststore password.

  11. Install Cloudera Surveyor with helm install. 
    helm install cloudera-surveyor \
  --namespace [***NAMESPACE***] \
  --values [***VALUES FILE***] \
  --set-file surveyorConfig.surveyor.authentication.keys.active=[***PATH TO AUTHENTICATION KEY FILE***] \
  --set image.registry=[***REGISTRY HOSTNAME***]:[***PORT***] \
  --set 'image.imagePullSecrets=[***REGISTRY CREDENTIALS SECRET***]' \
  --set-file clouderaLicense.fileContent=[***PATH TO LICENSE FILE***] \
  oci://[***YOUR REGISTRY***]/cloudera-helm/csm-operator/surveyor \
  --version 1.4.0-b199

    • The string cloudera-surveyor is the Helm release name of the chart installation. This is an arbitrary, user defined name. Cloudera recommends that you use a unique and easily identifiable name.

    • [***VALUES FILE***] is the values file you prepared in Step 10.

    • surveyorConfig.surveyor.authentication.keys.active specifies the file containing the authentication key that you generated in Step 9. The key is required for LDAP authentication to function.

    • imagePullSecrets specifies what Secret is used to pull images from the specified registry. Ensure that you replace [***REGISTRY CREDENTIALS SECRET***] with the name of the Secret you created in Step 4.

    • clouderaLicense.fileContent is used to register your license. If this property is set, a Secret is generated that contains the license you specify. Setting this property is mandatory. Cloudera Surveyor will not function without a valid license. Ensure that you replace [***PATH TO LICENSE FILE***] with the full path to your Cloudera license file.

    • You can use --set to override properties that are defined in your values file, or add additional properties that are not present in your values file.

  12. Verify your installation.
    This is done by listing the Deployments and Pods in your namespace. If installation is successful, a Cloudera Surveyor Deployment and two Pods will be present in the cluster.
    kubectl get deployments --namespace [***NAMESPACE***]
    NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
#...
cloudera-surveyor   2/2     2            2           13m
    kubectl get pods --namespace [***NAMESPACE***]
    NAME                       READY   STATUS   RESTARTS   AGE
#...
cloudera-surveyor-649f755f6d-689gx   1/1     Running            0           13m
cloudera-surveyor-649f755f6d-xj2kp   1/1     Running            0           13m
  13. Access the Cloudera Surveyor UI.
    The UI is accessible by connecting to the Ingress.
    kubectl get ingress cloudera-surveyor-ingress --namespace [***NAMESPACE***]
    NAME                         CLASS   HOSTS                      ADDRESS     PORTS  
cloudera-surveyor-ingress   nginx   my-domain.example.com  10.14.91.1  80, 443

    Typically you will be able to access the UI through the host and port listed. However, the exact port that you have to use might be infrastructure dependent. If you are unable to connect, check the configuration of your Ingress controller. You can also try connecting directly to the Service of the Ingress.

Cloudera Surveyor is installed. You can now manage and monitor your Kafka clusters using the UI.

Installing Cloudera Surveyor for evaluation

Complete these steps to install a basic deployment of Cloudera Surveyor that has no security configured. Use these instructions if you want to install quickly in a development environment for proof of concept or evaluation purposes.

  • Ensure that your Kubernetes environment meets requirements listed in System requirements.

  • Your Kubernetes cluster requires internet connectivity to complete these steps. It must be able to reach the Cloudera Docker registry.

  • Ensure that you have access to your Cloudera credentials (username and password). Credentials are required to access the Cloudera Archive and Cloudera Docker registry where installation artifacts are hosted.

  • Ensure that you have access to a valid Cloudera license.

  • Review the Helm chart reference before installation.

    The Helm chart accepts various configuration properties that you can set during installation. Using these properties you can customize your installation.

  1. Create a namespace in your Kubernetes cluster. 
    kubectl create namespace [***NAMESPACE***]
    This is the namespace where you install Cloudera Surveyor. Use the namespace you create in all installation steps that follow.
  2. Prepare a Docker configuration JSON file that includes your Cloudera credentials. 
    {
  "auths": {
    "container.repository.cloudera.com": {
      "auth": "[***BASE64 ENCODED USERNAME:PASSWORD***]"
    }
  }
}

    The value of auth is the base64-encoded string of your credentials, with your username and password separated by a colon (:). For example, if your username is myuser and your password is pass, you must encode myuser:pass.

  3. Create a Kubernetes Secret containing your Cloudera credentials. 
    kubectl create secret docker-registry [***CLOUDERA CREDENTIALS SECRET***] \
  --from-file=.dockerconfigjson=[***PATH TO CLOUDERA CREDENTIALS JSON***] \
  --namespace=[***NAMESPACE***]
  4. Log in to the Cloudera Docker registry with helm. 
    helm registry login container.repository.cloudera.com

    Enter your Cloudera credentials when prompted.

  5. Prepare a custom values file (values.yaml).
    The values file contains configuration for Cloudera Surveyor. This file specifies the Kafka clusters that Cloudera Surveyor connects to as well as various other configuration properties.
    clusterConfigs:
  clusters:
    - clusterName: [***CLUSTER NAME***]
      tags:
        - [***TAG1***]
        - [***TAG2***]
      bootstrapServers: [***BOOTSTRAP SERVERS***]
      adminOperationTimeout: PT1M
      authorization:
        enabled: false
      commonClientConfig:
        security.protocol: PLAINTEXT
surveyorConfig:
  surveyor:
    authentication:
      enabled: false
tlsConfigs:
  enabled: false

    • clusterConfigs.clusters[*] – An array of Kafka clusters and their configuration. Each entry defines the configuration for a single Kafka cluster.

    • clusterConfigs.clusters[*].clustername – The name of the cluster. This name is displayed on the UI.

    • clusterConfigs.clusters[*].bootstrapServers – A comma-separated list of the bootstrap servers for the Kafka cluster that Cloudera Surveyor connects to. Specify multiple servers for highly available connections.

    • clusterConfigs.clusters[*].commonClientConfig – Kafka client configuration properties applied to all clients for this cluster. Must contain upstream Kafka client properties as a map. The exact properties that you specify here depend on the security configuration of the Kafka cluster that you want to connect. This example specifies a Kafka cluster that is unsecure. For more information, see Registering Kafka clusters.

    • All security-related properties are set false to disable security. These properties must be explicitly set to false as the default value for all of them is true.

  6. Install Cloudera Surveyor with helm install. 
    helm install cloudera-surveyor \
  --namespace [***NAMESPACE***] \
  --values [***VALUES FILE***] \
  --set 'image.imagePullSecrets=[***CLOUDERA CREDENTIALS SECRET***]' \
  --set-file clouderaLicense.fileContent=[***PATH TO LICENSE FILE***] \
  oci://container.repository.cloudera.com/cloudera-helm/csm-operator/surveyor \
  --version 1.4.0-b199

    • The string cloudera-surveyor is the Helm release name of the chart installation. This is an arbitrary, user defined name. Cloudera recommends that you use a unique and easily identifiable name.

    • [***VALUES FILE***] is the values file you prepared in Step 5.

    • imagePullSecrets specifies what secret is used to pull images from the Cloudera registry. Setting this property is mandatory, otherwise, Helm cannot pull the necessary images from the Cloudera Docker registry. Ensure that you replace [***CLOUDERA CREDENTIALS SECRET***] with the name of the Secret you created in Step 3.

    • clouderaLicense.fileContent is used to register your license. If this property is set, a Secret is generated that contains the license you specify. Setting this property is mandatory. Cloudera Surveyor will not function without a valid license. Ensure that you replace [***PATH TO LICENSE FILE***] with the full path to your Cloudera license file.

    • You can use --set to override properties that are defined in your values file, or add additional properties that are not present in your values file.

  7. Verify your installation.
    This is done by listing the Deployments and Pods in your namespace. If installation is successful, a Cloudera Surveyor Deployment and two Pods will be present in the cluster.
    kubectl get deployments --namespace [***NAMESPACE***]
    NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
#...
cloudera-surveyor   2/2     2            2           13m
    kubectl get pods --namespace [***NAMESPACE***]
    NAME                       READY   STATUS   RESTARTS   AGE
#...
cloudera-surveyor-649f755f6d-689gx   1/1     Running            0           13m
cloudera-surveyor-649f755f6d-xj2kp   1/1     Running            0           13m
  8. Access the Cloudera Surveyor UI.
    Installation by default creates a NodePort type Service for Cloudera Surveyor. The UI is accessible from any of the Kubernetes cluster nodes on the external port of the Service. List Services to get the external port.
    kubectl get service cloudera-surveyor-service --namespace [***NAMESPACE***]
    NAME                        TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
cloudera-surveyor-service   NodePort   10.43.196.52   <none>        8080:30525/TCP   16m
    In this example, the external port is 30525.
Cloudera Surveyor is installed. You can now manage and monitor your Kafka clusters using the UI.