Installing Cloudera Surveyor for Apache Kafka on Taikun CloudWorks [Technical Preview]

Learn how to install Cloudera Surveyor in Cloudera Streams Messaging - Kubernetes Operator on Taikun CloudWorks. Installation involves importing the Cloudera Surveyor repository, adding Cloudera Surveyor to a new or existing catalog, and installing Cloudera Surveyor using the Taikun CloudWorks UI.

You have access to a project and Kubernetes cluster in Taikun CloudWorks.
Your Kubernetes environment meets requirements listed in System requirements.
The worker nodes in your cluster meet the minimum sizing requirements listed in Sizing and performance considerations.
Access to your cluster with kubectl is configured. For more information, see Accessing Cluster with Kubeconfig.
Your Kubernetes cluster requires internet connectivity to complete these steps. It must be able to reach the Cloudera Docker registry.
You have access to your Cloudera credentials (username and password). Credentials are required to access the Cloudera Archive and Cloudera Docker registry where installation artifacts are hosted.
You have access to a valid Cloudera license.
Review the Helm chart reference before installation.
The Helm chart accepts various configuration properties that you can set during installation. Using these properties you can customize your installation.

Importing the Cloudera Surveyor repository and adding Cloudera Surveyor to a catalog

Complete these steps to import the Cloudera Surveyor repository and to add Cloudera Surveyor to a new or existing catalog in Taikun CloudWorks.

Import the Cloudera Surveyor repository.
1. In Taikun CloudWorks, go to Repositories and select the Private tab.
2. Click Import Repository.
3. Enter the following in Import Repository:
  - Enter a unique name in Name.
  - Enter the following OCI repository URL in URL:
```
oci://container.repository.cloudera.com/cloudera-helm/csm-operator/surveyor
```
  - Enter your Cloudera credentials in Username and Password.
4. Click Import.
Add Cloudera Surveyor to a catalog.

tip
These instructions create a new catalog. You can also add your application to an existing catalog.
1. Go to Catalogs and click Add Catalog.
2. Enter a catalog name and description in Create Catalog.
3. Click Save.
4. Go to [***YOUR CATALOG***] and click Add Applications.
5. Select [***YOUR REPOSITORY***] from the Repository drop-down list and click Apply.
6. Find the surveyor application in the list of available applications and click .
7. Click Add to the catalog.
Add catalog app parameters.
1. Click Add Parameters.
2. Find and add the clouderaLicense.secretRef parameter.
3. Set the clouderaLicense.secretRef parameter to csm-op-license.
4. Click Save.

Installing Cloudera Surveyor

Complete these steps to install Cloudera Surveyor on Taikun CloudWorks.

These instructions walk you through installing an unsecure instance of Cloudera Surveyor on Taikun CloudWorks. Use these instructions for quick installation in development environments for proof of concept and evaluation purposes.

While the security configuration of Cloudera Surveyor itself is not covered, the example configuration demonstrates how to register a Kafka cluster that has TLS encryption enabled.

Create a namespace in your Kubernetes cluster.
```
kubectl create namespace [***NAMESPACE***]
```
Use this namespace in all of the following installation steps.

Create a Kubernetes Secret containing your Cloudera license.

kubectl create secret generic csm-op-license \
  --namespace [***NAMESPACE***] \
  --from-file=license=[***PATH TO LICENSE FILE***]

Create a Kubernetes Secret containing your Cloudera credentials.
```
kubectl create secret docker-registry [***REGISTRY CREDENTIALS SECRET***] \
  --namespace [***NAMESPACE***] \
  --docker-server container.repository.cloudera.com \
  --docker-username [***USERNAME***] \
  --docker-password "$(echo -n 'Enter your Cloudera password: ' >&2; read -s password; echo >&2; echo $password)"
```
- Take note of the name you specify as [***REGISTRY CREDENTIALS SECRET***]. You will need to specify the name in a later step.
- Replace [***USERNAME***] with your Cloudera username.
- Enter your Cloudera password when prompted.
Create Secrets for sensitive Kafka client configuration values.
Cloudera Surveyor connects to Kafka clusters as any other Kafka client and requires a client configuration. If the Kafka cluster is secured, the client configuration will include sensitive property values. Cloudera recommends that you store sensitive values in Secrets, mount the Secrets to the Cloudera Surveyor Container, and reference the values in your configuration instead of hard-coding them.
Typically, you need to create two Secrets. One contains the Kafka cluster truststore and password, and the other contains a JAAS configuration. The Kafka truststore must be in JKS or PKCS12 format.
```
kubectl create secret generic [***KAFKA TRUSTSTORE SECRET***] \
  --namespace [***NAMESPACE***] \
  --from-file=[***KAFKA TRUSTSTORE KEY***]=[***PATH TO TRUSTSTORE***] \
  --from-file=[***KAFKA TRUSTSTORE PASSWORD KEY***]=[***PATH TO TRUSTSTORE PASSWORD FILE***]
```
```
kubectl create secret generic [***KAFKA SASL.JAAS.CONFIG SECRET***] \
  --namespace [***NAMESPACE***] \
  --from-file=[***KAFKA SASL.JAAS.CONFIG KEY***]=[***PATH TO KAFKA SASL.JAAS.CONFIG FILE***]
```
- Take note of the Secret names as well as the key names you configure. You will need to specify them in a later step.
- All key names like [***KAFKA TRUSTSTORE KEY***] or [***KAFKA SASL.JAAS.CONFIG KEY***] are arbitrary.
- [***PATH TO KAFKA SASL.JAAS.CONFIG FILE***] is a path to a file containing JAAS configuration similar to the following example:
```
org.apache.kafka.common.security.plain.PlainLoginModule required username="my-user" password="my-password";
```
  The contents of this file are set in a later step as the value of the sasl.jaas.config Kafka client property for Cloudera Surveyor internal Kafka clients. Ensure that the format of the configuration is valid for the sasl.jaas.config property. That is, its a single line of configuration.
Install Cloudera Surveyor.
1. In Taikun CloudWorks, go to Projects > [***YOUR PROJECT***] > Applications.
2. Click Install.
3. Search for surveyor.
4. Find the surveyor application in the list of available applications. Select the one that is in [*** YOUR CATALOG ***]and click .
5. Click Bind if you get a prompt to bind the catalog to your project.
6. Configure the following common settings in Application Instance:
  - Enter a name in Application Instance Name.
  - In Namespace, select the namespace you created in Step 1.
  - Enable the Extra Values tab by clicking the Extra Values toggle.
7. Click Continue.
8. In Installation Params, set clouderaLicense.secretRef to the name of the Secret you created in Step 2.
  
  tip
  Required parameters might already use correct default values.
9. Click Continue.
10. Provide the following values in Extra Values:
```
image:
 imagePullSecrets: [***REGISTRY CREDENTIALS SECRET***]
clusterConfigs:
 clusters:
   - clusterName: [***CLUSTER NAME***]
     tags:
       - [***TAG1***]
       - [***TAG2***]
     bootstrapServers: [***BOOTSTRAP SERVERS***]
     commonClientConfig:
       security.protocol: "SSL"
       ssl.truststore.type: "pkcs12"
       ssl.truststore.location: "/opt/secrets/[***KAFKA TRUSTSTORE SECRET***]/[***KAFKA TRUSTSTORE FILE***]"
       ssl.truststore.password: "\\${dir:/opt/secrets/[***KAFKA TRUSTSTORE SECRET***]:[***KAFKA TRUSTSTORE PASSWORD FILE***]}"
       sasl.jaas.config: "\\${dir:/opt/secrets/[***KAFKA SASL.JAAS.CONFIG SECRET***]:[***KAFKA SASL.JAAS.CONFIG FILE***]}"
     adminOperationTimeout: PT1M
     authorization:
       enabled: false
secretsToMount:
 - create: false
   secretRef: [***KAFKA TRUSTSTORE SECRET***]
   items:
     - key: [***KAFKA TRUSTSTORE KEY***]
       path: [***KAFKA TRUSTSTORE FILE***]
     - key: [***KAFKA TRUSTSTORE PASSWORD KEY***]
       path: [***KAFKA TRUSTSTORE PASSWORD FILE***]
 - create: false
   secretRef: [***KAFKA SASL.JAAS.CONFIG SECRET***]
   items:
     - key: [***KAFKA SASL.JAAS.CONFIG KEY***]
       path: [***KAFKA SASL.JAAS.CONFIG FILE***]
surveyorConfig:
 surveyor:
   authentication:
     enabled: false
tlsConfigs:
 enabled: false
ingress:
 enabled: true
 protocol: HTTP
 className: taikun
 rules:
   host: "[***SUBDOMAIN***].[***DOMAIN NAME***]"
```
  image contains all configuration settings for the container image. This can include the image repository, tag version, pull policy, and any required authentication Secrets for accessing the registry.
  In this example, image.imagePullSecrets is set. This property specifies the Secret that contains your Cloudera credentials. These credentials are used to access the Cloudera Docker registry to pull required images. Replace [***REGISTRY CREDENTIALS SECRET***] with the name of the Secret you created in Step 3.
  
  clusterConfigs specifies the Kafka clusters that Cloudera Surveyor connects to. Clusters specified here are the ones that will be available on the UI for monitoring and management. For more information and additional examples, see Registering Kafka clusters.
  
  clusterConfigs.clusters[*] – An array of Kafka clusters and their configuration. Each entry defines the configuration for a single Kafka cluster.
  
  clusterConfigs.clusters[*].clustername – The name of the cluster. This name is displayed on the UI.
  
  clusterConfigs.clusters[*].bootstrapServers – A comma-separated list of the bootstrap servers for the Kafka cluster that Cloudera Surveyor connects to. Specify multiple servers for highly available connections.
  
  clusterConfigs.clusters[*].tags – User defined tags. Used for organization and filtering.
  
  clusterConfigs.clusters[*].commonClientConfig – Kafka client configuration properties applied to all clients for this cluster. Must contain upstream Kafka client properties as a map. The exact properties that you specify here depend on the security configuration of the Kafka cluster that you want to connect. This example specifies a Kafka cluster that uses PLAIN authentication with TLS.
  
  Sensitive property values are referenced from Secrets instead of being hardcoded. Secrets containing sensitive properties are mounted using the secretsToMount property. References use Kafka DirectoryConfigProvider syntax.
  
  note
  References in the client configurations must be escaped because Cloudera Surveyor itself uses the same syntax for references.
  
  clusterConfigs.clusters[*].authorization.enabled – Enables or disables authorization for this cluster.
  
  secretsToMount specifies the Secrets to mount in the Cloudera Surveyor Container. You use secretsToMount to mount the sensitive values required for Kafka client configuration specified in clusterConfigs. For more information and additional examples, see Managing sensitive data in client configuration
  
  secretsToMount[*].create – Specifies whether to create the Secret. Set to false in this example as the Secrets are assumed to already exist.
  
  secretsToMount[*].secretRef – The name of the Secret to mount.
  
  secretsToMount[*].items[*].key – The key in the Secret to mount.
  
  secretsToMount[*].items[*].path – The path where the item is mounted. The path is relative to /opt/secrets/[***SECRET NAME***]/ in the Cloudera Surveyor Container.
  
  ingress enables the creation of an Ingress. The Ingress provides secure external access to the Cloudera Surveyor UI.
  
  ingress.enabled – Enables or disables Ingress.
  
  ingress.protocol – Configures the Ingress protocol.
  
  ingress.className – The class name of the Ingress controller. This example configures the Ingress-Nginx controller.
  
  ingress.rules.host – Specifies the DNS hostname that the Ingress controller should match for incoming HTTP/HTTPS requests.
  
  Configure this property as follows:
  
  [***SUBDOMAIN***] – Arbitrary and unique subdomain or service name that identifies the application. For example: surveyor.
  
  [***DOMAIN NAME***] – A domain that resolves to the access IP of your Kubernetes cluster.
  
  If you do not have DNS configured or do not know your domain, use a wildcard DNS service like sslip.io. If you use a wildcard DNS service, the value you enter must have the following format:
  [***SUBDOMAIN***].[***ACCESS IP***].[***WILDCARD DNS SERVICE***]
  For example:
  surveyor.203.0.113.255.sslip.io
  You can find the access IP of the cluster in Taikun CloudWorks by going to Projects > [***YOUR PROJECT***]
11. Validate your extra values by clicking Check extra values.
12. Click Run installation.
Go to Projects > [***YOUR PROJECT***] > LiveOps to verify your installation.
If installation is successful, a Cloudera Surveyor Deployment and two Pods will be present in the cluster in the installation namespace. These resources use the name you specified in Application Instance Name.
Access the Cloudera Surveyor UI.
You can access the UI through the host you configured in ingress.rules.host.

Cloudera Surveyor is installed. You can now manage and monitor your Kafka clusters using the UI.