Cloudera Docs

Initializing virtual clusters

You must initialize the cluster by performing manual steps for each virtual cluster you create.

  1. Download cde-utils.sh to your local machine.
  2. Create a directory to store the files, and change to that directory: 
    mkdir -p /tmp/cde-utils && cd /tmp/cde-utils
  3. Copy the script

    Embedded Container Service (ECS)

    Copy the extracted utility script (cde-utils.sh) to the Embedded Container Service (ECS) cluster host which has the ECS Master installed. To identify the ECS cluster hosts:

    1. Log in to the Cloudera Manager web interface.
    2. Click Clusters tab.
    3. Click the relevant ECS cluster from the list of the clusters displayed.
    4. Under Status, click Hosts link.
    5. Select the master host from the list and copy the script to that host.

    Red Hat OpenShift Container Platform (OCP)

    Copy the extracted utility script (cde-utils.sh) and the OpenShift kubeconfig file to one of the HDFS service gateway hosts, and install the kubectl utility:

    • Log in to the Cloudera Manager web interface.
    • Go to Clusters > Base Cluster > HDFS > Instances.
    • Select one of the Gateway hosts, log in using the security password that was set, and copy the script to that host.
    • Copy the OCP kubeconfig file to the same host.
    • Export the OCP kubeconfig file:
      export KUBECONFIG=[***path_of_the_copied_OCP_Kubeconfig_file***]
    • On that host, install the kubectl utility following the instructions in the Kubernetes documentation. Cloudera recommends installing the version that matches the Kubernetes version installed on the OpenShift cluster.
  4. On the cluster host that you copied the script to, set the script permissions to be executable: 
    chmod +x /path/to/cde-utils.sh
  5. Identify the virtual cluster endpoint:
    1. In the Cloudera Manager web UI, go to the Data Services page, and then click Open CDP Private Cloud Data Services.
    2. Click the Data Engineering tile.
    3. Select the CDE service containing the virtual cluster you want to activate.
    4. Click Cluster Details.
    5. Click JOBS API URL to copy the URL to your clipboard.
    6. Paste the URL into a text editor to identify the endpoint host. For example, the URL is similar to the following:
      http://dfdj6kgx.cde-2cdxw5x5.apps.ecs-demo.example.com/dex/api/v1

      The endpoint host is dfdj6kgx.cde-2cdxw5x5.apps.ecs-demo.example.com.

  6. On the ECS or HDFS gateway host you selected previously, initialize the virtual cluster using the cde-utils.sh script. You can either generate and use a self-signed certificate, or provide a signed certificate and private key.
    Generate a self-signed certificate 
    ./cde-utils.sh init-virtual-cluster -h <endpoint_host> -a
    For example, using the previous example URL, the endpoint host is dfdj6kgx.cde-2cdxw5x5.apps.ecs-demo.example.com:
    ./cde-utils.sh init-virtual-cluster -h dfdj6kgx.cde-2cdxw5x5.apps.ecs-demo.example.com -a

    Use a signed certificate and private key

    Make sure that the certificate is a wildcard certificate for the cluster endpoint and it includes the Subject Alternative Names (SAN) for all relevant endpoints. For example, *.cde-2cdxw5x5.apps.ecs-demo.example.com

    When generating the certificate, make sure that the SAN includes all the possible subdomains that you will use. For example, dfdj6kgx.cde-2cdxw5x5.apps.ecs-demo.example.com

    ./cde-utils.sh init-virtual-cluster -h <endpoint_host> -c /path/to/cert -k /path/to/keyfile
    For example, using the previous example URL, the endpoint host is dfdj6kgx.cde-2cdxw5x5.apps.ecs-demo.example.com:
    ./cde-utils.sh init-virtual-cluster -h dfdj6kgx.cde-2cdxw5x5.apps.ecs-demo.example.com  -c /tmp/cde-pvc.crt -k /tmp/cde-pvc.key

    You must perform this procedure for each virtual cluster you create.