Configuring TLS Encryption for Cloudera Manager and CDH Using Auto-TLS

Auto-TLS, first introduced in Cloudera Manager 5.13 on Cloudera Director 2.6, is now available for on-premises clusters in Cloudera Enterprise 6.

An Enterprise license is required to enable Auto-TLS.

Auto-TLS greatly simplifies the process of enabling and managing TLS encryption on your cluster. It automates the creation of an internal certificate authority (CA) and deployment of certificates across all cluster hosts. It can also automate the distribution of existing certificates, such as those signed by a public CA. Adding new cluster hosts or services to a cluster with auto-TLS enabled automatically creates and deploys the required certificates.

To configure TLS encryption manually, see Manually Configuring TLS/SSL Encryption for CDH Services and Manually Configuring TLS Encryption for Cloudera Manager.

Starting with Cloudera Manager 6.2, you can enable auto-TLS on existing deployments. For instructions on enabling auto-TLS as part of a new installation, see the installation guide.

Auto-TLS Requirements and Limitations

  • You must install the Cloudera Manager Agent software on the Cloudera Manager Server host.
  • You can enable auto-TLS using certificates created and managed by a Cloudera Manager certificate authority (CA), or certificates signed by a trusted public CA or your own internal CA. If you want to use a trusted public CA or your own internal CA, you must obtain all of the host certificates before enabling auto-TLS. For instructions on obtaining certificates from a CA, see Generate TLS Certificates.
The following Cloudera Enterprise services support auto-TLS:
  • Cloudera Navigator Audit Server
  • Cloudera Navigator Metadata Server
  • Cloudera Manager Host Monitor Debug Interface
  • Cloudera Manager Service Monitor Debug Interface
  • Flume
  • HDFS Client Configuration
  • HDFS NameNode Web UI
  • HiveServer2
  • HttpFS
  • Hue Client
  • Hue Load Balancer
  • Hue Server
  • Impala Catalog Server
  • Impala Server
  • Impala StateStore
  • Kafka Broker Server
  • Oozie
  • Spark History Server
  • YARN Web UI

For more information on how to use Auto-TLS with Flume, see Configuring TLS/SSL for Flume.

For unlisted Cloudera Enterprise services, you must enable TLS manually.

The certmanager Utility

Auto-TLS is managed using the certmanager utility, which is included in the Cloudera Manager Agent software, and not the Cloudera Manager Server software. You must install the Cloudera Manager Agent software on the Cloudera Manager Server host to be able to use the utility. You can use certmanager to manage auto-TLS on a new installation.

The certmanager syntax is as follows:

/opt/cloudera/cm-agent/bin/certmanager [OPTIONS] COMMAND [ARGS]...
  • Options:
    • --location <certmanager-dir-root>

      The directory where certmanager stores all of its files on the Cloudera Manager Server host. If omitted, defaults to /var/lib/cloudera-scm-server/certmanager. This directory is created automatically, and must not exist before running the command. If it does exist, you can use the --rotate argument (documented below) to back up the existing directory and create a new one in its place.

      The agent host certificates and other files are stored elsewhere on each agent (see Auto-TLS Agent File Locations).

    • --help

      Displays the help message.

  • Commands:
    • add_custom_cert

      Adds a custom certificate and key for a host. Use this command only if you have configured a custom certificate directory (using the setup_custom_certdir command). You must run this command before adding a host in Cloudera Manager.

    • export_ca_cert

      Displays the Cloudera Manager internal CA certificate. You can export the certificate to a file using a redirect operator (> or >>).

    • setup

      Initializes the certificate manager and the internal CA, and configures Cloudera Manager Server to enable auto-TLS.

      • --configure-services

        Configures Cloudera Manager Server to enable automatic configuration of TLS for supported components, such as HDFS, YARN, and so on. If you omit this option, auto-TLS will only be configured for Cloudera Manager agent/server communication.

      • --rotate

        Backs up the certmanager root directory (/var/lib/cloudera-scm-server/certmanager by default, or specified by the --location option) if it exists, and creates a new one in its place. If the directory does not exist, it is created. If the directory exists, and you do not use the --rotate argument, the command fails.

      • --override ca_dn="<CA_DN>"

        Overrides the default CA distinguished name (DN) with the provided DN. Use this if your environment requires that the common name (CN) matches the hostname. For example:

        --override ca_dn="CN=cm01,DC=example,DC=com"
      • --stop-at-csr

        Stops the setup process after generating the private key and certificate signing request (CSR) for an intermediate CA certificate, and outputs the CSR file location to the screen. Submit the provided CSR to your internal root CA for signing. After receiving the signed intermediate CA certificate, continue the setup using the --signed-ca-cert parameter.

        When using the --stop-at-csr and --signed-ca-cert arguments, make sure that the remaining command options and arguments are the same.

      • --signed-ca-cert=<intermediate_CA_cert>

        Resumes the setup process using the provided signed intermediate CA certificate.

        When using the --stop-at-csr and --signed-ca-cert arguments, make sure that the remaining command options and arguments are the same.

    • setup_custom_certdir

      Initializes the certificate manager using a custom certificate directory. Use this command if you are using existing certificates signed by a trusted public CA or your own internal CA.

      • --configure-services

        Configures Cloudera Manager Server to enable automatic configuration of TLS for supported components, such as HDFS, YARN, and so on. If you omit this option, auto-TLS will only be configured for Cloudera Manager agent/server communication.

      • --rotate

        Backs up the certmanager root directory (/var/lib/cloudera-scm-server/certmanager by default, or specified by the --location option) if it exists, and creates a new one in its place. If the directory does not exist, it is created. If the directory exists, and you do not use the --rotate argument, the command fails.

Enabling Auto-TLS with an Existing Root CA

In Cloudera Manager 6.3 and higher, for new cluster installations, auto-TLS can create and use an intermediate CA for an existing internal root CA. This creates a chain of trust to your existing internal root CA, and allows internal hosts that trust your root CA to access cluster web resources without browser security warnings.

This feature is not supported for existing deployments. For new installations:

  1. Initialize the certificate manager using the --stop-at-csr argument:
    sudo JAVA_HOME=/usr/java/jdk1.8.0_141 /opt/cloudera/cm-agent/bin/certmanager --location /var/lib/cloudera-scm-server/certmanager setup --configure-services --stop-at-csr

    Replace the JDK version with the correct path for your environment.

    The directory specified by the --location parameter is created automatically, and must not exist before running the command. Check the /var/log/cloudera-scm-agent/certmanager.log log file to confirm that the directories were created.

    The command output displays the path to the generated certificate signing request (CSR). For example:

    Stopping after CSR generation. CSR is located at: /var/lib/cloudera-scm-server/certmanager/CMCA/private/ca_csr.pem
  2. Submit the provided CSR to your internal root CA.
  3. After receiving the signed CA certificate, continue the setup process using the --signed-ca-cert argument:
    sudo JAVA_HOME=/usr/java/jdk1.8.0_141 /opt/cloudera/cm-agent/bin/certmanager --location /var/lib/cloudera-scm-server/certmanager setup --configure-services --signed-ca-cert=/path/to/signed_ca_chain.pem
  4. If Cloudera Manager Server is running, restart it:
    RHEL 7 Compatible, Ubuntu 16 or later, SLES 12
    sudo systemctl restart cloudera-scm-server
    RHEL 6 Compatible
    sudo service cloudera-scm-server restart
  5. Continue with the installation.

Enabling Auto-TLS with Existing Certificates

To use auto-TLS with existing certificates on new installations:

  1. Make sure that you have host certificates for all cluster hosts. For instructions on obtaining certificates from a certificate authority (CA), see Generate TLS Certificates.
  2. Initialize the certificate manager using a custom certificate directory:
    sudo JAVA_HOME=/usr/java/jdk1.8.0_141 /opt/cloudera/cm-agent/bin/certmanager --location /var/lib/cloudera-scm-server/certmanager setup_custom_certdir --configure-services --host-cert /path/to/hostcert.pem --host-key /path/to/hostkey.pem --ca-cert /path/to/cacert.pem --keystore-pw-file /path/to/keystore_pw_file --truststore-pw-file /path/to/truststore_pw_file

    Replace the JDK version with the correct path for your environment. Replace /path/to/hostcert.pem and /path/to/hostkey.pem with the certificate and key, respectively, of the Cloudera Manager Server host.

    The directory specified by the --location parameter is created automatically, and must not exist before running the command. Check the /var/log/cloudera-scm-agent/certmanager.log log file to confirm that the directories were created.

  3. For each cluster host (with unique certificates and keys), run the following command:
    sudo JAVA_HOME=/usr/java/jdk1.8.0_141 /opt/cloudera/cm-agent/bin/certmanager --location /var/lib/cloudera-scm-server/certmanager add_custom_cert --host-cert <host_cert> --host-key <host_key> <hostname>

    Replace the JDK version with the correct path for your environment.

  4. If Cloudera Manager Server is running, restart it:
    RHEL 7 Compatible, Ubuntu 16 or later, SLES 12
    sudo systemctl restart cloudera-scm-server
    RHEL 6 Compatible
    sudo service cloudera-scm-server restart
  5. Continue with the installation.

Enabling Auto-TLS on Existing Deployments

Consider the following before enabling auto-TLS on existing deployments using the Cloudera Manager web UI:

Use the following instructions for enabling auto-TLS on existing deployments using the Cloudera Manager web UI:

  1. Navigate to Administration > Security. Click Enable Auto-TLS to launch the wizard.
  2. In the CMCA Directory field, enter the full path to the directory you want to use for the CMCA certificates and configuration files. For example, /var/lib/cloudera-scm-server/certmanager.
  3. If you want to distribute a list of trusted CA certificates, in the Trusted CA Certificates Location field, enter the full path to a file containing a concatenated list of CA certificates in PEM format. These certificates are imported into the Auto-TLS truststore and distributed to all hosts. This file must be readable by the cloudera-scm user.
  4. Select whether you want to enable auto-TLS for all existing and future clusters, or future clusters only. Enabling auto-TLS enables TLS for all hosts managed by Cloudera Manager, whether they are assigned to a cluster or not. Additionally, TLS will be configured and enabled for all future hosts added to Cloudera Manager.
  5. Enter SSH credentials for the root user, and then click Next.
  6. The Remaining Steps page provides instructions for restarting Cloudera Manager Server from the command line. Click Finish.
  7. Restart Cloudera Manager Server from the command line:
    RHEL 7 Compatible, Ubuntu 16 or later, SLES 12
    sudo systemctl restart cloudera-scm-server
    RHEL 6 Compatible
    sudo service cloudera-scm-server restart
  8. Refresh the Cloudera Manager web UI in your browser. You are redirected to https://<server_host>:7183, and a security warning is displayed. You might need to indicate that you trust the certificate, or click to proceed to the Cloudera Manager Server host.
  9. Log in to Cloudera Manager. All services report bad health.
  10. Restart the Cloudera Management Service (Cloudera Management Service > Actions > Restart). After the restart completes, you might need to wait a minute or two for the health status to update. Several CDH services remain in bad health.
  11. On the Cloudera Manager home page, click any of the orange stale configuration icons to launch the Stale Configuration wizard.
  12. Click Restart Stale Services.
  13. Make sure that Re-deploy client configuration is checked, and then click Restart Now. After the restart completes, click Finish.

Rotating Auto-TLS Certificate Authority and Host Certificates

Your cluster's security requirements may require that you rotate the auto-TLS CA and certificates.

  1. Navigate to Administration > Security. Click the Rotate Auto-TLS Certificates button to launch the wizard.
  2. Follow the same workflow as in Enabling Auto-TLS on Existing Deployments.

Auto-TLS Agent File Locations

The certificates, keystores, and password files generated by auto-TLS are stored in /var/lib/cloudera-scm-agent/agent-cert on each Cloudera Manager Agent. The filenames are as follows:

Auto-TLS Agent Files
Filename Description
cm-auto-global_cacerts.pem CA certificate and other trusted certificates in PEM format
cm-auto-global_truststore.jks CA certificate and other trusted certificates in JKS format
cm-auto-in_cluster_ca_cert.pem CA certificate in PEM format
cm-auto-in_cluster_truststore.jks CA certificate in JKS format
cm-auto-host_key_cert_chain.pem Agent host certificate and private key in PEM format
cm-auto-host_cert_chain.pem Agent host certificate in PEM format
cm-auto-host_key.pem Agent host private key in PEM format
cm-auto-host_keystore.jks Agent host private key in JKS format
cm-auto-host_key.pw Agent host private key password file

Enabling Auto-TLS on a Cluster

If you enabled auto-TLS, but configured it to enable TLS on future clusters only, existing clusters do not have TLS enabled.

To enable auto-TLS on a cluster, click the drop-down menu next to the cluster name, and then click Configure the Cluster for Auto-TLS and follow the prompts.