Enabling TLS with Cloudera Director

...
cloudera-manager {

    tlsEnabled: true

...
}
...

...
  "tlsEnabled": true,
...

When auto-TLS is enabled for a Cloudera Manager deployment, every CDH cluster managed by that Cloudera Manager installation is also automatically configured for TLS. If it is necessary for a CDH cluster to not be configured for TLS while Cloudera Manager is, then follow the instructions for Manual TLS to configure Cloudera Manager instead.

Cloudera Director automatically retrieves the root certificate of the bespoke CA used in auto-TLS from the Cloudera Manager instance, and establishes trust in it. This creates an opportunity for a man-in-the-middle attack on the SSH connection Cloudera Director uses for the retrieval. For protection against such an attack, consider configuring SSH host key retrieval and verification.

When a browser or other web client accesses a Cloudera Manager instance set up under auto-TLS, the browser or web client should emit a warning that the server certificate for Cloudera Manager is not trusted. This is because the certificate is signed by the bespoke CA resident on the Cloudera Manager instance, and that CA is not trusted by any of the well-known default CAs installed in the client. It is possible to override or ignore these warnings, but a more secure option is to import the root certificate of the CA into the client.

There are a few ways to retrieve the root certificate of the bespoke CA, so that it can be imported into the trusted certificate store of a client. One easy way is to use the Cloudera Director REST API, or an API client library, to get the deployment corresponding to the Cloudera Manager instance. The result data includes the root certificate as the trustedCertificate field. The value of the field is a single line containing the entire certificate contents; to use the value as a proper encoded certificate, replace each continuous sequence of \r and \n strings with a single newline.

{
  "name": "dep_name",
  "hostname": "203.0.113.101",
  "port": 7183,
  ...
  "tlsEnabled": true,
  "trustedCertificate": "-----BEGIN CERTIFICATE-----\r\nMII...\r\n-----END CERTIFICATE-----\r\n",
  ...
}

Once the root certificate is imported for a client, it should be trusted, and no connection warnings should appear.

The trusted_ca_certs TLS configuration property is useful when cluster services need to connect over TLS to external endpoints, and the server certificates for those endpoints are not ultimately signed by widely acknowledged CAs. The self-signed server or CA certificates for those endpoints, in PEM format, should be included in the file indicated by the property. Since the property value is a path to a file containing the certificates, and not the certificates themselves, measures must be taken to ensure that the file is in place at the indicated path on the Cloudera Manager instance; for example, the file could be included in the instance's base image, or the file could be written through a bootstrap script.

The host_expiration and ca_expiration TLS configuration properties set the expiration dates for server certificates issued for cluster services and for the Cloudera Manager server, respectively. They default to a date about one year in the future from the time when the certificates are issued. Beyond the expiration dates, the certificates are not accepted by TLS clients. Auto-TLS does not support the renewal of automatically generated certificates, so for long-lived clusters, specify expiration dates further in the future.

If the keytool TLS configuration property is passed to Cloudera Director, then its value is used as is. Otherwise, Cloudera Director attempts to locate keytool on the running Cloudera Manager instance. If Cloudera Director succeeds in locating keytool, it passes the path to keytool to Cloudera Manager; otherwise, it passes a default of keytool, which works if keytool is on the default PATH.

...
cloudera-manager {

    tlsEnabled: true
    tlsConfigurationProperties {
        ca_key_args: 4096
        host_key_args: 4096
    }

...
}
...

...
  "tlsEnabled": true,
  "tlsConfigurationProperties": {
    "ca_key_args": "4096",
    "host_key_args": "4096"
  },
...

Cloudera Director does not attempt to validate TLS configuration properties, so any unknown or invalid properties are passed through as-is to Cloudera Manager. Check Cloudera Director and Cloudera Manager logs if the configuration properties appear to cause problems or have no effect.

Because auto-TLS normally uses the FQDN of an instance as the value of the CN attribute in that instance's certificate's subject DN, it is possible for certificate generation to fail if that FQDN is too long. When the FQDN exceeds 64 characters, a certificate with the FQDN as the value for the CN attribute cannot be generated.

This is of particular importance when using Google Compute Engine, which includes an instance's Cloudera Director virtual instance ID, a 36-character UUID, in the instance's hostname. Google Compute Engine also uses the project name as part of the domain name, consuming more characters from the allowed total of 64.

provider {
  instanceNamePrefix: abc
  ...
}

In order to be compatible with Cloudera Director, the server certificate for the Cloudera Manager instance must include its private IP address as a subject alternative name (SAN). If the instance changes its private IP address, perhaps by being stopped and started again, then Cloudera Director will not be able to communicate with Cloudera Manager over TLS due to the mismatch.

There is currently no mechanism in auto-TLS to accommodate a change in the private IP address of the Cloudera Manager instance. Therefore, it is essential that the private IP address for the Cloudera Manager instance be assigned until the instance is terminated, and not released when it is stopped.

This is the normal behavior for instances under AWS and Google Cloud Platform. However, the default behavior for Microsoft Azure is to use dynamic private IP address allocation, which can cause the private IP address to change when the VM is stopped and then started again. In constrast, a static private IP address allocation remains with a VM until it is terminated. To change the private IP address allocation for an Azure VM from dynamic to static, use the Azure portal or Powershell. For more information about configuring Cloudera Director on Azure to use static IP addresses, see the first note in Notes on Pausing a Cluster on the page Pausing Cloudera Director Instances.

Server certificates signed by the bespoke CA are not trusted by web browsers or other clients, leading them to either produce warnings or refuse to complete TLS handshakes. There are several strategies for coping with the certificates.

First, clients can be configured to trust the root certificate of the CA by importing that certificate into the client's trusted certificate store. Once this is done, the client trusts any server certificates signed by the CA, just as it would for those signed by widely recognized CAs. However, this process must be performed for each new CA that is created by auto-TLS, and it usually requires manual steps.

A more user-friendly solution is to place a reverse proxy in front of Cloudera Manager and the cluster services. A reverse proxy can be configured to listen over TLS, using a server certificate signed by a recognized CA so that clients need not import any new trusted certificates. The server name for the proxy may be in your organization's domain and managed by your organization's CA, whether internal or third-party. In response to requests, the reverse proxy then can route new requests, again over TLS, to Cloudera Manager and cluster services.

server {

  listen 443 ssl;
  server_name hadoopproxy;

  # public key certificate for nginx, signed by recognized CA
  ssl_certificate /etc/nginx/cert.pem;
  # private key for nginx
  ssl_certificate_key /etc/nginx/key.pem;

  location / {
    proxy_pass https://nn.cloudprovider.internal:50470/;
    # root certificate for bespoke CA on Cloudera Manager instance
    proxy_ssl_trusted_certificate /etc/nginx/cmca.crt;
    proxy_ssl_verify on;
    proxy_ssl_verify_depth 2;
    proxy_ssl_session_reuse on;
  }

}

Cloud provider load balancing services can also work as reverse proxies. For example, an application load balancer created through AWS Elastic Load Balancing (ELB) may be configured as a reverse proxy, listening for requests over TLS using a server certificate signed by a recognized CA and forwarding them over TLS to Cloudera Manager or cluster service web UIs. Check the documentation for your cloud provider for the available options.

Start by bootstrapping a deployment and cluster normally, without TLS enabled. Once the deployment and cluster are READY in Cloudera Director, perform the usual manual steps for configuring TLS in Cloudera Manager, with a slight modification explained below. See Level 0: Basic TLS/SSL Configuration for more information. The only steps that must be performed before updating Cloudera Director are those for Level 0, which reconfigure the Cloudera Manager server to use TLS. For complete Cloudera Manager TLS configuration, after updating Cloudera Director, be sure to continue the steps after Level 0 as usual for securing Cloudera Manager and CDH manually.

$ sudo keytool -genkeypair -alias $(hostname -f)-server -keyalg RSA \
 -keystore /opt/cloudera/security/pki/$(hostname -f)-server.jks \
 -keysize 4096 -dname "CN=$(hostname -f),O=cloudera.com,ST=CA,C=US" \
 -ext "san=ip:203.0.113.101,dns:$(hostname -f)" \
 -storepass cloudera -keypass cloudera

$ sudo keytool -exportcert -rfc \
 -keystore /opt/cloudera/security/pki/$(hostname -f)-server.jks \
 -alias $(hostname -f)-server \
 -file /opt/cloudera/security/pki/$(hostname -f)-server.crt \
 -storepass cloudera -keypass cloudera

A self-signed certificate is untrusted by browser and other web clients, so there is often no reason to use one instead of taking advantage of auto-TLS.

Retrieve the deployment template for Cloudera Manager from Cloudera Director's REST API. You must use version 10 or higher of the Cloudera Director API. API access is available through the API console at http://director_host:7189/api-console/, or through a Cloudera Director API client library, or directly using your preferred REST client framework.

http://director_host:7189/api/v10/environments/env_name/deployments/dep_name/template

"tlsEnabled": true,
"port": 7183,
"trustedCertificate": "-----BEGIN CERTIFICATE-----\nMII...\n-----END CERTIFICATE-----"

Finally, update the deployment by issuing an HTTP PUT to the deployment update endpoint. Include the entire modified deployment template in the request body, not just the new fields.

http://director_host:7189/api/v10/environments/env_name/deployments/dep_name

During the deployment update process, Cloudera Director attempts to connect to Cloudera Manager over TLS, using the port and trusted certificate supplied in the update. The update will fail if Cloudera Director cannot successfully connect.

After successful deployment update, Cloudera Director will communicate with Cloudera Manager exclusively over TLS. Be sure to return to the documented process of configuring TLS for other components of Cloudera Manager, such as its agents, to complete the work of securing Cloudera Manager and CDH clusters.

If it becomes necessary to disable TLS for Cloudera Manager, it is possible to update Cloudera Director so that it communicates over unencrypted HTTP to Cloudera Manager. Performing such an update is only necessary if TLS is disabled for the Cloudera Manager API and web interface; if TLS is only disabled, say, for Cloudera Manager agent communication, then Cloudera Director does not need to be updated. In other words, Cloudera Director should be updated only if Level 0 is unconfigured for Cloudera Manager.

Also, if TLS is disabled only temporarily for Cloudera Manager, and then re-enabled soon afterward, then Cloudera Director will re-establish communication on its own. While TLS is disabled for Cloudera Manager, Cloudera Director will be unable to communicate with it. Often this is not a problem. However, health reports, refresh processes, and usage-based billing will not function while Cloudera Director is out of contact.

Disabling TLS for Cloudera Manager should not be necessary when auto-TLS is in use. These instructions assume that TLS was configured manually for Cloudera Manager.

Begin by retrieving the deployment template for Cloudera Manager from Cloudera Director's REST API. You must use version 10 or higher of the Cloudera Director API. API access is available through the API console at http://director_host:7189/api-console/, or through a Cloudera Director API client library, or directly using your preferred REST client framework.

http://director_host:7189/api/v10/environments/env_name/deployments/dep_name/template

"tlsEnabled": false,
 "port":7180

Finally, update the deployment by issuing an HTTP PUT to the deployment update endpoint. Include the entire modified deployment template in the request body, not just the updated fields.

http://director_host:7189/api/v10/environments/env_name/deployments/dep_name

During the deployment update process, Cloudera Director attempts to connect to Cloudera Manager over an unencrypted connection, using the port supplied in the update. The update will fail if Cloudera Director cannot successfully connect.

After successful deployment update, Cloudera Director will communicate with Cloudera Manager without TLS. If or when TLS is enabled once again for Cloudera Manager, repeat the procedure above for updating Cloudera Director to communicate over TLS.

Additional properties are supported. Consult the Spring Boot documentation on configuring SSL for details.

After setting TLS configuration properties, restart Cloudera Director for it to begin requiring TLS. Director continues to listen on the same port as before (default 7189) unless configured for a different port.

Once Cloudera Director is configured for TLS, access its web interface using https URLs. If the server certificate is not signed by a certificate authority recognized by the client (for example, the web browser), then take the appropriate steps to either add an exception for the server or add the server certificate, or one from a responsible certificate authority, to the client's truststore.

$ keytool -genkeypair -alias director -keyalg RSA \
  -keystore director.jks \
  -keysize 4096 -dname "CN=$(hostname -f),O=cloudera.com,ST=CA,C=US" \
  -storepass cloudera -keypass cloudera

Use a valid hostname for the CN (common name) component of the subject DN (distinguished name) for the public key certificate. Doing so allows clients to perform hostname verification.

server.ssl.key-store: /path/to/director.jks
server.ssl.key-store-password: cloudera

$ keytool -exportcert -rfc \
  -keystore director.jks -alias director \
  -file director.crt -storepass cloudera -keypass cloudera

Avoid using a self-signed certificate for Cloudera Director in a production environment. Instead, use a certificate signed by a recognized certificate authority.

You may use a MySQL server launched from AWS RDS to host Cloudera Director's database. RDS automatically configures each MySQL database instance to support TLS connections, generating an appropriate server certificate for the instance during launch time. The server certificate is signed by a CA maintained by RDS itself.

You must import a certificate from RDS into the Java truststore for Cloudera Director to be able to communicate with the RDS instance over TLS. RDS maintains a root CA that governs most of its regions. RDS also maintains intermediate CAs for each region. Either the root CA certificate or the intermediate CA certificate for the region where the MySQL server resides should work to establish trust from Cloudera Director to the RDS instance over TLS.

Download the RDS root CA certificate from the link available at SSL Support for MySQL DB Instances in the AWS documentation. Download the RDS intermediate CA certificate for the region where the MySQL server resides from the appropriate link available at Intermediate Certificates in the AWS documentation.

If using the root CA certificate results in the error "Path does not chain with any of the trust anchors," then switch to using the intermediate CA certificate.