Renew host certificates on Data Lake and Data Hub clusters

Steps to rotate host certificates on Data Lake and Data Hub clusters. Host certificates are valid for one year; to keep the clusters running, you must renew the certificates before they expire.

During cluster provisioning, Cloudera Manager creates an intermediate certificate (CMCA) signed by FreeIPA CA. The CMCA is used to create certificates for every host with Auto-TLS. Host certificates have a default expiration date of one year. After the certificates expire, the cluster is not functional, so you must renew them before expiration.

You must have Full Administration user role.

Collect the username and password of a Full Administration Cloudera Manager user:
1. From the Management Console, navigate to Environments > [Select a cluster] > Summary.
2. Under the FreeIPA section, copy the FreeIPA IP address.
3. SSH into the FreeIPA instance: ssh -i [env-ssh-key] cloudbreak@[FreeIPA IP address].
4. Switch to root user: sudo su.
5. Open the file /srv/pillar/freeipa/init.sls and copy the admin_user and password fields under the freeipa object.

Rotate the host certificates on the Data Lake cluster:

From the Management Console, navigate to Environments > [Select a cluster] > Data Lake > Hardware.
Under Master, copy the IP address of the CM server.

Copy the following script to the Data Lake CM Server instance:

scp -i [env-ssh-key] [script location on local computer] cloudbreak@[Data Lake CM Server IP address]:/home/cloudbreak/rotate_hostcerts.sh

#!/bin/bash

[[ "$TRACE" ]] && set -x

: ${USER_NAME?USER_NAME is required}
: ${PASSWORD?PASSWORD is required}

TEMP_SSHKEY_SUBDIR=generate_hostcerts_sshkey
TEMP_SSHKEY_DIRECTORY=/srv/salt/${TEMP_SSHKEY_SUBDIR}

setup_root_ssh_access_on_hosts() {
  mkdir -p ${TEMP_SSHKEY_DIRECTORY}

  if [[ ! -f ${TEMP_SSHKEY_DIRECTORY}/id_rsa ]]; then
    ssh-keygen -t rsa -f ${TEMP_SSHKEY_DIRECTORY}/id_rsa -q -P ""
  fi
  ssh-keygen -y -f ${TEMP_SSHKEY_DIRECTORY}/id_rsa > ${TEMP_SSHKEY_DIRECTORY}/id_rsa.pub

  salt '*' ssh.set_auth_key_from_file root salt://${TEMP_SSHKEY_SUBDIR}/id_rsa.pub
}

generate_host_certs() {
  local hostIp=$(hostname -i)
  local privateKey=$(sed '$!s/$/\\n/' ${TEMP_SSHKEY_DIRECTORY}/id_rsa | tr -d '\n')
  local hostIds=$(curl -k -u "${USER_NAME}:${PASSWORD}" -X GET --header 'Accept: application/json' "https://${hostIp}/clouderamanager/api/v41/hosts" | jq -r '.items[] | select(.clusterRef) | .hostId')

  for hostId in ${hostIds}; do
    curl -k -u "${USER_NAME}:${PASSWORD}" -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d "{\"userName\": \"root\",\"privateKey\": \"${privateKey}\"}" "https://${hostIp}/clouderamanager/api/v41/hosts/${hostId}/commands/generateHostCerts"
  done
}

destroy_root_ssh_access_on_hosts() {
  salt '*' ssh.rm_auth_key_from_file root salt://${TEMP_SSHKEY_SUBDIR}/id_rsa.pub
  rm -rf ${TEMP_SSHKEY_DIRECTORY}
}

list_host_certs_validity() {
  salt '*' cmd.run 'openssl x509 -text -noout -in /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem | grep -A 2 Validity'
}

main() {
  . activate_salt_env
  setup_root_ssh_access_on_hosts
  generate_host_certs
  destroy_root_ssh_access_on_hosts
  list_host_certs_validity
}

[[ "$0" == "$BASH_SOURCE" ]] && main "$@"

SSH into the Data Lake CM Server instance: ssh -i [env-ssh-key] cloudbreak@[Data Lake CM Server IP address].
Set the script to executable: chmod +x /home/cloudbreak/rotate_hostcerts.sh.
Switch to root user: sudo su.

Execute the rotation script with the admin user and password you collected earlier:

USER_NAME=[collected admin user] PASSWORD=[collected admin password] <TRACE=1> /home/cloudbreak/rotate_hostcerts.sh > /home/cloudbreak/hostcerts_rotation.log

Rotate the host certificates on the Data Hub clusters:
1. From the Management Console, navigate to Data Hub Clusters > [Select a cluster] > Hardware.
2. Under Master, copy the IP address of the CM server.
3. Copy the following script to the Data Hub CM Server instance:
```
scp -i [env-ssh-key] [script location on local computer] cloudbreak@[Data Hub CM Server IP address]:/home/cloudbreak/rotate_hostcerts.sh
```
4. SSH into the Data Hub CM Server instance: ssh -i [env-ssh-key] cloudbreak@[Data Hub CM Server IP address].
5. Set the script to executable: chmod +x /home/cloudbreak/rotate_hostcerts.sh.
6. Switch to root user: sudo su.
7. Execute the rotation script with the admin user and password you collected earlier:
```
USER_NAME=[collected admin user] PASSWORD=[collected admin password] <TRACE=1> /home/cloudbreak/rotate_hostcerts.sh > /home/cloudbreak/hostcerts_rotation.log
```
Restart the CM server: systemctl restart cloudera-scm-server.
Restart services via the CM UI:
1. From the Environments > [Select a cluster] > Data Lake > Services, follow the link to the CM-UI.
  The Cloudera Manager UI opens.
2. From the Status tab, click the drop-down menu and select Restart.
3. From Clusters > Cloudera Management Service, open the Actions menu and select Restart.
Verify the renewal was successful by checking the certification expiration dates:
- Via the Cloudbreak UI, you can find the cluster creation date:
  For Data Lake clusters: Management Console > Environments in the Time Created column.
  For Data Hub clusters: Management Console > Data Hubs in the Created column.
- Via CLI: SSH into the cluster hosts and execute the following command as root: openssl x509 -text -noout -in /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem | grep -A 2 Validity.
  The command output should contain the expiration date:
```
[root@fschneider-aws-master0 cloudbreak]# openssl x509 -text -noout -in /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem | grep -A 2 Validity
        Validity
            Not Before: Sep 15 14:11:12 2020 GMT
            Not After : Sep 15 23:59:59 2021 GMT
```