ECS Day Two Operations GuidePDF version

Adjusting the expiration time of Cloudera Embedded Container Service cluster certificates

The RKE Kubernetes, Vault, and Cloudera Embedded Container Service webhook certificate expiration times are set to one year by default. To avoid certificate expiration errors, you may want to extend the expiration times.

  • These steps describe how to adjust the expiration time of internal cluster certificates in an existing Cloudera Embedded Container Service cluster.
  • For a new cluster, if the nodes have been added to Cloudera Manager before creating the Cloudera Embedded Container Service cluster, you can edit the cluster_signing_duration configuration property in Cloudera Manager before creating the Cloudera Embedded Container Service cluster.
  1. In Cloudera Manager, select the ECS cluster, then click ECS.
  2. Click the Configuration tab, then use the Search box to locate the cluster_signing_duration configuration property.
  3. The cluster_signing_duration configuration property sets the expiration time for the RKE Kubernetes, Vault, and ECS webhook certificates, and is set to 1 year (365 days) by default. In the example below, the certificate expiration has been reset to 5 years (1825 days):
  4. Click Save Changes.
  5. On the ECS Cluster landing page, click Actions > Refresh Cluster.
  6. For versions:
    1. Upto 1.5.4 SP1after the Refresh is complete, click Actions > Rolling Restart.
    2. From 1.5.4 SP2 and later, after the Refresh is complete, click Actions > Restart.
  7. After the restart is complete, the certificate expiration time is reset to the new value. You can also use the CLI to verify the new certificate expiration setting.
  1. In Cloudera Manager, select the ECS cluster, then click ECS.
  2. Click the Configuration tab, then use the Search box to locate the cluster_signing_duration configuration property.
  3. The the cluster_signing_duration configuration property sets the expiration time for the RKE Kubernetes, Vault, and ECS webhook certificates, and is set to 1 year (365 days) by default. In the example below, the certificate expiration has been reset to 5 years (1825 days):
  4. Click Save Changes.
  5. Contact Cloudera Support to obtain the rotate-vault-cert.sh file. Once obtained, copy it to the ECS master host. Set JAVA_HOME if needed.

    Perform the manual copying and execution of the rotate-vault-cert.sh script for ECS versions upto 1.5.4 CHF3. For versions, 1.5.4 SP1 and later, the scripts are already present on the ECS hosts under the parcel directory.

  6. Run the following command:

    ./rotate-vault-cert.sh APP_DOMAIN

  7. Unseal Vault.
  8. Restart all of the pods in the CDP namespace by executing the following command:
    for a in `kubectl get pod --no-headers=true -n cdp | grep -v -E 'Running|Complete' | cut -d' ' -f1`; do kubectl delete pod $a -n cdp --force; done
  9. If you are using a default self-signed ingress controller certificate, update the ingress controller certificate (follow the steps in the script output).
  10. You can use the CLI to verify the new certificate expiration setting: 
    root       49076   48970  2 16:49 ?        00:00:10 kube-controller-manager 
--flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true 
--allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig 
--authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 
--cluster-cidr=10.42.0.0/16 --cluster-signing-duration=43800h
<snip!>
  1. In Cloudera Manager, select the ECS cluster, then click ECS.
  2. Click the Configuration tab, then use the Search box to locate the cluster_signing_duration configuration property.
  3. The the cluster_signing_duration configuration property sets the expiration time for the RKE Kubernetes, Vault, and ECS webhook certificates, and is set to 1 year (365 days) by default. In the example below, the certificate expiration has been reset to 5 years (1825 days):
  4. Click Save Changes.
  5. Copy the rotate-webhook-cert.sh file to the ECS master host.
  6. Run the following command:

    ./rotate-webhook-cert.sh APP_DOMAIN

  7. Check for any pods in the Pending state whose status shows that they cannot tolerate the node-role.kubernetes.io/control-plane toleration. Restart those pods.
  8. You can use the CLI to verify the new certificate expiration setting: 
    root       49076   48970  2 16:49 ?        00:00:10 kube-controller-manager 
--flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true 
--allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig 
--authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 
--cluster-cidr=10.42.0.0/16 --cluster-signing-duration=43800h
<snip!>