Adjusting the expiration time of ECS cluster certificates
The RKE Kubernetes, Vault, and ECS webhook certificate expiration times are set to one year by default. To avoid certificate expiration errors, you may want to extend the expiration times.
- These steps describe how to adjust the expiration time of internal cluster certificates in an existing ECS cluster.
- For a new cluster, if the nodes have been added to Cloudera Manager before creating
the ECS cluster, you can edit the
cluster_signing_duration
configuration property in Cloudera Manager before creating the ECS cluster.
Adjusting the expiration time of the RKE Kubernetes cluster certificate
- In Cloudera Manager, select the ECS cluster, then click ECS.
- Click the Configuration tab, then use the Search box to locate the
cluster_signing_duration
configuration property. - The the
cluster_signing_duration
configuration property sets the expiration time for the RKE Kubernetes, Vault, and ECS webhook certificates, and is set to 1 year (365 days) by default. In the example below, the certificate expiration has been reset to 5 years (1825 days): - Click Save Changes.
- On the ECS Cluster landing page, click Actions > Refresh Cluster.
- After the Refresh is complete, click Actions > Rolling Restart.
- After the restart is complete, the certificate expiration time is reset to the new
value. You can also use the CLI to verify the new certificate expiration setting:
[root@host-1 ~]# cat /proc/47803/environ CDH_PIG_HOME=/usr/lib/pigLD_LIBRARY_PATH=:/opt/cloudera/cm-agent/libCMF_AGENT_ARGS=CDH_KAFKA_HOME=/usr/lib/kafka CONF_DIR=/var/run/cloudera-scm-agent/process/1546342871-ecs-ECS_SERVERCDH_PARQUET_HOME=/usr/lib/parquet PARCELS_ROOT=/opt/cloudera/parcelsPARCEL_DIRNAMES=ECS-1.5.2-b866-ecs-1.5.2-b866.p0.46395126LANG=en_US.UTF-8 CDH_HADOOP_BIN=/usr/bin/hadoopCDH_KMS_HOME=/usr/lib/hadoop-kmsCGROUP_GROUP_CPU=CMF_PACKAGE_DIR=/opt/cloudera/cm-agent/service ORACLE_HOME=/usr/share/oracle/instantclientMGMT_HOME=/opt/cloudera/cmINVOCATION_ID=04c94a229a2b4684a95f8ec63783c81e JSVC_HOME=/usr/libexec/bigtop-utilsCDH_IMPALA_HOME=/usr/lib/impalaKRB5_CONFIG=/etc/krb5.conf CDH_YARN_HOME=/usr/lib/hadoop-yarnCLOUDERA_POSTGRESQL_JDBC_JAR=/opt/cloudera/cm/lib/postgresql-42.5.1.jar CDH_SOLR_HOME=/usr/lib/solrHIVE_DEFAULT_XML=/etc/hive/conf.dist/hive-default.xml CLOUDERA_ORACLE_CONNECTOR_JAR=/usr/share/java/oracle-connector-java.jarCGROUP_GROUP_BLKIO=system.slice/cloudera-scm-agent.service CGROUP_ROOT_BLKIO=/sys/fs/cgroup/blkioCGROUP_ROOT_CPU=/sys/fs/cgroup/cpu,cpuacctKEYTRUSTEE_KP_HOME=/usr/share/keytrustee-keyprovider CLOUDERA_MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jarCMF_SERVER_ROOT=/opt/cloudera/cm CGROUP_ROOT_CPUACCT=/sys/fs/cgroup/cpu,cpuacctCDH_FLUME_HOME=/usr/lib/flume-ng CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=1825 <snip!>
[root@host-1 ~]# openssl x509 -in /var/lib/rancher/rke2/agent/serving-kubelet.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4005696761303552502 (0x379717fb376e51f6)
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN = rke2-server-ca@1697759349
Validity
Not Before: Oct 19 23:49:09 2023 GMT
Not After : Oct 17 23:49:10 2028 GMT
Subject: CN = host-1.rke-1019.kcloud.cloudera.com
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:92:81:74:b8:fb:aa:6c:c5:9a:40:2c:5f:91:60:
35:16:9a:d5:41:b2:bf:d8:29:f4:ed:68:ed:cd:3d:
87:0e:59:db:27:26:c5:d8:a7:79:c7:23:8f:0b:71:
c2:f5:d4:36:fe:97:a9:b5:62:ee:9d:9b:6d:ed:25:
60:fd:26:3a:08
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Authority Key Identifier:
keyid:26:8F:9F:A1:04:CE:2D:04:3A:03:11:87:9D:DF:5A:B7:5C:06:72:32
X509v3 Subject Alternative Name:
DNS:host-1.rke-1019.kcloud.cloudera.com, DNS:localhost, IP Address:127.0.0.1, IP Address:10.17.130.15
Signature Algorithm: ecdsa-with-SHA256
30:46:02:21:00:fc:5c:89:ab:99:a6:79:33:a9:28:da:a8:47:
52:cf:1f:43:13:8c:06:2e:23:67:4c:b4:b0:d6:e3:f9:b6:ad:
50:02:21:00:c7:64:aa:86:97:5a:f3:12:7e:3f:a2:f1:ab:93:
17:6c:3a:37:34:01:ef:ba:7f:08:85:70:2c:c9:40:e0:30:f5
Adjusting the expiration time of the Vault certificate
- In Cloudera Manager, select the ECS cluster, then click ECS.
- Click the Configuration tab, then use the Search box to locate the
cluster_signing_duration
configuration property. - The the
cluster_signing_duration
configuration property sets the expiration time for the RKE Kubernetes, Vault, and ECS webhook certificates, and is set to 1 year (365 days) by default. In the example below, the certificate expiration has been reset to 5 years (1825 days): - Click Save Changes.
- Copy the
rotate-vault-cert.sh
file to the ECS master host. SetJAVA_HOME
if needed. - Run the following command:
./rotate-vault-cert.sh APP_DOMAIN
- Unseal Vault.
- Restart all of the pods in the CDP namespace.
- If you are using a default self-signed ingress controller certificate, update the ingress controller certificate (follow the steps in the script output).
- You can use the CLI to verify the new certificate expiration setting:
root 49076 48970 2 16:49 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-duration=43800h <snip!>
[root@host-1 ~]# openssl x509 -in vault.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
db:b7:a7:c3:79:86:4c:54:e8:97:49:bf:99:3d:df:a9
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN = rke2-server-ca@1697759349
Validity
Not Before: Oct 19 23:46:38 2023 GMT
Not After : Oct 17 23:46:38 2028 GMT
Subject: O = system:nodes, CN = "system:node:vault.vault-system.svc;"
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:94:93:2e:9d:5c:01:5a:95:46:b2:9d:aa:23:c4:
4e:0f:92:07:7e:0e:3a:21:7d:ef:95:e8:09:d3:88:
38:ac:e9:9f:c2:36:37:04:56:43:87:3a:6f:34:08:
09:8f:3f:df:31:79:d6:12:db:78:f6:1c:9b:0e:c2:
d0:f5:25:50:86:37:d5:ff:f7:a0:82:6f:55:d1:ff:
03:54:f8:ce:8b:02:87:2d:af:3f:71:f8:c4:a9:f0:
24:50:7b:07:70:3d:7a:be:9d:41:f0:15:2f:56:c3:
d3:0d:1a:e1:87:8e:69:89:ff:bf:1b:f2:84:87:6c:
5e:f9:13:8b:2c:5c:de:64:9e:ae:de:6a:f0:7c:ae:
d9:01:41:aa:39:00:b3:2d:4f:5c:db:fb:2b:80:31:
88:b5:40:24:e1:06:08:c4:ad:82:70:a1:9e:4c:3e:
00:0d:61:d9:1a:5c:c7:11:a7:79:68:66:34:b2:c2:
e9:63:a8:5d:d1:13:be:e6:f1:8f:03:87:3d:be:eb:
b7:ce:a5:eb:56:81:37:5b:9d:ce:82:34:15:99:16:
4c:65:20:d9:df:e6:63:56:c2:49:79:e8:66:ce:c1:
01:9d:87:a2:ba:02:c0:7c:2b:e5:37:30:c5:23:bd:
87:a1:c8:2b:a9:49:be:67:31:22:8d:a4:68:f9:bd:
be:23
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
keyid:26:8F:9F:A1:04:CE:2D:04:3A:03:11:87:9D:DF:5A:B7:5C:06:72:32
X509v3 Subject Alternative Name:
DNS:vault, DNS:vault.vault-system, DNS:vault.vault-system.svc, DNS:vault.vault-system.svc.cluster.local, DNS:vault.localhost.localdomain, DNS:*.apps.host-1.rke-1019.kcloud.cloudera.com, IP Address:127.0.0.1
Signature Algorithm: ecdsa-with-SHA256
30:46:02:21:00:d9:5e:38:fc:31:9b:5a:eb:fc:7d:c2:8f:b3:
54:5e:28:f0:8f:00:eb:36:65:9f:d3:70:ae:a2:79:77:ee:b5:
f7:02:21:00:f4:e8:6f:c9:bd:bb:92:9d:63:81:69:55:67:8b:
8a:f3:a4:5d:c1:67:66:b0:40:ff:22:a6:c3:6f:4f:8e:b2:8e
Adjusting the expiration time of the ECS webhook certificate
- In Cloudera Manager, select the ECS cluster, then click ECS.
- Click the Configuration tab, then use the Search box to locate the
cluster_signing_duration
configuration property. - The the
cluster_signing_duration
configuration property sets the expiration time for the RKE Kubernetes, Vault, and ECS webhook certificates, and is set to 1 year (365 days) by default. In the example below, the certificate expiration has been reset to 5 years (1825 days): - Click Save Changes.
- Copy the
rotate-webhook-cert.sh
file to the ECS master host. - Run the following command:
./rotate-webhook-cert.sh APP_DOMAIN
- Check for any pods in the Pending state whose status shows that they cannot tolerate
the
node-role.kubernetes.io/control-plane
toleration. Restart those pods. - You can use the CLI to verify the new certificate expiration setting:
root 49076 48970 2 16:49 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-duration=43800h <snip!>
[root@host-1 ~]# openssl x509 -in ecs-tolerations-webhook-cert.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
a5:31:94:f4:84:bb:3b:a2:a4:63:8d:ec:de:b5:37:53
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN = rke2-server-ca@1697759349
Validity
Not Before: Oct 19 23:45:48 2023 GMT
Not After : Oct 17 23:45:48 2028 GMT
Subject: O = system:nodes, CN = "system:node:ecs-tolerations-webhook.ecs-webhooks.svc;"
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:cc:12:e1:54:b8:aa:42:94:aa:11:a5:f7:35:0e:
0c:de:76:5b:d5:c6:c1:34:0b:b8:b7:2b:15:08:1d:
02:44:0f:2e:e1:17:dc:73:6a:e4:6c:df:5b:ac:43:
97:2e:34:73:f7:c9:6f:cf:c2:a8:52:79:b1:89:ea:
51:22:e1:41:b8:6a:ba:fd:22:a2:bf:a2:46:a4:8e:
f5:c6:2d:05:c3:a5:1d:6b:60:da:e8:40:a5:e1:e1:
5a:55:0e:94:2d:91:dd:71:d1:e9:aa:27:5d:e6:fc:
ea:5f:ea:c6:8e:52:71:27:ce:c2:a7:1b:10:ca:db:
db:27:c8:46:6d:14:d1:d0:b3:f5:ab:74:a9:63:8b:
71:83:31:eb:ad:87:1b:3b:8d:ff:ce:d0:7f:d1:1b: