CDP Private Cloud Data Services 1.5.4-CHF3

New features introduced in this cumulative hotfix release of CDP Private Cloud Data Services 1.5.4-CHF3.

New known issues in the 1.5.4 cumulative hotfix CHF3 release of CDP Private Cloud Data Services.

DOCS-22277 - CDP Private Cloud ECS longhorn upgrade failure

The helm-install-longhorn pod enters a crash loop state during ECS upgrade.

Provide the Longhorn diagnostic bundle to facilitate issue identification.

Resuming Longhorn Upgrade: After resolving the underlying longhorn upgrade failure issues, follow these steps to resume the upgrade:

# Get the history of longhorn helm chart so that we can identify the chart for which installation is failing. # helm history longhorn -n longhorn-system
REVISION    UPDATED                     STATUS        CHART             APP VERSION    DESCRIPTION
1           Thu Sep 26 21:31:05 2024    superseded    longhorn-1.5.4    v1.5.4         Install complete
2           Fri Sep 27 05:17:44 2024    failed        longhorn-1.6.2    v1.6.2         Upgrade "longhorn" failed: post-upgrade hooks failed: 1 error occurr...# Get the log of the failing helm-install-longhorn job in the longhorn namespace

The job log should indicate that the due to failure policy being "abort", it is waiting for manaul intervention:
"Release status is 'failed' and failure policy is 'abort', not 'reinstall'; waiting for operator intervention"# We want to rollback              

# Find all jobs in longhorn-system and delete those. These jobs will be re-triggered as part of the manual patch.
# kubectl get jobs -n longhorn-system
NAME                    COMPLETIONS   DURATION   AGE
helm-install-longhorn   0/1           9h         9h
longhorn-post-upgrade   1/1           11m        10h
              
# Delete all the longhorn jobs if any

# kubectl delete job helm-install-longhorn longhorn-post-upgrade -n longhorn-system# Rollback longhorn to the version prior to the upgrade
# In this case, revision 1 marks the step of a successful longhorn 1.5.4 install
# helm rollback longhorn -n longhorn-system <revision number># Resume longhorn upgrade from Cloudera Manager UI
Running commands > All recent commands > find the failed upgrade command and click on resume

OPSX-5573/OPSAPS-69892 - Intermittent kube-proxy failures - 1.5.4 CHF3

After rebooting/restarting an ECS agent node, the kube-proxy Linux process may not start due to a race condition in the kubelet. When this happens, ECS cluster networking and other services – such as Vault, DNS, authentication, Longhorn storage, etc. – are affected. At the Kubernetes pod level, errors such as "connection refused", "connection timed out" and "i/o timeout" may be observed. If you suspect possible networking issues in your ECS cluster, checking kube-proxy is a good first step.

To fix this issue, perform the following steps on all of the affected nodes:

1. To identify which agent needs to be restarted, check the status of each kube-proxy pod to make sure it is in the "ready" state by running the following command on each host in the cluster.

   kubectl describe pod [***POD-NAME***] -n kube-system

   Here, [***POD-NAME***] should have a format such as: kube-proxy-<hostname>.

   In the Conditions section of the describe pod output, confirm that the "ready" condition is "True".

   Conditions:
     Type              Status
     Initialized       True 
     Ready             True 
     ContainersReady   True 
     PodScheduled      True 

   Another option is to run the following command:

   kubectl get pods -n kube-system -l component=kube-proxy -o go-template='{{range .items}}
   {{.metadata.name}}{{"\n"}}{{"  "}}{{range .status.conditions}}{{ if eq .type "Ready" }}
   Ready:{{.status}}{{"\n\n"}}{{end}}{{end}}{{end}}'

   The sample output displays the status of all of the kube-proxy pods in the cluster:

   
   kube-proxy-host-1.cloudera.com
     Ready:True
   
   kube-proxy-host-2.cloudera.com
     Ready:True
   
   kube-proxy-host-3.cloudera.com
     Ready:True

2. If the "ready" state is False, kube-proxy is not functioning properly, regardless of whether the kube-proxy process is running on that host or not. On each of the affected nodes, run the following command to delete the kube-proxy pod manifest:

    rm /var/lib/rancher/rke2/agent/pod-manifests/kube-proxy.yaml

3. Start the agent role.

   After the agent role is started, you may not immediately see the kube-proxy process running, but a new kube-proxy process should start shortly. Check the pod status to make sure it is ready. After all of the problem agents have been restarted, the cluster may complain that the vault is sealed – if so, unseal it. At this point, the Control Plane should be functioning properly.

If current version is 1.5.2 perform above steps. If it is 1.5.3 or above, perform step 1 from the above procedure to identify the problematic nodes. Perform stop and start of ECS roles on the hosts where the problem exists.

Additional details about this issue are available here: https://www.suse.com/support/kb/doc/?id=000021284

COMPX-18031 - [ECS] [154CHF1-CHF3] Any new pods are stuck in pending state post ecs upgrade

When pods are stuck in pending state post ECS upgrade, you will not be able to schedule new workload (warehouse, virtual cluster etc) . Pending pods are normal if resources or quotas are exhausted. After high cluster loads, pods will be left in a pending state even if enough resources are available. The pods are not evaluated as part of normal scheduling.

The issue will be fixed after restarting the YuniKorn scheduler pod in yunikorn namespace. YuniKorn scheduler will trigger a re-evaluation of all pods and schedule the pending pods.

OBS-4176 - Prometheus reports duplicate metric alert for Kubernetes state metrics

After installing CDP Private Cloud Data Services 1.5.4, the EnvPrometheusDuplicateTimestamps warning message appears on the Control Plane Monitoring dashboard.

Perform the following steps:

1. On the Management Console home page, select Administration > Alerts.
2. On the Alerts page, search for the EnvPrometheusDuplicateTimestamps rule alert and from the drop-down menu select Disable Alert Rule to disable the alert.

The URLs for CDP Private Cloud Data Services 1.5.4-CHF3 are listed in the following table:

https://username:password@archive.cloudera.com/p/cdp-pvc-ds/1.5.4-h4/

https://username:password@archive.cloudera.com/p/cdp-pvc-ds/1.5.4-h4/manifest.json

https://username:password@archive.cloudera.com/p/cdp-pvc-ds/1.5.4-h4/parcels/

Review the Common vulnerabilities and Exposures (CVEs) that were fixed in 1.5.4 CHF3 release of CDP Private Cloud Data Services.