Known issues for the CDP Private Cloud Data Services Management Console
This section lists known issues that you might run into while using the CDP Private Cloud Management Console service.
Known Issues in Management Console 1.5.4
- DOCS-20088/OPSX-4781: Vault pods may take long time to be ready during upgrades from 1.5.2 to 1.5.3
- The 'vault-0' pod takes longer time to attach volume in some upgrade cases than usual. Due to the excess time taken the cluster upgrade may fail. But, usually in 15 minutes the volume can attach automatically and the pod would start running. In that case, the user can resume the upgrade.
- OPSX-5155: OS Upgrade | Pods are not starting after the OS upgrade from RHEL 8.6 to 8.8
- After an OS upgrade and start of the ECS service, pods fail to come up due to stale state.
- OPSX-5055: ECS upgrade failed at Unseal Vault step
-
During an ECS upgrade from 1.5.2 to 1.5.4 release, the vault pod fails to start due to an error caused by the Longhorn volume unable to attach to the host. The error is as below:
Warning FailedAttachVolume 3m16s (x166 over 5h26m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-0ba86385-9064-4ef9-9019-71976b4902a5" : rpc error: code = Internal desc = volume pvc-0ba86385-9064-4ef9-9019-71976b4902a5 failed to attach to node host-1.cloudera.com with attachmentID csi-7659ab0e6655d308d2316536269de47b4e66062539f135bf6012bfc8b41fc345: the volume is currently attached to different node host-2.cloudera.com
- OPSX-4308: Display error in UI if listEnvironments failed
-
On the Environments page, if the listEnvironments API call fails, the error is hidden, and instead no environments are displayed, even though they do exist. This can be due to vault issues or connectivity issues.
- OPSX-4684: Start ECS command shows green(finished) even though start docker server failed on one of the hosts
-
The Docker service starts, but one or more Docker roles fail to start because the corresponding host is unhealthy.
- OPSX-735: Kerberos service should handle Cloudera Manager downtime
-
The Cloudera Manager Server in the base cluster operates to generate Kerberos principals for Private Cloud. If there is downtime, you may observe Kerberos-related errors.
Known Issues in Management Console 1.5.3
- OPSX-4754 [ECS Restart Stability] DaemonSet rollout is stuck post rolling restart - DaemonSet kube-system/rke2-canal has not finished or progressed for at least 15 minutes
-
On RHEL 9.1, an ECS service DaemonSet rollout health alert appears after a rolling restart.
- OPSAPS-69892: kube-proxy failure causing issues with cluster
-
After rebooting/restarting an ECS agent node, the kube-proxy Linux process may not start due to a race condition in the kubelet. When this happens, ECS cluster networking and other services – such as Vault, DNS, authentication, Longhorn storage, etc. – are affected. At the Kubernetes pod level, errors such as "connection refused", "connection timed out" and "i/o timeout" may be observed. If you suspect possible networking issues in your ECS cluster, checking kube-proxy is a good first step.
- OPSX-4766: [ECS Restart]| Host Reboot | start command failed with error - "Timed out waiting for kube-apiserver to be ready"
-
In an ECS cluster with HA enabled, ECS Start fails with an error after stopping the cluster and rebooting the hosts.
Steps to reproduce:
- Stop ECS.
- Reboot hosts.
- Start ECS.
The start command fails with the following error message:
"Timed out waiting for kube-apiserver to be ready"
Known Issues in Management Console 1.5.2
- OPSAPS-68923: CM - After CM upgrade from 7.9.5 to 7.11.3.x ECS cluster showing stale config
- After Cloudera Manager upgrade from 7.9.5 to 7.11.3.x, an ECS 1.5.0 cluster may show a stale config to add ""limit_fds": 1048576"
- OPSX-4594: [ECS Restart Stability] Post rolling restart few volumes are in detached state (vault being one of them)
-
After rolling restart there may be some volumes in detached state.
- OPSAPS-68558: [7.9.5->7.11.3.2] CM upgrade failed with BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState'
-
After upgrading the Cloudera Manager package, the Cloudera Manager Server does not start. An error about "Active Commands" is shown in the Cloudera Manager Server log.
This may happen when the Private Cloud Data Services Control Plane is actively issuing requests to Cloudera Manager while an upgrade is being performed.
- OPSX-4392: Getting the real client IP address in the application
- CML has a feature for adding the audit event for each user action (Monitoring User Events). In Private Cloud, instead of the client IP, we are getting the internal IP, which is logged into the internal DB.
- OPSX-4552: [ECS Restart] One of the docker servers failed to come up after starting the cluster post hosts reboot
- At times the Docker server may fail to come up and return the following error
message:
/var/run/docker.sock: Is a directory
- CDPVC-1137, CDPAM-4388, COMPX-15083, and COMPX-15418: OpenShift Container Platform version upgrade from 4.10 to 4.11 fails due to a Pod Disruption Budget (PDB) issue
- PDB can prevent a node from draining which makes the nodes to
report the
Ready,SchedulingDisabled
state. As a result, the node is not updated to correct the Kubernetes version when you upgrade OCP from 4.10 to 4.11.
- PULSE-944 and PULSE-941 Observability namespace not created after platform upgrade from 151 to 152
- The Cloudera Observability namespace is not created after a
platform upgrade from PvC DS 1.5.1 to PvC DS 1.5.2.
During the creation of the resource pool the Cloudera Observability namespace is provided by the CDP Private Cloud Service. If the provisioning flow is not completed, such as due to a timing difference between the start of the computeAPI pod and the call to the computeAPI pod by the service, the namespace is not created.
- PULSE-921 Observability namespace has no pods
- The Cloudera Observability namespace should have the same number of pods and nodes. When the Cloudera Observability namespace has no pods the prometheus-node-exporter-1.6.0 helm release state becomes invalid and the CDP Private Cloud Service is unable to uninstall and reinstall the namespace. Also, as the Node Exporter is not installed into the Cloudera Observability namespace its metrics are unavailable when querying Prometheus in the control plane, for example the node_cpu_seconds_total metric.
- PULSE-697 Add node-exporter to PvC DS
- When expanding a cluster with new nodes and there is insufficient CPU and memory resources, the Node Exporter will encounter difficulties deploying new pods on the additional nodes.
- PULSE-935 Longhorn volumes are over 90% of the capacity alerts on Prometheus volumes
- Cloudera Manager displays the following alert about your
Prometheus volumes: Concerning: Firing alerts for Longhorn: The actual used
space of Longhorn volume is over 90% of the capacity.
Longhorn stores historical data as snapshots that are calculated with the active data for the volume's actual size. This size is therefore greater than the volume's nominal data value.
- PULSE-937 Private-Key field change in Update Remote Write request does not reflect in enabling the metric flow
- When using the Management Console UI for Remote Storage the Disable option does not deactivate the remote write configuration, even when the action returns a positive result message. Therefore, when disabling a remote storage configuration use the CLI client to disable the remote storage configuration directly from the API.
- PULSE-841 Disabling the remote write configuration logs an error in both cp prometheus and env prometheus
- When a metric replication is set up between the cluster and Cloudera Observability and the connection is disabled or deleted, Prometheus writes an error message that states that it cannot replicate the metrics.
- PULSE-895 Disabling the remote write config in the UI is broken in cdp-pvc
- The Remote Write Enable and Disable options in the Management Console’s User Interface do not work when a Remote Storage configuration is created with a requestSignerAuth type from either the HTTP API or using the CDP-CLI tool.
- PULSE-936 No Alert to prompt the metric flow being affected b/c of wrong private key configuration
- A remote write alert was not triggered when the wrong private key was used in a Remote Storage configuration.
Known Issues in Management Console 1.5.1
- External metadata databases are no longer supported on OCP
-
As of CDP Private Cloud Data Services 1.5.1, external Control Plane metadata databases are no longer supported. New installs require the use of an embedded Control Plane database. Upgrades from CDP Private Cloud Data Services 1.4.1 or 1.5.0 to 1.5.1 are supported, but there is currently no migration path from a previous external Control Plane database to the embedded Control Plane database. Upgrades from 1.4.0 or 1.5.0 with external Control Plane metadata databases also require additional steps, which are described in the CDP Private Cloud Data Services 1.5.1 upgrade topics.
- DOCS-15855: Networking API is deprecated after upgrade to CDP Private Cloud Data Services 1.5.1 (K8s 1.24)
-
When the control plane is upgraded from 1.4.1 to 1.5.1, the Kubernates version changes to 1.24. The Livy pods running in existing Virtual Clusters (VCs) use a deprecated networking API for creating ingress for Spark driver pods. Because the old networking API is deprecated and does not exist in Kubernates 1.24, any new job run will not work for the existing VCs.
- CDPQE-24295: Update docker client on docker.lab.eng.hortonworks machine
-
When you attempt to execute the Docker command to fetch the Cloudera-provided images into your air-gapped environment, you may encounter an issue where Docker pulls an incorrect version of the HAProxy image, especially if you are using an outdated Docker client. This situation arises due to the Cloudera registry containing images with multiple platform versions. Unfortunately, older Docker clients may lack the capability to retrieve the appropriate architecture version, such as amd64.
- OPSX-4266: ECS upgrade from 1.5.0 to 1.5.1 is failing in Cadence schema update job
-
When upgrading from ECS 1.5.0 to 1.5.1, the CONTROL_PLANE_CANARY fails with the following error:
Firing alerts for Control Plane: Job did not complete in time, Job failed to complete.
And the
cdp-release-cdp-cadence-schema-update
job fails.
- OPSX-4076:
- When you delete an environment after the backup event, the restore operation for the backup does not bring up the environment.
- OPSX-4024: CM truststore import into unified truststore should handle duplicate CommonNames
-
If multiple CA certificates with the exact same value for the Common Name field are present in the Cloudera Manager truststore when a Private Cloud Data Services cluster is installed, only one of them may be imported into the Data Services truststore. This may cause certificate errors if an incorrect/old certificate is imported.
- COMOPS-2822: OCP error x509: certificate signed by unknown authority
-
The error
x509: certificate signed by unknown authority
usually means that the Docker daemon that is used by Kubernetes on the managed cluster does not trust the self-signed certificate.
- OPSX-3073 [ECS] First run command failed at setup storage step with error "Timed out waiting for local path storage to come up"
- Pod stuck in pending state on host for a long time. Error in
Role log related to CNI plugin:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 3m5s (x269 over 61m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "70427e9b26fb014750dfe4441fdfae96cb4d73e3256ff5673217602d503e806f": failed to find plugin "calico" in path [/opt/cni/bin]
- OPSX-3528: [Pulse] Prometheus config reload fails if multiple remote storage configurations exist with the same name
- It is possible to create multiple remote storage configurations with the same name. However, if such a situation occurs, the metrics will not flow to the remote storage as the config reload of the original prometheus will fail.
- OPSX-1405: Able to create multiple CDP PVC Environments with the same name
- If two users try to create an environment with the same name at the same time, it might result in an unusable environment.
- OPSX-1412: Creating a new environment through the CDP CLI reports intermittently that "Environment name is not unique" even though it is unique
- When multiple users try to create the same environment at the same time or use automation to create an environment with retries, create environment may fail on collision with a previous request to create an environment.
Known Issues in Management Console 1.5.0
- Somehow the Rebuilding field inside volume.meta is set to true causing the volume to get stuck in attaching/detaching loop
-
This is a condition that can occur in ECS Longhorn storage.
Known Issues in Management Console identified before 1.5.0
- INSIGHT-2469: COE Insight from case 922848: Not able to connect to bit bucket
- After installing CML on an ECS cluster, users were not able to connect the internal bitbucket repo.
- OPSX-2484: FileAlreadyExistsException during timestamp filtering
- The timestamp filtering may result in FileAlreadyExistsException
when there is a file with same name already existing in the
tmp
directory. - OPSX-2772: For Account Administrator user, update roles functionality should be disabled
- An Account Administrator user holds the biggest set of privileges, and is not allowed to modify via current UI, even user try to modify permissions system doesn't support changing for account administrator.
- Recover fast in case of a Node failures with ECS HA
- When a node is deleted from cloud or made unavailable, it is observed that the it takes more than two minutes until the pods were rescheduled on another node.
- CDP Private Cloud Data Services ECS Installation: Failed to perform First Run of services.
- If an issue is encountered during the Install Control Plane step of Containerized Cluster First Run, installation will be re-attempted infinitely rather than the command failing.
- Environment creation through the CDP CLI fails when the base cluster includes Ozone
- Problem: Attempt to create an environment using the CDP command-line interface fails in a CDP Private Cloud Data Services deployment when the Private Cloud Base cluster is in a degraded state and includes Ozone service.
- Filtering the diagnostic data by time range might result in a FileAlreadyExistsException
- Problem:Filtering the collected diagnostic data might result in a FileAlreadyExistsException if the /tmp directory already contains a file by that name.
- Kerberos service does not always handle Cloudera Manager downtime
- Problem: The Cloudera Manager Server in the base cluster must be running to generate Kerberos principals for CDP Private Cloud. If there is downtime, you might observe Kerberos-related errors.
- Updating user roles for the admin user does not update privileges
- In the Management Console, changing roles on the User Management page does not change privileges of the admin user.
- Upgrade applies values that cannot be patched
- If the size of a persistent volume claim in a Containerized Cluster is manually modified, subsequent upgrades of the cluster will fail.