Known issues for the CDP Private Cloud Data Services Management Console
This section lists known issues that you might run into while using the CDP Private Cloud Management Console service.
Known Issues in Management Console 1.5.1
- OPSX-4560: ECS server restart failed in an air gap environment as it requires "yum install" from repo
- ECS service restart may fail in an air gap environment with a yum download error.
- OPSAPS-68558: [7.9.5->7.11.3.2] CM upgrade failed with BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState'
-
After upgrading the Cloudera Manager package, the Cloudera Manager Server does not start. An error about "Active Commands" is shown in the Cloudera Manager Server log.
This may happen when the Private Cloud Data Services Control Plane is actively issuing requests to Cloudera Manager while an upgrade is being performed.
- External metadata databases are no longer supported on OCP
-
As of CDP Private Cloud Data Services 1.5.1, external Control Plane metadata databases are no longer supported. New installs require the use of an embedded Control Plane database. Upgrades from CDP Private Cloud Data Services 1.4.1 or 1.5.0 to 1.5.1 are supported, but there is currently no migration path from a previous external Control Plane database to the embedded Control Plane database. Upgrades from 1.4.0 or 1.5.0 with external Control Plane metadata databases also require additional steps, which are described in the CDP Private Cloud Data Services 1.5.1 upgrade topics.
- DOCS-18031: Nodes are in "Not Ready" status during Rolling Restart of ECS
-
During a rolling restart of ECS, nodes are in a "Not Ready" state, and the
dmesg
command returns the following error message on the applicable nodes.[Tue Aug 8 16:30:50 2023] nfs: server 10.46.157.145 not responding, timed out [Tue Aug 8 16:31:16 2023] nfs: server 10.46.157.145 not responding, timed out [Tue Aug 8 16:31:30 2023] nfs: server 10.46.157.145 not responding, timed out
Also, the
df
command may hang on these hosts.
- OPSAPS-67214: Single Node | Restart Stability | Rolling start is failing with "global timeout reached: 10m0s, error when evicting pods"
-
For the ECS service, rolling restart is not applicable to a single node cluster.
- CDPVC-1098: How to refresh the YuniKorn configuration
-
Sometimes it is possible for the scheduler state to go out of sync from the cluster state. This may result in pods in Pending and ApplicationRejected states, with pod events showing Placement Rule related errors. To recover from this, you may need to refresh the YuniKorn configuration.
- OPSAPS-67340: L1 runs failing as service monitor is in bad health state
-
Service Monitor (SMON) ends up in a bad health state after restarting the Cloudeara Manager (CM) server, reporting problems with descriptor and metric schema age, when Kerberos and CM SPNEGO authentication are both enabled.
- DOCS-15855: Networking API is deprecated after upgrade to CDP Private Cloud Data Services 1.5.1 (K8s 1.24)
-
When the control plane is upgraded from 1.4.1 to 1.5.1, the Kubernates version changes to 1.24. The Livy pods running in existing Virtual Clusters (VCs) use a deprecated networking API for creating ingress for Spark driver pods. Because the old networking API is deprecated and does not exist in Kubernates 1.24, any new job run will not work for the existing VCs.
- CDPQE-24295: Update docker client on docker.lab.eng.hortonworks machine
-
When you attempt to execute the Docker command to fetch the Cloudera-provided images into your air-gapped environment, you may encounter an issue where Docker pulls an incorrect version of the HAProxy image, especially if you are using an outdated Docker client. This situation arises due to the Cloudera registry containing images with multiple platform versions. Unfortunately, older Docker clients may lack the capability to retrieve the appropriate architecture version, such as amd64.
- OPSX-4326: OCP upgrade from 1.5.0 to 1.5.1 – Restore is unsuccessful after upgrade
-
After upgrading CDP Private Cloud Data Services on OCP from 1.5.0 to 1.5.1, Restore using a 1.5.0 backup could not be performed successfully.
- OPSX-4266: ECS upgrade from 1.5.0 to 1.5.1 is failing in Cadence schema update job
-
When upgrading from ECS 1.5.0 to 1.5.1, the CONTROL_PLANE_CANARY fails with the following error:
Firing alerts for Control Plane: Job did not complete in time, Job failed to complete.
And the
cdp-release-cdp-cadence-schema-update
job fails.
- OPSX-4076:
- When you delete an environment after the backup event, the restore operation for the backup does not bring up the environment.
- OPSX-4295:
- The logs for the backups created in CDP Private Cloud Data Services 1.5.0 version do not appear after you upgrade to version 1.5.1.
- OPSX-4024: CM truststore import into unified truststore should handle duplicate CommonNames
-
If multiple CA certificates with the exact same value for the Common Name field are present in the Cloudera Manager truststore when a Private Cloud Data Services cluster is installed, only one of them may be imported into the Data Services truststore. This may cause certificate errors if an incorrect/old certificate is imported.
- OPSX-2713: PVC ECS Installation: Failed to perform First Run of services
-
If an issue is encountered during the Install Control Plane step of the Containerized Cluster First Run, installation will be re-attempted infinitely rather than the command failing.
- OPSX-3666: mlx_crud_app DB connection fails with error "unable to create connection: x509: certificate relies on legacy Common Name field, use SANs instead"
-
After upgrade, the mlx-crud-app fails with the error "unable to create connection: x509: certificate relies on legacy Common Name field, use SANs instead"
- OPSAPS-66166: FreeIPA cmadminrole needs more privileges for PvC+ after upgrade
-
After upgrade, the Cloudera Manager admin role may be missing the Host Administrators privilege in an upgraded cluster.
- COMOPS-2822: OCP error x509: certificate signed by unknown authority
-
The error
x509: certificate signed by unknown authority
usually means that the Docker daemon that is used by Kubernetes on the managed cluster does not trust the self-signed certificate.
- OPSX-4225: Upgrade failed as cadence pods are crashlooping post upgrade
-
When doing a fresh install of CDP Private Cloud Data Services 1.5.1, external metadata databases are no longer supported. Instead, the CDP Private Cloud Data Services installer will create an embedded database pod by default, which runs inside the Kubernetes cluster to host the databases required for installation.
If you are upgrading from CDP Private Cloud Data Services 1.4.1 or 1.5.0 to 1.5.1, and you were previously using an external database, you must run the following
psql
commands to create the required databases. You should also ensure that the two new databases are owned by the common database users known by the control plane.CREATE DATABASE db-cadence; CREATE DATABASE db-cadence-visibility;
- OPSAPS-67046: Docker Server role fails to come up and returns a connection error during ECS upgrade
-
When upgrading from 1.4.1 to 1.5.1, a Docker server role can sometimes fail to come up and return the following error:
grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}. Err :connection error: desc = "transport: error while dialing: dial unix:///var/run/docker/containerd/containerd.sock: timeout". Reconnecting... module=grpc failed to start containerd: timeout waiting for containerd to start
This error appears in the
stderr
file of the command, and can be caused by a mismatch in the pid of containerd.
- Longhorn-4212 Somehow the Rebuilding field inside volume.meta is set to true causing the volume to get stuck in attaching/detaching loop
-
This is a condition that can occur in ECS Longhorn storage.
- OPSX-3073 [ECS] First run command failed at setup storage step with error "Timed out waiting for local path storage to come up"
- Pod stuck in pending state on host for a long time. Error in
Role log related to CNI plugin:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 3m5s (x269 over 61m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "70427e9b26fb014750dfe4441fdfae96cb4d73e3256ff5673217602d503e806f": failed to find plugin "calico" in path [/opt/cni/bin]
- OPSX-3528: [Pulse] Prometheus config reload fails if multiple remote storage configurations exist with the same name
- It is possible to create multiple remote storage configurations with the same name. However, if such a situation occurs, the metrics will not flow to the remote storage as the config reload of the original prometheus will fail.
- OPSX-1405: Able to create multiple CDP PVC Environments with the same name
- If two users try to create an environment with the same name at the same time, it might result in an unusable environment.
- OPSX-1412: Creating a new environment through the CDP CLI reports intermittently that "Environment name is not unique" even though it is unique
- When multiple users try to create the same environment at the same time or use automation to create an environment with retries, create environment may fail on collision with a previous request to create an environment.
- OPSX-3323: Custom Log Redaction | String is not getting redacted from all places in diagnostic bundle
- Custom redaction rule for URLs does not work.
- Cloudera Data Engineering service fails to start due to Ozone
- If the Ozone service is missing, misconfigured, or having other issues when an Environment is registered in the Management Console, CDE fails to start.
Known Issues in Management Console 1.5.0
- Longhorn-4212 Somehow the Rebuilding field inside volume.meta is set to true causing the volume to get stuck in attaching/detaching loop
-
This is a condition that can occur in ECS Longhorn storage.
- COMPX-13185 Upgrade from 1.4.1 to 1.5.0 failed - error: timed out waiting for the condition on jobs/helm-install-longhorn
- Before ECS upgrade, you must update a specific ECS server node toleration explicitly to ensure a cleaner upgrade process.
- OPSX-3073 [ECS] First run command failed at setup storage step with error "Timed out waiting for local path storage to come up"
- Pod stuck in pending state on host for a long time. Error in
Role log related to CNI plugin:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 3m5s (x269 over 61m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "70427e9b26fb014750dfe4441fdfae96cb4d73e3256ff5673217602d503e806f": failed to find plugin "calico" in path [/opt/cni/bin]
- OPSX-3528: [Pulse] Prometheus config reload fails if multiple remote storage configurations exist with the same name
- It is possible to create multiple remote storage configurations with the same name. However, if such a situation occurs, the metrics will not flow to the remote storage as the config reload of the original prometheus will fail.
- OPSX-2062: Platform not shown on the Compute Cluster UI tab
- On the CDP Private Cloud Management Console UI in ECS, when listing the compute clusters, the Platform field is empty (null) instead of displaying RKE as the Platform.
- OPSX-1405: Able to create multiple CDP PVC Environments with the same name
- If two users try to create an environment with the same name at the same time, it might result in an unusable environment.
- OPSX-1412: Creating a new environment through the CDP CLI reports intermittently that "Environment name is not unique" even though it is unique
- When multiple users try to create the same environment at the same time or use automation to create an environment with retries, create environment may fail on collision with a previous request to create an environment.
- OPSX-2062: Platform not shown on the Compute Cluster UI tab
- On CDP Private Console UI in ECS, when listing the compute clusters, the Platform field is empty (null) instead of displaying RKE as the Platform.
- OPSX-3323: Custom Log Redaction | String is not getting redacted from all places in diagnostic bundle
- Custom redaction rule for URLs does not work.
- Cloudera Data Engineering service fails to start due to Ozone
- If the Ozone service is missing, misconfigured, or having other issues when an Environment is registered in the Management Console, CDE fails to start.
Known Issues in Management Console 1.4.1
- INSIGHT-2469: COE Insight from case 922848: Not able to connect to bit bucket
- After installing CML on an ECS cluster, users were not able to connect the internal bitbucket repo.
- OPSAPS-67046: Docker Server role fails to come up and returns a connection error
-
A Docker server role can sometimes fail to come up and return the following error:
grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}. Err :connection error: desc = "transport: error while dialing: dial unix:///var/run/docker/containerd/containerd.sock: timeout". Reconnecting... module=grpc failed to start containerd: timeout waiting for containerd to start
This error appears in the
stderr
file of the command, and can be caused by a mismatch in the pid of containerd. - OPSX-1405: Able to create multiple CDP PVC Environments with the same name
- If two users try to create an environment with the same name at the same time, it might result in an unusable environment.
- OPSX-1412: Creating a new environment through the CDP CLI reports intermittently that "Environment name is not unique" even though it is unique
- When multiple users try to create the same environment at the same time or use automation to create an environment with retries, create environment may fail on collision with a previous request to create an environment.
- OPSX-3323: Custom Log Redaction | String is not getting redacted from all places in diagnostic bundle
- Custom redaction rule for URLs does not work.
- Cloudera Data Engineering service fails to start due to Ozone
- If the Ozone service is missing, misconfigured, or having other issues when an Environment is registered in the Management Console, CDE fails to start.
Known Issues in Management Console 1.4.0
- Cloudera Data Engineering service fails to start due to Ozone
- If the Ozone service is missing, misconfigured, or having other issues when an Environment is registered in the Management Console, CDE fails to start.
- OPSX-2062: Platform not shown on the Compute Cluster UI tab
- On CDP Private Console UI in ECS, when listing the compute clusters, the Platform field is empty (null) instead of displaying RKE as the Platform.
- OPSX-2713: ECS Installation: Failed to perform First Run of services.
- If an issue is encountered during the Install Control Plane step of Containerized Cluster First Run, installation will be re-attempted infinitely rather than the command failing.
- OPSX-735: Kerberos service should handle CM downtime
- The Cloudera Manager Server in the base cluster must be running in order to generate Kerberos principals for Private Cloud. If there is downtime, you may observe Kerberos-related errors.
- OPSX-1405: Able to create multiple CDP PVC Environments with the same name
- If two users try to create an environment with the same name at the same time, it might result in an unusable environment.
- OPSX-1412: Creating a new environment through the CDP CLI intermittently reports that, "Environment name is not unique" even though it is unique
- When multiple users try to create the same environment at the same time or use automation to create an environment with retries, create environment may fail on collision with a previous request to create an environment.
- OPSX-2484: FileAlreadyExistsException during timestamp filtering
- The timestamp filtering may result in
FileAlreadyExistsException when there is a file with same name
already existing in the
tmp
directory. - OPSX-2772: For Account Administrator user, update roles functionality should be disabled
- An Account Administrator user holds the biggest set of privileges, and is not allowed to modify via current UI, even user try to modify permissions system doesn't support changing for account administrator.
Known Issues for Management Console 1.3.x and lower
- Recover fast in case of a Node failures with ECS HA
- When a node is deleted from cloud or made unavailable, it is observed that the it takes more than two minutes until the pods were rescheduled on another node.
- Cloudera Manager 7.6.1 is not compatible with CDP Private Cloud Data Servicesversion 1.3.4.
- You must use Cloudera Manager version 7.5.5 with this release.
- CDP Private Cloud Data Services ECS Installation: Failed to perform First Run of services.
- If an issue is encountered during the Install Control Plane step of Containerized Cluster First Run, installation will be re-attempted infinitely rather than the command failing.
- Environment creation through the CDP CLI fails when the base cluster includes Ozone
- Problem: Attempt to create an environment using the CDP command-line interface fails in a CDP Private Cloud Data Services deployment when the Private Cloud Base cluster is in a degraded state and includes Ozone service.
- Filtering the diagnostic data by time range might result in a FileAlreadyExistsException
- Problem:Filtering the collected diagnostic data might result in a FileAlreadyExistsException if the /tmp directory already contains a file by that name.
- Full cluster name does not display in the Register Environment Wizard
- Kerberos service does not always handle Cloudera Manager downtime
- Problem: The Cloudera Manager Server in the base cluster must be running to generate Kerberos principals for CDP Private Cloud. If there is downtime, you might observe Kerberos-related errors.
- Management Console allows registration of two environments of the same name
- Problem: If two users attempt to register environments of the same name at the same time, this might result in an unusable environment.
- Not all images are pushed during upgrade
- A retry of a failed upgrade intermittently fails at the Copy Images to Docker Registry step due to images not being found locally.
- The Environments page on the Management Console UI for an environment in a deployment using ECS does not display the platform name
- Problem: When you view the details of an environment using the Management Console UI in a CDP Private Cloud Data Services deployment using ECS, the Platform field appears blank.
- Updating user roles for the admin user does not update privileges
- In the Management Console, changing roles on the User Management page does not change privileges of the admin user.
- Upgrade applies values that cannot be patched
- If the size of a persistent volume claim in a Containerized Cluster is manually modified, subsequent upgrades of the cluster will fail.
- Incorrect warning about stale Kerberos client configurations
-
If Cloudera Manager is configured to manage krb5.conf, ECS clusters may display a warning that they have stale Kerberos client configurations. Clicking on the warning may show an "Access denied" error.
- Vault becomes sealed
- If a host in an ECS cluster fails or restarts, the Vault may have become sealed. (You may see a Health Test alert in Cloudera Manager for the ECS service stating Vault instance is sealed.)