TroubleshootingPDF version

Handling project volume size increase in Cloudera AI

When the storage capacity values of the Persistent Volume (PV) and Persistent Volume Claim (PVC) are increased, the update must also be reflected in Cloudera AI.

If the PVC size is not updated in Cloudera AI, you may encounter the following error messages when creating a workload:
  • Cannot bind to requested volume <volume name>: requested PV is too small.
  • Unschedulable: 0/8 nodes are available: 8 pods have unbound immediate PersistentVolumeClaims.

Cloudera AI has different services for handling the workload creation. The ds-operator service is responsible for creating the workload. The ds-vfs service is responsible for handling the project file system. During the startup, the ds-operator pod reads the project volume PVC and uses it as a reference to create the project volumes for each workload. This process does not automatically refresh when changes are made.

Workaround:

Scale down the ds-operator and the ds-vfs deployments to 0, then scale them back up to 1. This will prompt the ds-operator to read the updated PVC information and use it for subsequent workload creation..

  1. Scale down the ds-operator deployment.
    kubectl scale deployment --replicas=0 ds-operator -n <workspace namespace>

  2. Scale down the ds-vfs deployment.

    kubectl scale deployment --replicas=0 ds-vfs -n <workspace namespace>
  3. Scale up the ds-operator deployment.
    kubectl scale deployment --replicas=1 ds-operator -n <workspace namespace>
  4. Scale up the ds-vfs deployment.
    kubectl scale deployment --replicas=1 ds-vfs -n <workspace namespace>