Cloudera Docs

Suspending and resuming Cloudera Data Engineering virtual clusters using the CDP CLI [Technical Preview]

Learn about how to suspend and resume Cloudera Data Engineering Virtual Clusters (VCs) using the CDP CLI. The Virtual Cluster-level suspend and resume feature enables you to temporarily pause one or more Cloudera Data Engineering VCs during idle periods. Cloudera has observed that suspending a larger number of VCs offers significant cost savings, primarily on the compute resources.

The virtual cluster-level suspend and resume feature is supported on both AWS and Azure with no operational differences on these two cloud providers.

The virtual cluster-level suspend and resume feature is implemented as a convenient, user-friendly, and one-step procedure.

When you suspend a VC
  1. The VC is transitioned to maintenance mode, where:
    • The VC is locked
    • Running jobs and sessions are killed
    • Job schedules are paused
  2. The suspend operation is started by scaling down all the infrastructure deployments of the given VC.

    Depending on the distribution of the pods, the associated node can be released.

When you resume a VC
  1. First the VC is resumed.
  2. The VC exits maintenance mode, which involves:
    • Unpausing all the job schedules
    • Unlocking the VC

To suspend and resume your Cloudera Data Engineering virtual cluster, you need to:

  • Request the DE_ENABLE_VC_SUSPEND_RESUME entitlement
  • Have DEAdmin user privileges at the environment level
  • Set up ~/.cdp/config
  • Set up ~/.cdp/credentials
  • Make sure that you are using the CDP CLI Beta version because the suspend-vc and the resume-vc CLI options are only available in it. To install the CDP CLI Beta version, see Installing Beta CDP CLI.
  1. Install and run the BETA CDP CLI.
  2. Suspend your Cloudera Data Engineering virtual cluster. 
    cdp de suspend-vc --cluster-id <cluster-xxxxxx> --vc-id <dex-app-xxxxxx>
    cluster-id
    To obtain the cluster ID, log in to the Cloudera Data Engineering UI and navigate to Administration in the left-navigation menu. Select the respective service, then click on the Service Details icon, and copy the CLUSTER ID.
    vc-id
    To obtain the VC ID, log in to the Cloudera Data Engineering UI and navigate to Administration in the left-navigation menu. Select the respective service and then the respective VC. Click on the Virtual Cluster Details icon, and copy the VC-ID.
  3. Resume the previously suspended Cloudera Data Engineering virtual cluster. 
    cdp de resume-vc --cluster-id <cluster-xxxxxx> --vc-id <dex-app-xxxxxx>
  4. Check the suspend-resume status of the virtual cluster. 
    cdp de get-suspend-resume-status --cluster-id <cluster-xxxxxx> --vc-id <dex-app-xxxxxx>