Overview of suspending and resuming Cloudera Data Engineering virtual clusters [Technical Preview]

Benefits of suspending virtual clusters

Suspending a VC scales down all associated infrastructure and job pods to zero. Depending on the distribution of the pods, suspending a VC can release one or more nodes, leading to cost savings. Cloudera has observed that suspending a larger number of VCs offers significant cost savings, primarily on the compute resources.

During suspension, all VC configurations, as well as jobs, and job runs are preserved, similar to hibernating a laptop during idle periods. While a VC is in suspended state, it does not allow the submission or the execution of jobs on the respective VC. However, other VCs remain operational.

To reduce operational costs, you can suspend a larger number of VCs during weekends, or overnight on weekdays.

The suspend and the resume operation can be initiated by:

• A user
• A scheduled event
• An automatic action

Prerequisites for suspending and resuming virtual clusters

Limitations

• You can suspend and resume VCs through the CDP CLI and the Cloudera Data Engineering API, but not through the Cloudera Data Engineering UI.
• Often you might not save on infrastructure nodes when you suspend a VC. This heavily depends on the distribution of infrastructure pods across nodes.
• When you a suspend a VC, the following operations are not supported:
  • Upgrading the Cloudera Data Engineering service
  • Backing up the Cloudera Data Engineering service
  • Backing up the suspended VC jobs (you can back up jobs from other running VCs)
• Attempting to suspend multiple VCs simultaneously can sometimes lead to failures. As a best practice, suspend VCs in a sequential manner.