Cloudera Data Engineering cluster types

Learn more about the available cluster types that you can use in your Cloudera Data Engineering service.

On the Cloudera Data Engineering UI, DE and Platform administrators can create new Virtual Clusters and select the required cluster type at Administration > Create a Virtual Cluster.

Cloudera Data Engineering offers these cluster types:

Core
All purpose

In the same Cloudera Data Engineering Service, you can run some workloads on Core and some workloads on All purpose clusters.

Core cluster functionalities:

Autoscaling cluster
Spot instances
SDX
Open Lakehouse with Iceberg
Job lifecycle management
Centralized monitoring
Workflow orchestration (Airflow)

All Purpose cluster functionalities (on top of functionalities available with Core clusters as well):

Interactive sessions
External IDE connectivity
Spark streaming
JDBC connector (Coming soon)

At Administration > Enable a Service > Capacity & Costs > Autoscale Range, you can set the autoscale range for the On-demand Instances, that is for the Core clusters. The range you set on this page for the On-demand Instances is used also for the All purpose Instances, that is for the All purpose clusters.

At Administration > Service Details, you can modify the autoscale value range of:

On-demand Instances
All purpose On-demand Instances
Spot Instances
All purpose Spot Instances

You can use Spot Instances and All purpose Spot Instances to save compute costs.

important

With All purpose On-demand Instances, consider the following:

Since version 1.20.3, Cloudera Data Engineering offers price tiering to provide better cost-control for the consumed features. The All purpose On-demand Instances are charged at a higher rate than the On-demand Instances (Core instances). For more information, see the pricing page.
To avoid being charged for All purpose compute nodes, even if you do not create an All purpose Virtual Cluster, set the minimum value of All purpose On-demand Instances to 0.
If you set the autoscale range minimum value of All purpose On-demand Instances to 0, if you create a Session, the Session can take several minutes to start.
If you set the autoscale range minimum value of All purpose On-demand Instances to 1 or higher, you are charged for All purpose compute nodes even if you do not create an All purpose cluster.