Creating virtual clusters

A virtual cluster is an individual auto-scaling cluster with defined CPU and memory ranges. Jobs are associated with virtual clusters, and virtual clusters are tied to an environment. You can create as many virtual clusters as you need.

To create a virtual cluster, you must have an environment with Cloudera Data Engineering (CDE) enabled.

  1. From the CDE Overview page, select the environment you want to create a virtual cluster in.
  2. In the Virtual Clusters column, click the plus icon at the top right to create a new virtual cluster.
    If the environment has no virtual clusters associated with it, the page displays a Create DE Cluster button that launches the same wizard.
  3. Enter a Cluster Name.
    Cluster names must:
    • Begin with a letter
    • Be between 3 and 50 characters (inclusive)
    • Contain only alphanumeric characters and hyphens
  4. Select the CDE Service
    The environment you selected before launching the wizard is selected by default, but you can use the wizard to create a virtual cluster in a different CDE service.
  5. Use the Auto-Scale Range sliders to set the maximum number of CPU cores and the maximum memory in gigabytes. The cluster will scale up and down as needed to execute the submitted Spark applications.
  6. Click Create.
On the CDE Overview page, select the environment to view the virtual cluster initialization status. You can click the menu icon for the virtual cluster to view the logs.