Cloudera Docs

Creating a virtual cluster

In Cloudera Data Engineering (CDE), a virtual cluster is an individual auto-scaling cluster with defined CPU and memory ranges. Jobs are associated with virtual clusters, and virtual clusters are associated with an environment. You can create as many virtual clusters as you need.

  1. From the CDE Overview page, select the CDE service that you created earlier.
  2. In the Virtual Clusters column, click to create a new virtual cluster.
    If the environment has no virtual clusters associated with it, the page displays a Create DE Cluster option that launches the same wizard.
  3. On the Create a Cluster page, enter a Cluster Name.
    Cluster names must:
    • begin with a letter
    • be between 3 and 30 characters (inclusive)
    • contain only alphanumeric characters and hyphens
  4. Select the CDE Service that you created earlier from the CDE Service drop-down menu.
    The CDE service you selected before launching the wizard is selected by default, but you can use the wizard to create a virtual cluster in a different CDE service.
  5. Accept the default settings for the Autoscale Max Capacity.
  6. Select Spark 2.4.7 as the Spark Version to use in the virtual cluster.
    You cannot use Spark 2 and Spark 3 in the same virtual cluster, but you can have separate Spark 2 and Spark 3 virtual clusters within the same CDE service.
  7. Click Create.
On the CDE Overview page, select the environment to view the virtual cluster initialization status. You can also click for the virtual cluster to view the logs.