Cloudera Docs

Creating virtual clusters

In Cloudera Data Engineering (CDE), a virtual cluster is an individual auto-scaling cluster with defined CPU and memory ranges. Jobs are associated with virtual clusters, and virtual clusters are associated with an environment. You can create as many virtual clusters as you need.

To create a virtual cluster, you must have an environment with Cloudera Data Engineering (CDE) enabled.

  1. From the CDE Overview page, select the environment you want to create a virtual cluster in.
  2. In the Virtual Clusters column, click the plus icon at the top right to create a new virtual cluster.
    If the environment has no virtual clusters associated with it, the page displays a Create DE Cluster button that launches the same wizard.
  3. Enter a Cluster Name.
    Cluster names must:
    • Begin with a letter
    • Be between 3 and 30 characters (inclusive)
    • Contain only alphanumeric characters and hyphens
  4. Select the CDE Service to create the virtual cluster in.
    The environment you selected before launching the wizard is selected by default, but you can use the wizard to create a virtual cluster in a different CDE service.
  5. Use the Auto-Scale Max Capacity sliders to set the maximum number of CPU cores and the maximum memory in gigabytes. The cluster will scale up and down as needed to run the submitted Spark applications.
  6. From the Driver and Executors will run on drop-down menu, select whether you want to run drivers and executors on spot instances or on-demand instances. By default, the driver runs on on-demand instances, and the executors run on spot instances. For SLA-bound workloads, select On-demand. For non-SLA workloads, Cloudera recommends leaving the default configuration to take advantage of the cost savings afforded by spot instances. For more information, see Cloudera Data Engineering Spot Instances.
  7. Select the Spark Version to use in the virtual cluster. You cannot use Spark 2 and Spark 3 in the same virtual cluster, but you can have separate Spark 2 and Spark 3 virtual clusters within the same CDE service.
  8. Optional: Select Restrict Access to add access control for the virtual cluster. You can search for users to add by name or email address. You can manage users using the Cloudera Data Platform Management Console. For more information, see Managing user access and authorization.
  9. Optional: Select Enable Airflow Authoring UI (Technical Preview).
    This option enables a graphical workflow editor where you can create multi-step Airflow pipelines by drag-and-dropping a combination of out-of-the-box operators on a canvas and creating relationships between them with a mouse click.

    For best user experience, Cloudera suggests using Google Chrome for this feature.

  10. Optional: Click Configure Airflow Email (Technical Preview) if you want to receive notification mails.The email configuration options appear.
    1. You must provide at least Sender Email Address and SMTP Host information.
    2. Test SMTP Configs: Click Test SMTP Configs to test the configurations set for SMTP. This helps you to test the SMTP configuration before creating the cluster.
  11. Optional: Click Enable Remote Shuffle Service (Technical Preview) if you want to store Spark shuffle data on remote servers. Usage of Remote Shuffle Service (RSS) improves resilience in the case of executor loss and it allows to run jobs with regular Dynamic Allocation (without shuffle tracking). It may also decrease the job execution time depending on the number of RSS instances.
    1. Override Instances: Click to override the recommended RSS instances for this virtual cluster.
      The recommended number of RSS instances: the recommendation on a number of RSS instances for a virtual cluster is according to the defined virtual cluster CPU quota. This number is obtained according to the RSS performance testing results. Increasing the number of RSS servers to some point could help decrease job execution time, but will also have cost implications.
    2. Instances: Drag the slider button to specify the number of RSS instances.

      Each RSS instance runs on i3.xlarge AWS EC2 instance, so each instance will contribute i3.xlarge cost to the total CDE cost . Overall shuffle data volume that can be stored by a virtual cluster at the same time is number of RSS instances * 880 GiB (gibibytes).

  12. Click Create.
On the CDE Overview page, select the environment to view the virtual cluster initialization status. You can also click the three-dot menu for the virtual cluster to view the logs.