Cloudera Docs

Creating your cluster

Once you have met the prerequisites, you are ready to create a managed and secured Flow Management cluster in CDP Public Cloud by using one of the prescriptive flow management cluster definitions. You can select the cluster definition that matches your cloud provider and choose from the light and heavy duty options available.

You can create your Flow Management cluster using the CDP UI or CDP CLI. The UI offers a graphical interface that simplifies the cluster creation process. You can select the desired configurations for your cluster easily.

When you create a new cluster through the UI, Java 8 is used as the default JDK. The ability to deploy a new Flow Management cluster with Java 11 is available only using the CLI. The CDP command-line interface also allows you to leverage scripting and automation capabilities to create clusters programmatically or perform bulk operations.

Steps

  1. Log into the CDP web interface.
  2. Navigate to Management Console > Environments, and select the environment where you would like to create a cluster.
  3. Click Create Data Hub.
    The Provision Data Hub page is displayed:
  4. Select Cluster Definition.
  5. Select the appropriate Flow Management cluster definition from the Cluster Definition dropdown depending on your operational objectives.

    There are six template options available:

    • Flow Management Light Duty for AWS
    • Flow Management Heavy Duty for AWS
    • Flow Management Light Duty for Azure
    • Flow Management Heavy Duty for Azure
    • Flow Management Light Duty for GCP
    • Flow Management Heavy Duty for GCP

    For more information on templates, see Flow Management cluster definitions and Flow Management cluster layout.

    The list of services is automatically shown below the selected cluster definition name.

  6. Give the cluster a name and add any tags you might need.

    You can define tags that will be applied to your cluster-related resources on your cloud provider account. For more information about tags, see Tags.

  7. Optional: Use the Configure Advanced Options section to customize the infrastructure settings.
  8. Click Provision Cluster.

Result

You will be redirected to the Data Hub cluster dashboard, and a new tile representing your cluster will appear at the top of the page.

Before you begin

You have installed and configured CDP CLI. For more information about setting up the CLI client, see CDP CLI.

Steps

Use the create-<cloud_provider>-cluster command and include the --java-version parameter to create your Flow Management cluster.

Valid inputs for the --java-version parameter are 8 and 11.

Example

cdp datahub create-aws-cluster \
  --cluster-name nifi-cluster \
  --environment-name example-sandbox-aws \
  --cluster-template-name "7.2.17 - Flow Management Light Duty with Apache NiFi, Apache NiFi Registry, Schema Registry" \
  --instance-groups nodeCount=3,instanceGroupName=nifi,instanceGroupType=CORE,instanceType=m5.2xlarge,rootVolumeSize=150,attachedVolumeConfiguration=\[\{volumeSize=500,volumeCount=4,volumeType=standard\}\],recoveryMode=MANUAL nodeCount=1,instanceGroupName=management,instanceGroupType=GATEWAY,instanceType=m5.2xlarge,rootVolumeSize=100,attachedVolumeConfiguration=\[\{volumeSize=100,volumeCount=1,volumeType=standard\}\],recoveryMode=MANUAL nodeCount=0,instanceGroupName=nifi_scaling,instanceGroupType=CORE,instanceType=m5.2xlarge,rootVolumeSize=150,attachedVolumeConfiguration=\[\{volumeSize=500,volumeCount=4,volumeType=standard\}\],recoveryMode=MANUAL \
  --image id=6e038b51-0aea-43ce-873a-a0aa438b54f9,catalogName=cdp-default \
  --datahub-database NON_HA \
  --java-version 11 \
  --no-enable-load-balancer

After you finish

After you have created your Flow Managament Data Hub cluster, you are ready to build your first data flow in CDP Public Cloud.