Setting up your Streams Messaging clusterPDF version

Creating your cluster

Once you have met the prerequisites, you are ready to create a managed and secured Streams Messaging cluster in CDP Public Cloud by using one of the prescriptive Streams Messaging cluster definitions.

  1. Log into the CDP web interface.
  2. Navigate to Management Console > Environments, and select the environment where you would like to create a cluster.
  3. Click Create Data Hub.
    The following page is displayed:
  4. Select Cluster Definition.
  5. Select the appropriate Streams Messaging cluster definition from the Cluster Definition dropdown depending on your operational objectives.
    The list of services is automatically shown below the selected cluster definition name. For more information on templates, see Data Hub cluster definitions and Streams Messaging cluster layout.
  6. Give the cluster a name and add any tags you might need.
    You can define tags that will be applied to your cluster-related resources on your cloud provider account. For more information about tags, see Tags.
  7. If using the Streams Messaging High Availability definitions, ensure that you select multiple subnets.
    Multiple subnets can be selected by navigating to Advanced Options > Network And Availability.
  8. Optional: Add Kafka KRaft to your cluster.
    Kraft nodes are not provisioned by default in any of the Streams Messaging cluster definitions. If you want to run Kafka in KRaft mode, you must set the Instance Count of the Kraft Nodes host group to an odd number. An odd number of nodes is required because KRaft requires an odd number of instances to function. Cluster provisioning fails if the number of KRaft nodes is set to an even number.
    1. Go to Advanced Options > Hardware and Storage.
    2. Locate the Kraft Nodes host group and click to edit host group details.
    3. Set the Instance Count to at an odd number
    4. Click to save your changes.
  9. Optional: Add SRM or Kafka Connect to Heavy Duty clusters.
    SRM and Kafka Connect nodes are not provisioned by default when using the Streams Messaging Heavy Duty definition. If you are using this definition and want to have SRM, Kafka Connect, or both deployed in the cluster, set the Instance Count of the Srm Nodes or Connect Nodes host group to at least one.
    1. Go to Advanced Options > Hardware and Storage.
    2. Locate the appropriate host group and click to edit host group details.
    3. Set the Instance Count to at least 1.
    4. Click to save your changes.
  10. Optional: Use the Advanced Options section to customize the infrastructure settings.
  11. Click Provision Cluster.
You will be redirected to the Data Hub cluster dashboard, and a new tile representing your cluster will appear at the top of the page.
What steps you take next depends on how you configured your cluster
  • If you have provisioned KRaft nodes in your cluster, you can optionally choose to remove ZooKeeper from your cluster. For more information, see Deleting ZooKeeper from Streams Messaging clusters.
  • If you configured Attached Volume per Instances and the volume count is not identical for the Broker and Core_brokers host groups, continue with Configuring data directories for clusters with custom disk configurations.
  • If you did not customize Attached Volume per Instances or do not want to remove ZooKeeper, continue with Give users access to your cluster.