Cloudera Docs

Creating a Streams Messaging cluster

Learn how to create a Streams Messaging cluster for the Business Intelligence at Scale pattern. A Data Hub cluster with the Streams Messaging cluster definition hosts the Kafka topics into which you capture real-time streaming data. For this example, you are importing the weather data.

For the Business Intelligence at Scale pattern proof of concept, use the Streams Messaging Light Duty template for AWS. When you are ready to move your data pipeline to production, you can use either a Light or Heavy duty template, depending on your use case. For this pattern, use the default cluster definition.
  1. Log into the CDP web interface.
  2. Navigate to Management Console > Environments, and select the environment where you would like to create a cluster.
  3. Click Create Data Hub.
    The following page is displayed:
  4. Select the Cluster Definition option.
  5. Select the appropriate Streams Messaging cluster definition from the Cluster Definition drop-down menu under the Services section depending on your operational objectives.
    There are two definition options available for each cloud provider:
    • Streams Messaging Heavy Duty for AWS
    • Streams Messaging Light Duty for AWS
    The list of services is automatically shown below the selected cluster definition name as shown in the following image:

    Cruise Control, Kafka, Schema Registery, SMM, Stream Replication Manager, and Zookeeper services are included as dependent services in the cluster when you select Streams Messaging as the Cluster Definition.
  6. Specify a name for your cluster and add any tags you might need.
  7. Click Provision Cluster.
You are redirected to the Data Hub cluster dashboard, and a new tile representing your cluster appears at the top of the page.