Cloudera Docs

Streams Messaging cluster layout

Describes the layout and capacity of the Streams Messaging: Light Duty and Streams Messaging: Heavy Duty cluster definitions

Streams Messaging: Light Duty cluster layout

You can use a Streams Messaging: Light Duty cluster definition in development, testing, or proof of concept scenarios. Each of the cluster nodes includes:

  • Kafka Core Broker, Kafka Broker, ZooKeeper, and the SRM driver are co-located on all instances

  • For each node hosting Kafka Core Broker, Kafka Broker, ZooKeeper, and the SRM driver instances:
    • AWS: m5.2xlarge
    • Azure: D8_v3
    • GCP: e2-standard-8
  • Storage configuration per Kafka Core Broker node:
    • AWS: 1 TB Volume EBS ST1
    • Azure: 1 TB Standard Locally-redundant SSD storage
    • GCP: 1 TB Zonal PD-SSD
  • Storage configuration per Kafka Broker node:
    • AWS: 1 TB Volume EBS ST1
    • Azure: 1 TB Standard Locally-redundant SSD storage
    • GCP: 1 TB Zonal PD-SSD
  • Master node instance
    • AWS: m5.2xlarge
    • Azure: Standard_D8_v3
    • GCP: e2-standard-8
  • Storage configuration for Master node:
    • AWS: 100 GB Volume EBS Magnetic
    • Azure: 100 GB Standard Locally-redundant SSD storage
    • GCP: 100 GB Zonal PD-Standard

For more information about the cloud provider-specific instance and storage types, see the Related Information section.

The Broker node is not provisioned by default. You have the option to manually set how many Broker nodes are created when provisioning the cluster. After the cluster is provisioned, the number of Broker nodes can be changed by scaling your cluster. For more information about scaling, Core Broker and Broker nodes, see Scaling Streams Messaging Clusters.

Streams Messaging: Heavy Duty cluster layout

You can use the Streams Messaging: Heavy Duty cluster definition in production scenarios. The cluster definition includes:

Azure

  • Master Node – Containing Knox, Cloudera Manager, ZooKeeper
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant SSD storage
  • Registry Nodes – Containing Schema Registry, ZooKeeper
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage
  • SMM Nodes – Containing SMM, Schema Registry, ZooKeeper
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage
  • Core Broker Nodes – Containing a Kafka Broker
    • Instance type – Standard_D8s_v3
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – Standard_D8s_v3
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • SRM Nodes – Containing the SRM Driver and Service
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage

AWS

  • Master Node – Containing Knox, Cloudera Manager, ZooKeeper
    • Instance type – r5.2xlarge
    • Storage configuration – 100 GB EBS SC1
  • Registry Nodes – Containing Schema Registry, ZooKeeper
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1
  • SMM Nodes – Containing SMM, Schema Registry, ZooKeeper
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1
  • Core Broker Nodes – Containing a Kafka Broker
    • Instance type – m5.2xlarge
    • Storage configuration – 1 TB GP2 SSD
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – m5.2xlarge
    • Storage configuration – 1 TB GP2 SSD
  • SRM Nodes - Containing the SRM Driver and Service
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1

GCP

  • Master Node – Containing Knox, Cloudera Manager, ZooKeeper
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • Registry Nodes – Containing Schema Registry, ZooKeeper
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • SMM Nodes – Containing SMM, Schema Registry, ZooKeeper
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • Core Broker Nodes – Containing a Kafka Broker
    • Instance type – e2-standard-8
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – e2-standard-8
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • SRM Nodes - Containing the SRM Driver and Service
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard

For more information about the cloud provider-specific instance and storage types, see the Related Information section.

The SRM and Broker nodes are not provisioned by default. When provisioning a cluster with this definition, you have to manually set the instance count of the appropriate host group to at least 1. Otherwise the host group and its nodes are not provisioned. After a cluster is provisioned, you also have the option to scale these nodes. For more information on scaling, see Scaling Streams Messaging Clusters.