Cloudera Docs

Streams Messaging cluster layout

Describes the layout and capacity of the Streams Messaging: Light Duty, Streams Messaging: Heavy Duty cluster and Streams Messaging: High Availability definitions

Streams Messaging: Light Duty cluster layout

You can use a Streams Messaging: Light Duty cluster definition in development, testing, or proof of concept scenarios.

  • Core Broker and Broker node instance:
    • AWS: m5.2xlarge
    • Azure: D8_v3
    • GCP: e2-standard-8
  • Storage configuration per Core Broker and Broker node:
    • AWS: 1 TB Volume EBS ST1
    • Azure: 1 TB Standard Locally-redundant SSD storage
    • GCP: 1 TB Zonal PD-SSD
  • Master node instance
    • AWS: m5.2xlarge
    • Azure: Standard_D8_v3
    • GCP: e2-standard-8
  • Storage configuration for Master node:
    • AWS: 100 GB Volume EBS Magnetic
    • Azure: 100 GB Standard Locally-redundant SSD storage
    • GCP: 100 GB Zonal PD-Standard

For more information about the cloud provider-specific instance and storage types, see the Related Information section.

The Broker node is not provisioned by default. You have the option to manually set how many Broker nodes are created when provisioning the cluster. After the cluster is provisioned, the number of Broker nodes can be changed by scaling your cluster. For more information about scaling, Core Broker and Broker nodes, see Scaling Streams Messaging Clusters.

Streams Messaging: Heavy Duty cluster layout

You can use the Streams Messaging: Heavy Duty cluster definition in production scenarios. The cluster definition includes:

Azure

  • Master Node – Containing Knox, Cloudera Manager, ZooKeeper
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant SSD storage
  • Registry Nodes – Containing Schema Registry, ZooKeeper
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage
  • SMM Nodes – Containing SMM, Schema Registry, ZooKeeper
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage
  • Core Broker Nodes – Containing a Kafka Broker
    • Instance type – Standard_D8s_v3
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – Standard_D8s_v3
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • SRM Nodes – Containing the SRM Driver and Service
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage
  • Connect Nodes – Containing a Kafka Connect role
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage

AWS

  • Master Node – Containing Knox, Cloudera Manager, ZooKeeper
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1
  • Registry Nodes – Containing Schema Registry, ZooKeeper
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1
  • SMM Nodes – Containing SMM, Schema Registry, ZooKeeper
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1
  • Core Broker Nodes – Containing a Kafka Broker
    • Instance type – m5.2xlarge
    • Storage configuration – 1 TB GP2 SSD
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – m5.2xlarge
    • Storage configuration – 1 TB GP2 SSD
  • SRM Nodes – Containing the SRM Driver and Service
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1
  • Connect Nodes – Containing a Kafka Connect role
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1

GCP

  • Master Node – Containing Knox, Cloudera Manager, ZooKeeper
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • Registry Nodes – Containing Schema Registry, ZooKeeper
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • SMM Nodes – Containing SMM, Schema Registry, ZooKeeper
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • Core Broker Nodes – Containing a Kafka Broker
    • Instance type – e2-standard-8
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – e2-standard-8
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • SRM Nodes – Containing the SRM Driver and Service
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • Connect Nodes – Containing a Kafka Connect role
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard

For more information about the cloud provider-specific instance and storage types, see the Related Information section.

The SRM, Broker, and Connect nodes are not provisioned by default. When provisioning a cluster with this definition, you have to manually set the instance count of the appropriate host group to at least 1. Otherwise, the host group and its nodes are not provisioned. After a cluster is provisioned, you also have the option to scale these nodes. For more information on scaling, see Scaling Streams Messaging Clusters.

Streams Messaging: High Availability cluster layout

You can use the Streams Messaging: High Availability cluster definition in production scenarios where having a highly available cluster spanning multiple Availability Zones (multi-AZ) is required. The cluster definition includes:

Azure

  • Master Node – Containing Cruise Control and SMM
    • Instance type – Standard_E16s_v3
    • Storage configuration – 100 GB Standard Locally-redundant SSD storage
  • Manager Nodes – Containing Cloudera Manager and Knox
    • Instance type – Standard_D16_v3
    • Storage configuration – 100 GB Locally-redundant storage
  • Core Zookeeper Nodes – Containing a Zookeeper and a Schema Registry Node
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage
  • Core Broker Nodes – Containing a Kafka Broker
    • Instance type – Standard_D8s_v3
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – Standard_D8s_v3
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • SRM Nodes – Containing the SRM Driver and Service
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage
  • Connect Nodes – Containing a Kafka Connect role
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage

AWS

  • Master Node – Containing Cruise Control and SMM
    • Instance type – r5.4xlarge
    • Storage configuration – 100 GB EBS SC1
  • Manager Nodes – Containing Cloudera Manager and Knox
    • Instance type – m5.4xlarge
    • Storage configuration – 100 GB Magnetic
  • Core Zookeeper Nodes – Containing a Zookeeper and a Schema Registry Node
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1
  • Core Broker Nodes – Containing a Kafka Broker
    • Instance type – m5.2xlarge
    • Storage configuration – 1 TB GP2 SSD
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – m5.2xlarge
    • Storage configuration – 1 TB GP2 SSD
  • SRM Nodes – Containing the SRM Driver and Service
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1
  • Connect Nodes – Containing a Kafka Connect role
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS SC1

GCP

  • Master Node – Containing Cruise Control and SMM
    • Instance type – e2-highmem-16
    • Storage configuration – 100 GB Zonal PD-Standard
  • Manager Nodes – Containing Cloudera Manager and Knox
    • Instance type – e2-standard-16
    • Storage configuration – 100 GB Standard persistent disks (HDD)
  • Core Zookeeper Nodes – Containing a Zookeeper and a Schema Registry Node
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • Core Broker Nodes – Containing a Kafka Broker
    • Instance type – e2-standard-8
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – e2-standard-8
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • SRM Nodes – Containing the SRM Driver and Service
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage
  • Connect Nodes – Containing a Kafka Connect role
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Standard Locally-redundant HDD storage

For more information about the cloud provider-specific instance and storage types, see the Related Information section.

The SRM, Broker, and Connect nodes are not provisioned by default. When provisioning a cluster with this definition, you have to manually set the instance count of the appropriate host group to at least 1. Otherwise, the host group and its nodes are not provisioned. After a cluster is provisioned, you also have the option to scale these nodes. For more information on scaling, see Scaling Streams Messaging Clusters.

When using the Streams Messaging High Availability definition, ensure that you select multiple subnets when provisioning the cluster. Otherwise, your cluster will not be highly available.