Cloudera Docs

Streams Messaging cluster layout

Describes the layout and capacity of the Streams Messaging: Light Duty and Streams Messaging: Heavy Duty cluster definitions

Streams Messaging: Light Duty cluster layout

You can use a Streams Messaging: Light Duty cluster definition in development, testing, or proof of concept scenarios. Each of the cluster nodes includes:

  • Kafka Broker, ZooKeeper, and the SRM driver are co-located on all instances

  • For each node hosting Kafka Broker, ZooKeeper, and the SRM driver instances:
    • AWS: m5.2xlarge
    • Azure: D8_v3
    • GCP: e2-standard-8
  • Storage configuration per Kafka Broker node:
    • AWS: 1 TB Volume EBS ST1
    • Azure: 1 TB Standard Locally-redundant SSD storage
    • GCP: 1 TB Zonal PD-SSD
  • Master node instance
    • AWS: m5.2xlarge
    • Azure: Standard_D8_v3
    • GCP: e2-standard-8
  • Storage configuration for Master node:
    • AWS: 100 GB Volume EBS Magnetic
    • Azure: 100 GB Standard Locally-redundant SSD storage
    • GCP: 100 GB Zonal PD-Standard

For more information, see your cloud provider-specific information about instance types and storage information.

Streams Messaging: Heavy Duty cluster layout

You can use the Streams Messaging: Heavy Duty cluster definition in production scenarios. The cluster definition includes:

Azure

  • Master Node – Containing Knox, Cloudera Manager, ZooKeeper
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant SSD storage
  • Registry Nodes – Containing Schema Registry, ZooKeeper
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant SSD storage
  • SMM Nodes – Containing SMM, Schema Registry, ZooKeeper
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant SSD storage
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – Standard_D8s_v3
    • Storage configuration – 1 TB Premium Locally-redundant SSD storage
  • SRM Nodes – Containing the SRM Driver and Service
    • Instance type – Standard_D8_v3
    • Storage configuration – 100 GB Standard Locally-redundant SSD storage

AWS

  • Master Node – Containing Knox, Cloudera Manager, ZooKeeper
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS ST1
  • Registry Nodes – Containing Schema Registry, ZooKeeper
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS ST1
  • SMM Nodes – Containing SMM, Schema Registry, ZooKeeper
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS ST1
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – m5.2xlarge
    • Storage configuration – 1 TB EBS ST1
  • SRM Nodes - Containing the SRM Driver and Service
    • Instance type – m5.2xlarge
    • Storage configuration – 100 GB EBS ST1

GCP

  • Master Node – Containing Knox, Cloudera Manager, ZooKeeper
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • Registry Nodes – Containing Schema Registry, ZooKeeper
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • SMM Nodes – Containing SMM, Schema Registry, ZooKeeper
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard
  • Broker Nodes – Containing a Kafka Broker
    • Instance type – e2-standard-8
    • Storage configuration – 1 TB Zonal PD-SSD
  • SRM Nodes - Containing the SRM Driver and Service
    • Instance type – e2-standard-8
    • Storage configuration – 100 GB Zonal PD-Standard

For more information, see your cloud provider-specific information about instance types and storage information.