Hardware Sizing Recommendations

Recommendations for Kafka

• Kafka Broker node: eight cores, 64 GB to128 GB of RAM, two or more 8-TB SAS/SSD disks, and a 10- GbE NIC.

• Minimum of three Kafka broker nodes

• Hardware Profile: More RAM and faster speed disks are better; 10 GbE NIC is ideal.

• 75 MB per sec per node is a conservative estimate. You can go much higher if more RAM and reduced lag between writing/reading and therefore 10 GB NIC is required.

With a minimum of 3 nodes in your cluster, you can expect 225 MB/sec data transfer.

You can perform additional further sizing by using the following formula: num_brokers = desired_throughput (MB/sec) / 75

Recommendations for Storm

• Storm Worker Node: 8 core, 64 GB RAM, 1 GbE NIC

• Minimum of 3 Storm worker nodes

• Nimbus Node: Minimum 2 Nimbus nodes , 4 core, 8 GB RAM

• Hardware profile: disk I/O is not that important; more cores are better.

• 50 MB per sec per node with low to moderate complexity topology reading from Kafka and no external lookups . Medium-complexity and high-complexity topologies might have reduced throughput.

With a minimum 2 nimbus, 2 worker cluster, you can expect to run 100 MB/sec of low to medium complexity topology.

Further sizing can be done as follows. Formula: num_worker_nodes = desired_throughput(MB/sec) / 50