Chapter 4. Hardware Sizing Recommendations
Recommendations for Kafka
Kafka Broker Node: eight cores, 64 GB to128 GB of RAM, two or more 8-TB SAS/SSD disks, and a 10- Gige Nic .
Minimum of three Kafka broker nodes
Hardware Profile: More RAM and faster speed disks are better; 10 Gige Nic is ideal.
75 MB/sec per node is a conservative estimate ( can go much higher if more RAM and reduced lag between writing/reading and therefore 10GB Nic is required ).
With a minimum of three nodes in your cluster, you can expect 225 MB/sec data transfer.
You can perform additional further sizing by using the following formula:
num_brokers = desired_throughput (MB/sec) / 75
Recommendations for Storm
Storm Worker Node: 8 core, 64 GB RAM, 1 Gige Nic
Minimum of 3 Storm worker nodes
Nimbus Node: Minimum 2 nimbus nodes , 4 core, 8 GB RAM
Hardware profile: disk I/O is not that important; more cores are better.
50 MB/sec per node with low to moderate complexity topology reading from Kafka and no external lookups . Medium-complexity and high-complexity topologies might have reduced throughput.
With a minimum 2 nimbus, 2 worker cluster, you can expect to run 100 MB/sec of low to medium complexity topology.
Further sizing can be done as follows. Formula: num_worker_nodes = desired_throughput(MB/sec) / 50
Recommendations for NiFi
NiFi is designed to take advantage of:
all the cores on a machine
all the network capacity
all the disk speed
many gigabytes of RAM (although usually not all) on a system
Hence, it is important that NiFi be running on dedicated nodes. Following are the recommended server and sizing specifications for NiFi:
Minimum of 3 nodes
8+ cores per node (more is better)
6+ disks per node (SSD or Spinning)
At least 8 GB
| If you want this sustained throughput… | Then provide this minimum hardware ... |
|---|---|
| 50 MB and thousands of events per second |
|
| 100 MB and tens of thousands of events per second |
|
| 200 MB and hundreds of thousands of events per second |
|
| 400 to 500 MB/sec and hundreds of thousands of events per second |
|