This document provides guidance from our experiences tuning the Apache Metron Storm topologies for maximum performance. You'll find suggestions for optimum configurations under a 1 Gbps load along with some guidance around the tooling we used to monitor and assess our throughput.

In the simplest terms, Hortonworks Cybersecurity Package powered by Apache Metron is a streaming architecture created on top of Kafka and three main types of Storm topologies: parsers, enrichment, and indexing. Each parser has its own topology and there is also a highly performant, specialized spout-only topology for streaming PCAP data to HDFS. We found that the architecture can be tuned almost exclusively through using a few primary Storm and Kafka parameters along with a few Metron-specific options. You can think of the data flow as being similar to water flowing through a pipe, and the majority of these options assist in tweaking the various pipe widths in the system.