Chapter 3. Component Tuning Properties
There are a number of services that you can use to tune the performance of your Metron cluster. These services include Kafka, Storm, and HDFS. Within these services, you can modify parsers, enrichment, and indexing (Elasticsearch or Solr).
When you consider tuning your HCP architecture, it is important to note where you can modify settings. For example, Storm gives you the ability to independently set tasks in executors for parser topologies. This is important if you want to set the number of tasks higher than the number of executors to accommodate for future performance tuning and rebalancing without the need to bring down your topologies. However, for enrichment and indexing topologies, HCP uses Flux, and there is no method for specifying the number of tasks from the number of executors in Flux. By default, the number of tasks equals the number of executors.
The following lists the major properties for each service that you can modify to tune your cluster:
Kafka
Number of partitions
Storm
Kafka spout
Polling frequency
Polling timeouts
Offset commit period
Maximum uncommitted offsets
Number of workers (OS processes)
Number of executors (threads in a process)
Number of ackers
Maximum spout pending
Spout and bolt parallelism
HDFS
Replication factor
Indexing
Elasticsearch
Solr