Tuning Topologies
Also available as:
PDF

Tune Additional Parser Storm Settings

After the number of parser executors has been determined and thoroughly tested, you can set or modify the last remaining Storm parameters.

  1. Based on the capacity you've seen during testing, reduce the overall number of ackers.
    Alternatively, you can leave a single acker per worker as it will ensure that there are no messages sent between Storm workers over the network interface.
  2. Set the Max Spout Pending parameter such that the maximum number of unacked tuples in the topology is close to the Parser Executor capacity (for example, ~0.950).
    Setting the maximum number of unacked tupples to the Parser Executor capacity ensures that if there is a large spike in incoming events, the topology will not become overloaded. For example, to determine this value you can increase the producer events per second by a large amount and test various values for Max Spout Pending. The value can be set under the Storm settings of the relevant parser.
  3. Check the Parser Executor capacity.
    The Parser Executor capacity should not exceed ~0.950. Assuming the number of events generated by the producer is far greater than the capacity of the Parser topology, capacity is the only value that needs to be monitored in the Storm UI.
  4. Set the Error Writer Parallelism and Num Tasks values.
    Generally, since a small number of errors is expected, it can be set quite low. There should be more than 3 for redundancy but going as high as 1 per worker may also be of benefit for even distribution.