Tune Additional Batch Indexing Storm Settings
After the number of executors has been determined and thoroughly tested, you can set or modify the last remaining Storm parameters.
-
Based on the capacity you've seen during testing, reduce the overall number of
ackers.
Alternatively, you an leave a single acker per worker as it will ensure that there are no messages sent between Storm workers over the network interface.
-
Set the Max Spout Pending parameter such that the maximum
number of unacked tuples in the topology is close to the Parser Executor capacity
(for example, ~0.950).
If this is the case then it can be assured that if there is a large spike in incoming events, the topology will not become overloaded. An example approach to determine this value would be to increase the producer events per second by a large amount and test various values for Max Spout Pending. The value can be set under the Storm settings of the relevant Parser.
vi ~/enrichment.properties
##### Storm ##### indexing.workers=3 indexing.acker.executors=3 topology.worker.childopts= topology.auto-credentials=[] topology.max-spout.pending=
-
Check the Executor capacity.
The executor capacity should not exceed ~0.950. Assuming the number of events generated by the producer is far greater than the capacity of the Parser topology, capacity is the only value that needs to be monitored in the Storm UI.
-
If you need to increase the Error Writer Num Executors
value, you can directly modify the Flux file and include the "parallelism" parameter
under the appropriate Storm Bolt declarations.
While the product has been rebranded to Cloudera Cybersecurity Platform (CCP), the directory currently is
hcp
.sudo vi /usr/hcp/current/metron/flux/indexing/batch/remote.yaml
id: "indexingErrorOutputBolt" className: "org.apache.metron.writer.bolt.BulkMesageWriterBolt" constructorArgs: - "${kafka.zk}" configMethods: - name: "withMessageWriter" args: - ref: "KafkaWriter" parallelism: 3
Generally, since a small number of errors is expected, theError Writer Num Executors
value does not need to be increased.