PCAP Tuning
PCAP is a specialized topology that is a Spout-only topology. Both Kafka topic consumption and HDFS writing is done within a spout to avoid the additional network hop required if using an additional bolt.
General Storm topology properties
``` topology.workers=16 topology.ackers.executors: 0 ``` +__Spout and Bolt properties__ +``` +kafkaSpout + parallelism: 128 + poll.timeout.ms=100 + offset.commit.period.ms=30000 + session.timeout.ms=39000 + max.uncommitted.offsets=200000000 + max.poll.interval.ms=10 + max.poll.records=200000 + receive.buffer.bytes=431072 + max.partition.fetch.bytes=10000000 + enable.auto.commit=false + setMaxUncommittedOffsets=20000000 + setOffsetCommitPeriodMs=30000 + +writerConfig + withNumPackets=1265625 + withMaxTimeMS=0 + withReplicationFactor=1 + withSyncEvery=80000 + withHDFSConfig + io.file.buffer.size=1000000 + dfs.blocksize=1073741824 +``` +