PCAP Tuning
PCAP is a specialized topology that is a Spout-only topology. Both Kafka topic consumption and HDFS writing is done within a spout to avoid the additional network hop required if using an additional bolt.
General Storm topology properties
topology.workers=16 topology.ackers.executors: 0
Spout and Bolt Properties
kafkaSpout parallelism: 128 poll.timeout.ms=100 offset.commit.period.ms=30000 session.timeout.ms=39000 max.uncommitted.offsets=200000000 max.poll.interval.ms=10 max.poll.records=200000 receive.buffer.bytes=431072 max.partition.fetch.bytes=10000000 enable.auto.commit=false setMaxUncommittedOffsets=20000000 setOffsetCommitPeriodMs=30000 writerConfig withNumPackets=1265625 withMaxTimeMS=0 withReplicationFactor=1 withSyncEvery=80000 withHDFSConfig io.file.buffer.size=1000000 dfs.blocksize=1073741824