Modifying Index Parameters Using Flux (Advanced)
Some of the tuning indexing properties, for example the HDFS sync policy setting, can only be modified using Flux. However, if you manually change your Flux file, if you perform an upgrade, Ambari will overwrite all of your changes. Be sure to back up your Flux changes prior to performing an upgrade.
You can find the indexing Flux file at
$METRON_HOME/flux/indexing/batch/remote.yaml
.
Category | Flux Property | Flux Section Location | Suggested Value |
---|---|---|---|
Kafka spout | session.timeout.ms |
line 80, id: kafkaProps |
Kafka consumer client property |
enable.auto.commit |
line 80, id: kafkaProps |
Kafka consumer client property | |
setPollTimeoutMs |
line 108, id: kafkaConfig |
Kafka consumer client property | |
setMaxUncommittedOffsets |
line 108, id: kafkaConfig |
Kafka consumer client property | |
setOffsetCommitPeriodMs |
line 108, id: kafkaConfig |
Kafka consumer client property | |
Output bolt | hdfsSyncPolicy |
line 47, id: hdfsWriter |
See notes below about adding this prop |
To modify index tuning properties, complete the following steps:
-
Add a new setter to the
hdfsWriter
around line 56.53 - name: "withRotationPolicy" 54 args: 55 - ref: "hdfsRotationPolicy 56 - name: "withSyncPolicy" 57 args: 58 - ref: "hdfsSyncPolicy
Lines are 53-55 provided for context.
-
Add an
hdfsSyncPolicy
after thehdfsRotationPolicy
that appears on line 41:41 - id: "hdfsRotationPolicy" ... 45 - "${bolt.hdfs.rotation.policy.units}" 46 47 - id: "hdfsSyncPolicy" 48 className: "org.apache.storm.hdfs.bolt.sync.CountSyncPolicy" 49 constructorArgs: 50 - 100000