Modifying Index Parameters Using Flux (Advanced)

Some of the tuning indexing properties, for example the HDFS sync policy setting, can only be modified using Flux. However, if you manually change your Flux file, if you perform an upgrade, Ambari will overwrite all of your changes. Be sure to back up your Flux changes prior to performing an upgrade.

You can find the indexing Flux file at $METRON_HOME/flux/indexing/batch/remote.yaml.

Category Flux Property Flux Section Location Suggested Value
Kafka spout session.timeout.ms line 80, id: kafkaProps Kafka consumer client property
enable.auto.commit line 80, id: kafkaProps Kafka consumer client property
setPollTimeoutMs line 108, id: kafkaConfig Kafka consumer client property
setMaxUncommittedOffsets line 108, id: kafkaConfig Kafka consumer client property
setOffsetCommitPeriodMs line 108, id: kafkaConfig Kafka consumer client property
Output bolt hdfsSyncPolicy line 47, id: hdfsWriter See notes below about adding this prop

To modify index tuning properties, complete the following steps:

  1. Add a new setter to the hdfsWriter around line 56.

     53             -   name: "withRotationPolicy"
     54                 args:
     55                     - ref: "hdfsRotationPolicy
     56             -   name: "withSyncPolicy"
     57                 args:
     58                     - ref: "hdfsSyncPolicy
    

    Lines are 53-55 provided for context.

  2. Add an hdfsSyncPolicy after the hdfsRotationPolicy that appears on line 41:

     41     -   id: "hdfsRotationPolicy"
    ...
     45           - "${bolt.hdfs.rotation.policy.units}"
     46
     47     -   id: "hdfsSyncPolicy"
     48         className: "org.apache.storm.hdfs.bolt.sync.CountSyncPolicy"
     49         constructorArgs:
     50           -  100000