Configuring the Hue Query Processor scan frequency
You can configure the frequency at which the Hue Query Processor fetches the data from a Hadoop Compatible File System by setting the time in milliseconds (ms) in the “hue.query-processor.event-pipeline.folder-scan-delay-millis” property in the Query Processor configurations.
hue.query-processor.event-pipeline.folder-scan-delay-millisproperty is 5 * 60 * 1000 ms (5 minutes). For a faster data refresh rate, you can set this value to less than 5 * 60 * 1000 ms (5 minutes). But this can result in higher storage read costs. However, if you want stronger performance, and a faster refresh rate is not a priority, then you can set it to a higher value to reduce storage read costs.
- Log in to the CDP web interface as an Administrator.
- Go to Data Hub Clusters and select your cluster.
- Click CM URL or CM-UI to open Cloudera Manager.
Go to Query Processor
Extra Configurations field:
and add the following line in the
"hue.query-processor.event-pipeline.folder-scan-delay-millis" : [***TIME-IN-MS***]
"hue.query-processor.event-pipeline.folder-scan-delay-millis" : 2000
- Click Save Changes.
- Restart the Query Processor service.