SCHEDULE_RANDOM_REPLICA Query Option (CDH 5.7 or higher only)

The SCHEDULE_RANDOM_REPLICA query option fine-tunes the algorithm for deciding which host processes each HDFS data block. It only applies to tables and partitions that are not enabled for the HDFS caching feature.

Type: Boolean; recognized values are 1 and 0, or true and false; any other value interpreted as false

Default: false

Added in: CDH 5.7.0 (Impala 2.5.0)

Usage notes:

In the presence of HDFS cached replicas, Impala randomizes which host processes each cached data block. To ensure that HDFS data blocks are cached on more than one host, use the WITH REPLICATION clause along with the CACHED IN clause in a CREATE TABLE or ALTER TABLE statement. Specify a replication value greater than or equal to the HDFS block replication factor.

The SCHEDULE_RANDOM_REPLICA query option applies to tables and partitions that do not use HDFS caching. By default, Impala estimates how much work each host has done for the query, and selects the host that has the lowest workload. This algorithm is intended to reduce CPU hotspots arising when the same host is selected to process multiple data blocks, but hotspots might still arise for some combinations of queries and data layout. When the SCHEDULE_RANDOM_REPLICA option is enabled, Impala further randomizes the scheduling algorithm for non-HDFS cached blocks, which can further reduce the chance of CPU hotspots.

Related information:

Using HDFS Caching with Impala (CDH 5.1 or higher only), Avoiding CPU Hotspots for HDFS Cached Data