Configure Impala Daemon to spill to HDFS
Impala occasionally needs to use persistent storage for writing intermediate files during large sorts, joins, aggregations, or analytic function operations. If your workload results in large volumes of intermediate data being written, it is recommended to configure the heavy spilling queries to use a remote storage location rather than the local one. The advantage of using remote storage for scratch space is that it is elastic and can handle any amount of spilling.
Before you begin
- Identify the HDFS scratch directory where you want your new Impala to write the temporary data.
- Identify the IP address, host name, or service identifier of HDFS.
- Identify the port number of the HDFS NameNode (if not-default).
- Configure Impala to write temporary data to disk during query processing.
Configuring the Start-up Option in Impala daemon
You can use the Impalad start option
scratch_dirs to specify the
locations of the intermediate files.
Use the following format for this start up option:
--scratch_dirs="hdfs://authority/path(:max_bytes), local_buffer_dir (,local_dir…)"
hdfs://authority/pathis the remote directory.
- authority may include
- max_bytes is optional.
Using the above format:
- You can specify only one remote directory. When you configure a remote directory, you must specify a local buffer directory as the buffer. However you can use multiple local directories with the remote directory. If you specify multiple local directories, the first local directory would be used as the local buffer directory.
- If you configure both remote and local directories, the remote directory is only used when the local directories are fully utilized.
- The size of a remote intermediate file could affect the query performance, and the
value can be set by
--remote_tmp_file_size=sizein the start-up option. The default size of a remote intermediate file is 16MB while the maximum is 512MB.
- A HDFS scratch dir with one local buffer dir, file size 64MB. The space of HDFS scratch
dir is limited to 300G.
- A HDFS scratch dir with one local buffer dir limited to 512MB, and one local dir
limited to 10GB. The space of HDFS scratch dir is limited to 300G. The HDFS NameNode uses
its default port (8020).
- A HDFS scratch dir with one local buffer dir, and multiple prioritized local dirs. The
space of HDFS scratch dir is unlimited. The HDFS service identifier is
max_bytes is optional, it is highly recommended to configure for
spilling to HDFS because the HDFS cluster space is limited.