Hive Performance Tuning
Also available as:
PDF

Chapter 3. Configuring Memory Usage

Setting memory usage is simple, but very important. If configured incorrectly, jobs fail or run inefficiently.

Memory Settings for YARN

YARN takes into account all of the available computing resources on each machine in the cluster. Based on the available resources, YARN negotiates resource requests from applications running in the cluster, such as MapReduce. YARN then provides processing capacity to each application by allocating containers. A container is the basic unit of processing capacity in YARN, and is an encapsulation of resource elements (for example, memory, CPU, and so on).

In a Hadoop cluster, it is important to balance the memory (RAM) usage, processors (CPU cores), and disks so that processing is not constrained by any one of these cluster resources. Generally, allow for 2 containers per disk and per core for the best balance of cluster utilization.

Selecting YARN Memory

It is important to leave enough memory for system tasks to run. Then divide the remaining memory into containers and see what makes sense.

In yarn-site.xml, set yarn.nodemanager.resource.memory-mb to the memory that YARN uses:

  • For systems with 16GB of RAM or less, allocate one-quarter of the total memory for system use and the rest can be used by YARN.

  • For systems with more than 16GB of RAM, allocate one-eighth of the total memory for system use and the rest can be used by YARN.

For more information, see the Non-Ambari Cluster Installation Guide

Tez Memory Settings

Tez must use the majority of the YARN container to run query fragments, but it must also leave some space for system tasks to run.

For HDP 2.2 and earlier versions, the following settings in hive-site.xml work well:

  • Xmx= 80% of container size

  • Xms= 80% of container size

For HDP 2.3 and later versions, verify that the TezConfiguration.TEZ_CONTAINER_MAX_JAVA_HEAP_FRACTION property is set to .8.

The following figure shows where you can configure the size of Tez containers in the Hive web UI of Ambari:

Figure 3.1. Tez Container Configuration in Hive Web UI of Ambari


Hive Memory Settings

There are 2 key memory size settings for Hive: Map Join hash table size and bytes per reducer. However, these settings have implications on other optimizations.

Map Join Hash Table Size

hive.auto.convert.join.noconditionaltask.size is the maximum size of a table that is converted into a hash table for Map Join. For smaller clusters, one-third of the total memory is sufficient. If you have many dimension tables that join a fact table, these can add up to a large size. In that case, memory can be limited to one-half GB to be on the safe side.

Bytes Per Reducer

hive.exec.reducers.bytes.per.reducer is the size of data processed per reducer. Setting this to a large number results in less parallelism. Setting it to a smaller number results in more parallelism. How you set this depends on the application and the user priorities in terms of latency and throughput in your environment. Usually, the default value works well and for each application query this can be set once for a HiveServer2.