Resource pool design

Allocating memory across resource pools is an important performance tuning task.

By: Manish Maheshwari, Data Architect and Data Scientist at Cloudera, Inc.

With admission control enabled, the most common approach used to allocate the total amount of Impala memory across all of the pools is to create a pool for each tenant as shown in the following diagrams:

However this strategy is usually counterproductive because of the following reasons:

• Unused memory in one tenant cannot be used by other tenants.
• Busy tenants queue up queries in admission control, causes an overall slowdown in query execution.
• Small tenants that run large queries spill to disk until the spill to disk limit is reached and the memory limit is reached.

Instead, Cloudera recommends that you design resource pools by query sizes as shown in the following diagram:

This method ensures that all memory allocated to Impala is optimally used. Also, by using this method, all users and groups have access to all the pools and users can choose the appropriate pool according to memory sizes.