Estimating memory limits

Use Cloudera Manager Chart Builder to determine memory limits for resource pools.

By: Manish Maheshwari, Data Architect and Data Scientist at Cloudera, Inc.

To identify what is the appropriate minimum, maximum, and default memory limit for each of the pools, Cloudera recommends that you run real queries on your datasets, and then review the memory usage during peak periods of this test.

After you have run a set of your real queries on your datasets, use Cloudera Manager Chart Builder to view charts that reflect memory usage:

1. On the Cloudera Manager Admin Console home page, from the left navigation tree, select Charts > Chart Builder:

   
   
   

2. On the Chart Builder page, enter the following query, and then click Build Chart:

   select memory_per_node_peak from IMPALA_QUERIES where
   service_name="impala"

   Chart Builder then returns a chart that might look similar to the following example:

   
   
   

   This type of chart can be used to determine the memory limits for a particular pool. Note that most queries use less than 1.2 GB per node. See Resource Management for more information about Impala admission control.