Cloudera Docs

Dynamic memory for Impala control service

Understand and configure memory allocation for Impala's control service queue to avoid errors and ensure consistent behavior across daemons with varying memory limits.

Condition

Impala daemon logs report errors when the control_service_queue_mem_limit is set to a smaller value:
impala-service-pool.cc: EndDataStream request on impala.DataStreamService from xx.xx.xx.xx:xxxx dropped due to backpressure.
The service queue contains xx items out of a maximum of xxxxxxxxxx; memory consumption is 50.05 MB.
In this message, the "maximum" value (e.g., xxxxxxxxxx or INT_MAX) indicates that the ControlService's RPC queue does not have an explicit limit based on the number of items. This queue's effective capacity is determined solely by its memory consumption, as controlled by the --control_service_queue_mem_limit parameter.

Cause

Before the implementation of IMPALA-10343, the default value of control_service_queue_mem_limit was 50MB. After IMPALA-10343 (starting from versions 7.1.7 SP2, 7.1.8, and 7.1.9), the value became dynamic and is now set to 1% of mem_limit.

The control service queue memory limit became dynamic, adjusting based on the daemon's memory. To avoid unintentional reductions in memory allocation, use the --control_service_queue_mem_limit_floor_bytes flag. This ensures consistent behavior for daemons with smaller memory limits, while increasing the limit for those with larger memory.

Remedy

  1. Log in to Cloudera Manager as an administrator.
  2. Navigate to Clusters > Impala > Configuration and search for Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve).
  3. Add the below value: 
    --control_service_queue_mem_limit=XXXMB
--control_service_queue_mem_limit_floor_bytes=XXMB (default is 50MB)
  4. The memory limit can be adjusted using the previously mentioned configurations to accommodate higher workloads. Ensure that you restart the Impala service for the changes to take effect.