Isolating workloads with Impala executor groups

Isolate heavy Impala workloads by dedicating hosts to an executor group, mapping the group to an admission control resource pool, and routing queries to that pool.

This example shows how to dedicate five Impala daemon hosts to extract, transform, and load (ETL) jobs to run them on separate hardware from other queries.

Impala assigns an executor group to a group set whose name is the part of the group name before the hyphen. For example, the etl-g1 executor group belongs to the etl group set. You map the group set to an admission control resource pool named after the set: root.etl.

  • You must have an Impala service managed by Cloudera Manager.
  • You must have five Impala daemon instances to dedicate to the executor group.
  1. Create an admission control resource pool that maps to the executor group set.
    You do not create the executor group set itself. The set name is the part of the executor group name before the hyphen. Select the etl set name and name the pool after it. In step b, you use the same name in the etl-g1 executor group name. Queries routed to the set then use the admission limits of this pool.
    1. In Cloudera Manager, go to Clusters > Impala Admission Control Configuration.
    2. Click Create Resource Pool.
    3. In the Pool Name field, enter etl, which creates the root.etl pool.
      The following screenshot shows the pool name in the Create Resource Pool modal:
      Figure 1. Create Resource Pool modal

      The Create Resource Pool modal with etl entered after the root prefix in the Pool Name field.
    4. Click Create.
    5. Refresh the Impala dynamic resource pool configuration.
  2. Dedicate hardware nodes to an executor group by using a role group.
    Cloudera recommends using Cloudera Manager role groups to dedicate hardware. The role group applies the executor group name to every Impala daemon instance moved into the group.
    1. In Cloudera Manager, go to Impala > Instances > Role Groups.
    2. Click Create Role Group.
    3. In the Group Name field, enter Dedicated ETL Executors.
    4. From the Role Type drop-down list, select Impala Daemon.
    5. From the Copy From drop-down list, select Impala Daemon Default Group.
      The following screenshot shows the completed Create Role Group modal:
      Figure 2. Create Role Group dialog

      The Create Role Group modal with Dedicated ETL Executors entered as the group name, Impala Daemon as the role type, and Impala Daemon Default Group as the group to copy from.
    6. Click Create.
    7. On the Dedicated ETL Executors role group page, click Actions > Configuration.
      The following screenshot shows the Actions menu for the new role group:
      Figure 3. Actions menu for the Dedicated ETL Executors role group

      The Role Groups page for Dedicated ETL Executors with the Actions menu open and the Configuration option highlighted.
    8. Search for the impalad_executor_group property.
    9. In the Impala Daemon Executor Group field, enter etl-g1.
      The following screenshot shows the property value for the role group:
      Figure 4. Impala Daemon Executor Group property

      The configuration page for the Dedicated ETL Executors role group with the Impala Daemon Executor Group property set to etl-g1.
    10. Click Save Changes.
    11. On the Role Groups page, select Impala Daemon Default Group.
    12. Select the five Impala daemon instances to dedicate to ETL workloads.
    13. Click Move Selected Instances To Group.
    14. From the list of role groups, select Dedicated ETL Executors.
      The following screenshot shows the selected instances and the destination role group:
      Figure 5. Moving Impala daemon instances to the Dedicated ETL Executors role group

      The Role Groups page for Impala Daemon Default Group with five hosts selected and the Move Selected Instances To Group list showing Dedicated ETL Executors.
    15. Restart the Impala service to apply the executor group configuration.
    • After the restart, the five Impala daemon instances are listed in the Dedicated ETL Executors role group and form the etl-g1 executor group.
    • Ensure that every executor in the cluster belongs to a custom executor group. If other executors do not belong to an executor group, repeat Step 1 and Step 2 for those executors. For example, assign the root.general resource pool to the general-g1 executor group. Otherwise, queries might run on executors in the default group instead of etl-g1, even when you set the request pool to etl.
  3. Direct query traffic to the dedicated pool.
    Before you submit a query, set the request pool in your Impala client to the executor group set name, without the root. prefix.
    1. In your Impala client, run the following statement:
      SET REQUEST_POOL=etl;
    2. Run the ETL query, for example:
      INSERT INTO massive_daily_load_table SELECT * FROM raw_massive_data;

Impala schedules queries submitted to the etl pool on the executors in the etl-g1 executor group.

Verify that the query ran. In the Impala daemon web UI, open the query profile. The profile displays the following lines:

Request Pool: root.etl
Executor Group: etl-g1