Pod placement policy and rack awareness in CDW Private Cloud

In Cloudera Data Warehouse (CDW), Kubernetes node affinity rules are based on the rack topology defined in Cloudera Manager. If sufficient resources are available, CDW prefers the same racks for scheduling HiveServer2 (HS2), executor, and coordinator pods in Impala and Hive Virtual Warehouses.

If two executors are present on different nodes in different racks and are connected using a network interconnect, then this can cause significant performance overhead. It is a best practice to schedule executors into the same rack.

On Embedded Container Service (ECS), the Kubernetes nodes are tagged with the label `rack=[***RACK-ID***]`. You can specify a 63-character long rack ID. Only alphanumeric characters are supported. On OpenShift Container Platform (OCP), you can tag the nodes yourself as needed.

Every time CDW needs to schedule new executor pods, the executor pods are scheduled next to the existing Hive or Impala executors, coordinators, and HS2. In other words, HS2 and Impala coordinators have an affinity to the first executor group within a particular rack.

In the case of race conditions where multiple executors get scheduled at once but in different racks, the rack with the highest number of executors attracts the rest of the executors. However, it is still possible to schedule executor groups across multiple racks in case of race conditions or if the first rack is full and no executors can fit into it.