Cluster Management
You can manage
resources for the applications running on your cluster by
allocating
resources through scheduling, limiting CPU usage by configuring cgroups, partitioning the
cluster into
subclusters
using node labels, and launching applications on Docker
containers.
Using Scheduling to Allocate Resources You can allocate CPU, GPU, and memory among users and groups in a Hadoop cluster. You can use scheduling to allocate the best possible nodes for application containers.GPU support for Docker You can use GPUs in big data applications such as machine learning, data analytics, and genome sequencing. Docker containerization makes it easier for you to package and distribute applications. You can enable GPU support when using YARN on Docker containers.Limit CPU Usage with Cgroups You can use cgroups to limit CPU usage in a Hadoop Cluster. Managing Device Plug-ins (Technical Preview) Hortonworks Data Platform (HDP) 3.x releases before 3.1.4 supported the YARN GPU/FPGA device plug-in by writing custom code. Implementing such a device plug-in required knowledge of YARN integration and internal configuration related to NodeManager.Partition a Cluster Using Node Labels You can use Node labels to partition a cluster into sub-clusters so that jobs run on nodes with specific characteristics.