Enabling node labels on a cluster to configure partition
You can configure partitions on a cluster by making configuration changes on the YARN ResourceManager host.
-
Create a Label Directory in HDFS.
-
Use the following commands to create a
node-labels
directory in which to store the Node Labels in HDFS:sudo su hdfs hadoop fs -mkdir -p /yarn/node-labels hadoop fs -chown -R yarn:yarn /yarn hadoop fs -chmod -R 700 /yarn
Where
-chmod -R 700
specifies that only theyarn
user can access thenode-labels
directory. -
Use the following command to confirm that the directory was created in
HDFS:
hadoop fs -ls /yarn
The newnode-labels
directory should appear. The owner should beyarn
, and the permission should bedrwx
:Found 1 items drwx------ - yarn yarn 0 2014-11-24 13:09 /yarn/node-labels
-
Use the following commands to create a
- In Cloudera Manager, navigate to YARN > Configuration.
- Search for node label.
- Find the Node Labels property and ensure that it is selected (enabled).
-
Find the Node Labels Directory property and set its value to
reference the HDFS node label directory.
For example:
hdfs://node-1.example.com:8020/yarn/node-labels/
.HDFS paths are automatically translated, however, any other file system requires a full path to the Node Labels Directory.
The default value of the Node Labels Directory property is
/yarn/node-labels
. - Start or Restart the YARN ResourceManager.