Enable node label on a cluster to configure partition
You can configure partitions on a cluster by making configuration changes on the YARN ResourceManager host.
To enable partition on a cluster, make the following configuration changes on the YARN ResourceManager host.
- Create a Label Directory in HDFS
Use the following commands to create a "node-labels" directory in which to store the Node Labels in HDFS.
sudo su hdfs hadoop fs -mkdir -p /yarn/node-labels hadoop fs -chown -R yarn:yarn /yarn hadoop fs -chmod -R 700 /yarn
-chmod -R 700
specifies that only the YARN user can access the node-labels directory.You can then use the following command to confirm that the directory was created in HDFS.
hadoop fs -ls /yarn
The new node label directory should appear in the list returned by the following command. The owner should be
yarn
, and the permission should bedrwx
.Found 1 items drwx------ - yarn yarn 0 2014-11-24 13:09 /yarn/node-labels
Use the following commands to create a
/user/<user_name>
directory that is required by the distributed shell.hadoop fs -mkdir -p /user/<user_name> hadoop fs -chown -R yarn:yarn /user/<user_name> hadoop fs -chmod -R 700 /user/<user_name>
- In Cloudera Manager, select the YARN service.
- Click the Configuration tab.
- Search for YARN Service Advanced Configuration.
- In YARN Service Advanced Configuration Snippet (Safety Valve) for
yarn-site.xml add the following:
- Set the following property to enable partition:
Name: yarn.node-labels.enabled Value: true
- Set the following property to reference the HDFS partition directory
Name: yarn.node-labels.fs-store.root-dir Value: hdfs://:/
For example,
Name: yarn.node-labels.fs-store.root-dir Value: hdfs://node-1.example.com:8020/yarn/node-labels/
- Set the following property to enable partition:
- Start or Restart the YARN ResourceManager.