Enable node label on a cluster to configure partition

You can configure partitions on a cluster by making configuration changes on the YARN ResourceManager host.

To enable partition on a cluster, make the following configuration changes on the YARN ResourceManager host.

  1. Create a Label Directory in HDFS

    Use the following commands to create a "node-labels" directory in which to store the Node Labels in HDFS.

    sudo su hdfs
    hadoop fs -mkdir -p /yarn/node-labels
    hadoop fs -chown -R yarn:yarn /yarn
    hadoop fs -chmod -R 700 /yarn

    -chmod -R 700 specifies that only the YARN user can access the node-labels directory.

    You can then use the following command to confirm that the directory was created in HDFS.

    hadoop fs -ls /yarn

    The new node label directory should appear in the list returned by the following command. The owner should be yarn, and the permission should be drwx.

    Found 1 items
    drwx------ - yarn yarn 0 2014-11-24 13:09 /yarn/node-labels

    Use the following commands to create a /user/<user_name> directory that is required by the distributed shell.

    hadoop fs -mkdir -p /user/<user_name>
    hadoop fs -chown -R yarn:yarn /user/<user_name>
    hadoop fs -chmod -R 700 /user/<user_name>
  2. In Cloudera Manager, select the YARN service.
  3. Click the Configuration tab.
  4. Search for YARN Service Advanced Configuration.
  5. In YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml add the following:
    • Set the following property to enable partition:
      Name: yarn.node-labels.enabled
      Value: true
    • Set the following property to reference the HDFS partition directory
      Name: yarn.node-labels.fs-store.root-dir
      Value: hdfs://:/

      For example,

      Name: yarn.node-labels.fs-store.root-dir
      Value: hdfs://node-1.example.com:8020/yarn/node-labels/ 
  6. Start or Restart the YARN ResourceManager.