Deleting partitions

Due to a known issue, it is not recommended to delete a partition if it is associated with queues and the queues have capacities configured for that partition.

You must have created a partition in the cluster using YARN /add-node-labels API, and assigned this partition to some queues and set the partition capacities to queues.

Use Queue Manager to create a partition in the cluster using YARN /add-node-labels API and assign this partition to some queues and set the partition capacities to queues as follows:

Create the partition demo
Assign the partition to queues:
- yarn.scheduler.capacity.root.accessible-node-labels=”*”
- yarn.scheduler.capacity.root.default.accessible-node-labels=”demo”
Set the partition capacities to queues:
- yarn.scheduler.capacity.root.accessible-node-labels.demo.capacity=100
- yarn.scheduler.capacity.root.accessible-node-labels.demo.maximum-capacity=100
- yarn.scheduler.capacity.root.default.accessible-node-labels.demo.capacity=100
- yarn.scheduler.capacity.root.default.accessible-node-labels.demo.maximum-capacity=100

Now, to delete the partition and its related configs that are created above perform the following:

Unassign node labels from the queues set in step-2 above:
- yarn.scheduler.capacity.root.accessible-node-labels=”*”
- yarn.scheduler.capacity.root.default.accessible-node-labels=””
Delete the partition created in step-1 using the YARN /remove-node-labels API.

Now the system is left with residue configs that was set in step-3. And there is no way to remove these and causes various issues when trying to remove the residue configs.

Solution to remove the configs

YARN’s mutation API is designed to be simple and very generic, it only handles key-value pairs. The unwanted capacities can be removed by using an existing mutation API in YARN. This is performed by setting values to an empty (“”) string (or null) and then those configs will be removed from the in-memory config.

For example:

{
    "update-queue":
    [
        {
            "queue-name": "root",
            "params":
            {
                "entry":
                [
                    {
                        "key": "accessible-node-labels.label1.maximum-capacity",
                        "value": ""
                    },
                    {
                        "key": "accessible-node-labels",
                        "value": "*"
                    },
                    {
                        "key": "accessible-node-labels.label1.capacity",
                        "value": ""
                    }
                ]
            }
        },
{
            "queue-name": "root.a",
            "params":
            {
                "entry":
                [
                    {
                        "key": "accessible-node-labels.label1.maximum-capacity",
                        "value": ""
                    },
                    {
                        "key": "accessible-node-labels",
                        "value": ""
                    },
                    {
                        "key": "accessible-node-labels.label1.capacity",
                        "value": ""
                    }
                ]
            }
        },
        {
            "queue-name": "root.a.a1",
            "params":
            {
                "entry":
                [
                    {
                        "key": "accessible-node-labels.label1.maximum-capacity",
                        "value": ""
                    },
                    {
                        "key": "accessible-node-labels",
                        "value": ""
                    },
                    {
                        "key": "accessible-node-labels.label1.capacity",
                        "value": ""
                    }
                ]
            }
        },
        {
            "queue-name": "root.a.a2",
            "params":
            {
"entry":
                [
                    {
                        "key": "accessible-node-labels.label1.maximum-capacity",
                        "value": ""
                    },
                    {
                        "key": "accessible-node-labels",
                        "value": ""
                    },
                    {
                        "key": "accessible-node-labels.label1.capacity",
                        "value": ""
                    }
                ]
            }
        }
    ],
    "global": null,
    "global-updates": null
}

{
    "key": "accessible-node-labels.label1.capacity",
    "value": ""
}

and

{
    "key": "accessible-node-labels.label1.maximum-capacity",
    "value": ""
}

As seen in the above JSON, add null or empty string values for all the queues where you want to clear values from the config.