CLI options of the fs2cs conversion tool

Before generating the scheduler output files, you must understand the CLI options available for the fs2cs conversion tool. These options help you to complete the scheduler conversion.

Option Description
-c,--cluster-resource <arg>
Needs to be specified if maxResources is defined as percentages for any queue, otherwise this parameter can be omitted. The cluster resource setting is optional, but it can be necessary in some cases. In Fair Scheduler, you can define the maximum capacity of the queue as a percentage of the total cluster resource. Capacity scheduler does not accept a vector of capacities (work is ongoing under YARN-9936), therefore, percentages are converted to absolute resources. The acceptable formats are:
--cluster-resource vcores=10,memory-mb=1024
--cluster-resource 1024 mb,10 vcores
-d,--dry-run

Performs a dry-run of the conversion. Outputs whether the conversion is possible or not.

-f,--fsconfig <arg>

Absolute path to a valid fair-scheduler.xml configuration file.

By default, yarn-site.xml contains the property which defines the path of fair-scheduler.xml. Therefore, the -f / --fsconfig settings are optional.

-h,--help

Displays the list of options

-o,--output-directory <arg>

Output directory for yarn-site.xml and capacity-scheduler.xml files. Must have write permission for the user who is running this script.

If -p or --print is specified, the xml files are emitted to the standard output, so the -o / --output-directory is ignored.

-p,--print

If defined, the converted configuration will only be emitted to the console.

If -p or --print is specified, the xml files are emitted to the standard output, so the -o / --output-directory is ignored.

-r,--rulesconfig <arg>

Optional parameter. If specified, should point to a valid path to the conversion rules file (property format).

-s, --skip-verification

It does not validate the converted Capacity Scheduler configuration. By default, the utility starts an internal Capacity Scheduler instance to see whether it can start up properly or not. This switch disables this behaviour.

-t,--no-terminal-rule-check

Disables checking whether a placement rule is terminal to maintain backward compatibility with configs that were made before YARN-8967.

By default, Fair Scheduler performs a strict check of whether a placement rule is terminal or not. This means that if you use a <reject> rule which is followed by a <specified> rule, then this is not allowed, because the latter is unreachable. However, before YARN-8967, Fair Scheduler was more lenient and allowed certain sequence of rules that are no longer valid. Inside the tool, a Fair Scheduler instance is instantiated to read and parse the allocation file. To have Fair Scheduler accept such configurations, the -t or --no-terminal-rule-check argument must be supplied to avoid the Fair Scheduler instance throwing an exception.

-y,--yarnsiteconfig <arg>

Path to a valid yarn-site.xml configuration file.

Option Description
-c,--cluster-resource <arg>
Needs to be specified if maxResources is defined as percentages for any queue, otherwise this parameter can be omitted. The cluster resource setting is optional, but it can be necessary in some cases. In Fair Scheduler, you can define the maximum capacity of the queue as a percentage of the total cluster resource. Capacity scheduler does not accept a vector of capacities (work is ongoing under YARN-9936), therefore, percentages are converted to absolute resources. The acceptable formats are:
--cluster-resource vcores=10,memory-mb=1024
--cluster-resource 1024 mb,10 vcores
-d,--dry-run

Performs a dry-run of the conversion. Outputs whether the conversion is possible or not.

-f,--fsconfig <arg>

Absolute path to a valid fair-scheduler.xml configuration file.

By default, yarn-site.xml contains the property which defines the path of fair-scheduler.xml. Therefore, the -f / --fsconfig settings are optional.

-h,--help

Displays the list of options

-o,--output-directory <arg>

Output directory for yarn-site.xml and capacity-scheduler.xml files. Must have write permission for the user who is running this script.

If -p or --print is specified, the xml files are emitted to the standard output, so the -o / --output-directory is ignored.

-p,--print

If defined, the converted configuration will only be emitted to the console.

If -p or --print is specified, the xml files are emitted to the standard output, so the -o / --output-directory is ignored.

-pc,-percentage

By default the fs2cs conversion utility converts into weight mode. Using -pc you can change it to relative (percentage) mode. Note that there are some scheduler transition limitations that are resolved in weight mode but not in relative (percentage) mode.

Relative mode can be considered the “legacy” mode of Capacity Scheduler, where capacities are expressed in percentages.

-r,--rulesconfig <arg>

Optional parameter. If specified, should point to a valid path to the conversion rules file (property format).

-s, --skip-verification

It does not validate the converted Capacity Scheduler configuration. By default, the utility starts an internal Capacity Scheduler instance to see whether it can start up properly or not. This switch disables this behaviour.

-t,--no-terminal-rule-check

Disables checking whether a placement rule is terminal to maintain backward compatibility with configs that were made before YARN-8967.

By default, Fair Scheduler performs a strict check of whether a placement rule is terminal or not. This means that if you use a <reject> rule which is followed by a <specified> rule, then this is not allowed, because the latter is unreachable. However, before YARN-8967, Fair Scheduler was more lenient and allowed certain sequence of rules that are no longer valid. Inside the tool, a Fair Scheduler instance is instantiated to read and parse the allocation file. To have Fair Scheduler accept such configurations, the -t or --no-terminal-rule-check argument must be supplied to avoid the Fair Scheduler instance throwing an exception.

-y,--yarnsiteconfig <arg>

Path to a valid yarn-site.xml configuration file.