Balancer
Advanced
Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml
- Description
- For advanced use only. A string to be inserted into hdfs-site.xml for this role only.
- Related Name
- Default Value
- API Name
-
balancer_config_safety_valve
- Required
- false
Java Configuration Options for Balancer
- Description
- These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here. Note: When CM version is 6.3.0 or greater, {{JAVA_GC_ARGS}} will be replaced by JVM Garbage Collection arguments based on the runtime Java JVM version.
- Related Name
- Default Value
- API Name
-
balancer_java_opts
- Required
- false
Balancer Logging Advanced Configuration Snippet (Safety Valve)
- Description
- For advanced use only, a string to be inserted into log4j.properties for this role only.
- Related Name
- Default Value
- API Name
-
log4j_safety_valve
- Required
- false
Logs
Balancer Log Directory
- Description
- Directory where Balancer will place its log files.
- Related Name
- Default Value
- /var/log/hadoop-hdfs
- API Name
-
balancer_log_dir
- Required
- false
Balancer Logging Threshold
- Description
- The minimum log level for Balancer logs
- Related Name
- Default Value
- INFO
- API Name
-
log_threshold
- Required
- false
Balancer Maximum Log File Backups
- Description
- The maximum number of rolled log files to keep for Balancer logs. Typically used by log4j or logback.
- Related Name
- Default Value
- 10
- API Name
-
max_log_backup_index
- Required
- false
Balancer Max Log Size
- Description
- The maximum size, in megabytes, per log file for Balancer logs. Typically used by log4j or logback.
- Related Name
- Default Value
- 200 MiB
- API Name
-
max_log_size
- Required
- false
Monitoring
Enable Configuration Change Alerts
- Description
- When set, Cloudera Manager will send alerts when this entity's configuration changes.
- Related Name
- Default Value
- false
- API Name
-
enable_config_alerts
- Required
- false
Rules to Extract Events from Log Files
- Description
- This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.. Each rule has some or all of the following fields:
alert
- whether or not events generated from this rule should be promoted to alerts. A value of "true" will cause alerts to be generated. If not specified, the default is "false".rate
(mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more thanrate
matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.periodminutes
- the number of minutes during which the publisher will only publishrate
events or fewer. If not specified, the default is one minutethreshold
- apply this rule only to messages with this log4j severity level or above. An example is "WARN" for warning level messages or higher.content
- match only those messages for which contents match this regular expression.exceptiontype
- match only those messages that are part of an exception message. The exception type must match this regular expression.
{"alert": false, "rate": 10, "exceptiontype": "java.lang.StringIndexOutOfBoundsException"}
This rule sends events to Cloudera Manager for everyStringIndexOutOfBoundsException
, up to a maximum of 10 every minute.{"alert": false, "rate": 1, "periodminutes": 1, "exceptiontype": ".*"}, {"alert": true, "rate": 1, "periodminutes": 1, "threshold":"ERROR"}
In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
- Related Name
- Default Value
- version: 0, rules: [ alert: false, rate: 1, periodminutes: 1, threshold: FATAL , alert: false, rate: 0, threshold: WARN, content: .* is deprecated. Instead, use .* , alert: false, rate: 0, threshold: WARN, content: .* is deprecated. Use .* instead , alert: false, rate: 0, exceptiontype: java.io.IOException , alert: false, rate: 0, exceptiontype: java.net.SocketException , alert: false, rate: 0, exceptiontype: java.net.SocketClosedException , alert: false, rate: 0, exceptiontype: java.io.EOFException , alert: false, rate: 0, exceptiontype: java.nio.channels.CancelledKeyException , alert: false, rate: 1, periodminutes: 2, exceptiontype: .* , alert: false, rate: 0, threshold: WARN, content: Unknown job [^ ]+ being deleted.* , alert: false, rate: 0, threshold: WARN, content: Error executing shell command .+ No such process.+ , alert: false, rate: 0, threshold: WARN, content: .*attempt to override final parameter.+ , alert: false, rate: 0, threshold: WARN, content: [^ ]+ is a deprecated filesystem name. Use.* , alert: false, rate: 1, periodminutes: 1, threshold: WARN ]
- API Name
-
log_event_whitelist
- Required
- false
Other
Dispatcher Threads
- Description
- Thread pool size for dispatching block moves.
- Related Name
-
dfs.balancer.dispatcherThreads
- Default Value
- 200
- API Name
-
dfs_balancer_dispatcher_threads
- Required
- false
Minimum Block Size
- Description
- Smallest block to consider for moving.
- Related Name
-
dfs.balancer.getBlocks.min-block-size
- Default Value
- 10 MiB
- API Name
-
dfs_balancer_get_blocks_min_block_size
- Required
- false
Block Metadata Batch Size
- Description
- Amount of block metadata to retrieve at a time.
- Related Name
-
dfs.balancer.getBlocks.size
- Default Value
- 2 GiB
- API Name
-
dfs_balancer_get_blocks_size
- Required
- false
Maximum Concurrent Moves
- Description
- Number of block moves to permit in parallel.
- Related Name
-
dfs.datanode.balance.max.concurrent.moves
- Default Value
- 50
- API Name
-
dfs_balancer_max_concurrent_moves
- Required
- false
Maximum Iteration Size
- Description
- Maximum amount of data to move per node in each iteration of the balancer.
- Related Name
-
dfs.balancer.max-size-to-move
- Default Value
- 10 GiB
- API Name
-
dfs_balancer_max_size_to_move
- Required
- false
Mover Threads
- Description
- Thread pool size for executing block moves.
- Related Name
-
dfs.balancer.moverThreads
- Default Value
- 1000
- API Name
-
dfs_balancer_mover_threads
- Required
- false
Excluded Hosts
- Description
- Hosts to exclude from the balancing process.
- Related Name
- Default Value
- API Name
-
rebalancer_exclude_hosts
- Required
- false
Included Hosts
- Description
- Hosts to include in the balancing process (uses all, if none specified).
- Related Name
- Default Value
- API Name
-
rebalancer_include_hosts
- Required
- false
Source Hosts
- Description
- Manual override to specify which DataNodes should be used to off-load data to less full nodes.
- Related Name
- Default Value
- API Name
-
rebalancer_source_hosts
- Required
- false
Rebalancing Threshold
- Description
- The percentage deviation from average utilization, after which a node will be rebalanced. (for example, '10.0' for 10%).
- Related Name
- Default Value
- 10.0 %
- API Name
-
rebalancer_threshold
- Required
- false
Rebalancing Policy
- Description
- The policy that should be used to rebalance HDFS storage. The default DataNode policy balances the storage at the DataNode level. This is similar to the balancing policy from prior releases. The BlockPool policy balances the storage at the block pool level as well as at the DataNode level. The BlockPool policy is relevant only to a Federated HDFS service.
- Related Name
- Default Value
- DataNode
- API Name
-
rebalancing_policy
- Required
- false
Resource Management
Java Heap Size of Balancer in Bytes
- Description
- Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.
- Related Name
- Default Value
- 1 GiB
- API Name
-
balancer_java_heapsize
- Required
- false
Suppressions
Suppress Parameter Validation: Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_balancer_config_safety_valve
- Required
- true
Suppress Parameter Validation: Java Configuration Options for Balancer
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Balancer parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_balancer_java_opts
- Required
- true
Suppress Parameter Validation: Balancer Log Directory
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Balancer Log Directory parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_balancer_log_dir
- Required
- true
Suppress Configuration Validator: CDH Version Validator
- Description
- Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_cdh_version_validator
- Required
- true
Suppress Parameter Validation: Balancer Logging Advanced Configuration Snippet (Safety Valve)
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Balancer Logging Advanced Configuration Snippet (Safety Valve) parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_log4j_safety_valve
- Required
- true
Suppress Parameter Validation: Rules to Extract Events from Log Files
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_log_event_whitelist
- Required
- true
Suppress Parameter Validation: Excluded Hosts
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Excluded Hosts parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_rebalancer_exclude_hosts
- Required
- true
Suppress Parameter Validation: Included Hosts
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Included Hosts parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_rebalancer_include_hosts
- Required
- true
Suppress Parameter Validation: Source Hosts
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Source Hosts parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_rebalancer_source_hosts
- Required
- true