HDFS Properties in Cloudera Runtime 7.0.x

Role groups:

Balancer

Categories:

Advanced

Display Name Description Related Name Default Value API Name Required
Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only. A string to be inserted into hdfs-site.xml for this role only. balancer_config_safety_valve false
Java Configuration Options for Balancer These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here. Note: When CM version is 6.3.0 or greater, {{JAVA_GC_ARGS}} will be replaced by JVM Garbage Collection arguments based on the runtime Java JVM version. balancer_java_opts false
Balancer Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false

Logs

Display Name Description Related Name Default Value API Name Required
Balancer Log Directory Directory where Balancer will place its log files. /var/log/hadoop-hdfs balancer_log_dir false
Balancer Logging Threshold The minimum log level for Balancer logs INFO log_threshold false
Balancer Maximum Log File Backups The maximum number of rolled log files to keep for Balancer logs. Typically used by log4j or logback. 10 max_log_backup_index false
Balancer Max Log Size The maximum size, in megabytes, per log file for Balancer logs. Typically used by log4j or logback. 200 MiB max_log_size false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Rules to Extract Events from Log Files This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.. Each rule has some or all of the following fields:
  • alert - whether or not events generated from this rule should be promoted to alerts. A value of "true" will cause alerts to be generated. If not specified, the default is "false".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is "WARN" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.
Example:
  • {"alert": false, "rate": 10, "exceptiontype": "java.lang.StringIndexOutOfBoundsException"}This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {"alert": false, "rate": 1, "periodminutes": 1, "exceptiontype": ".*"}, {"alert": true, "rate": 1, "periodminutes": 1, "threshold":"ERROR"}In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
version: 0, rules: [ alert: false, rate: 1, periodminutes: 1, threshold: FATAL , alert: false, rate: 0, threshold: WARN, content: .* is deprecated. Instead, use .* , alert: false, rate: 0, threshold: WARN, content: .* is deprecated. Use .* instead , alert: false, rate: 0, exceptiontype: java.io.IOException , alert: false, rate: 0, exceptiontype: java.net.SocketException , alert: false, rate: 0, exceptiontype: java.net.SocketClosedException , alert: false, rate: 0, exceptiontype: java.io.EOFException , alert: false, rate: 0, exceptiontype: java.nio.channels.CancelledKeyException , alert: false, rate: 1, periodminutes: 2, exceptiontype: .* , alert: false, rate: 0, threshold: WARN, content: Unknown job [^ ]+ being deleted.* , alert: false, rate: 0, threshold: WARN, content: Error executing shell command .+ No such process.+ , alert: false, rate: 0, threshold: WARN, content: .*attempt to override final parameter.+ , alert: false, rate: 0, threshold: WARN, content: [^ ]+ is a deprecated filesystem name. Use.* , alert: false, rate: 1, periodminutes: 1, threshold: WARN ] log_event_whitelist false

Other

Display Name Description Related Name Default Value API Name Required
Dispatcher Threads Thread pool size for dispatching block moves. dfs.balancer.dispatcherThreads 200 dfs_balancer_dispatcher_threads false
Minimum Block Size Smallest block to consider for moving. dfs.balancer.getBlocks.min-block-size 10 MiB dfs_balancer_get_blocks_min_block_size false
Block Metadata Batch Size Amount of block metadata to retrieve at a time. dfs.balancer.getBlocks.size 2 GiB dfs_balancer_get_blocks_size false
Maximum Concurrent Moves Number of block moves to permit in parallel. dfs.datanode.balance.max.concurrent.moves 50 dfs_balancer_max_concurrent_moves false
Maximum Iteration Size Maximum amount of data to move per node in each iteration of the balancer. dfs.balancer.max-size-to-move 10 GiB dfs_balancer_max_size_to_move false
Mover Threads Thread pool size for executing block moves. dfs.balancer.moverThreads 1000 dfs_balancer_mover_threads false
Excluded Hosts Hosts to exclude from the balancing process. rebalancer_exclude_hosts false
Included Hosts Hosts to include in the balancing process (uses all, if none specified). rebalancer_include_hosts false
Source Hosts Manual override to specify which DataNodes should be used to off-load data to less full nodes. rebalancer_source_hosts false
Rebalancing Threshold The percentage deviation from average utilization, after which a node will be rebalanced. (for example, '10.0' for 10%). 10.0 % rebalancer_threshold false
Rebalancing Policy The policy that should be used to rebalance HDFS storage. The default DataNode policy balances the storage at the DataNode level. This is similar to the balancing policy from prior releases. The BlockPool policy balances the storage at the block pool level as well as at the DataNode level. The BlockPool policy is relevant only to a Federated HDFS service. DataNode rebalancing_policy false

Resource Management

Display Name Description Related Name Default Value API Name Required
Java Heap Size of Balancer in Bytes Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx. 1 GiB balancer_java_heapsize false

Suppressions

Display Name Description Related Name Default Value API Name Required
Suppress Parameter Validation: Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter. false role_config_suppression_balancer_config_safety_valve true
Suppress Parameter Validation: Java Configuration Options for Balancer Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Balancer parameter. false role_config_suppression_balancer_java_opts true
Suppress Parameter Validation: Balancer Log Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Balancer Log Directory parameter. false role_config_suppression_balancer_log_dir true
Suppress Configuration Validator: Version Validator Whether to suppress configuration warnings produced by the Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: Balancer Logging Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Balancer Logging Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_log4j_safety_valve true
Suppress Parameter Validation: Rules to Extract Events from Log Files Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter. false role_config_suppression_log_event_whitelist true
Suppress Parameter Validation: Excluded Hosts Whether to suppress configuration warnings produced by the built-in parameter validation for the Excluded Hosts parameter. false role_config_suppression_rebalancer_exclude_hosts true
Suppress Parameter Validation: Included Hosts Whether to suppress configuration warnings produced by the built-in parameter validation for the Included Hosts parameter. false role_config_suppression_rebalancer_include_hosts true
Suppress Parameter Validation: Source Hosts Whether to suppress configuration warnings produced by the built-in parameter validation for the Source Hosts parameter. false role_config_suppression_rebalancer_source_hosts true

DataNode

Categories:

Advanced

Display Name Description Related Name Default Value API Name Required
DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only. A string to be inserted into hdfs-site.xml for this role only. datanode_config_safety_valve false
Java Configuration Options for DataNode These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here. Note: When CM version is 6.3.0 or greater, {{JAVA_GC_ARGS}} will be replaced by JVM Garbage Collection arguments based on the runtime Java JVM version. JAVA_GC_ARGS datanode_java_opts false
DataNode Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration. DATANODE_role_env_safety_valve false
Available Space Policy Balanced Preference Only used when the DataNode Volume Choosing Policy is set to Available Space. Controls what percentage of new block allocations will be sent to volumes with more available disk space than others. This setting should be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should be no reason to prefer that volumes with less available disk space receive more block allocations. dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction 0.75 dfs_datanode_available_space_balanced_preference true
Available Space Policy Balanced Threshold Only used when the DataNode Volume Choosing Policy is set to Available Space. Controls how much DataNode volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis. dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold 10 GiB dfs_datanode_available_space_balanced_threshold true
DataNode Volume Choosing Policy DataNode Policy for picking which volume should get a new block. The Available Space policy is only available starting with CDH 4.3. dfs.datanode.fsdataset.volume.choosing.policy org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy dfs_datanode_volume_choosing_policy true
Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties. hadoop_metrics2_safety_valve false
DataNode Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false
Heap Dump Directory Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role. oom_heap_dump_dir /tmp oom_heap_dump_dir false
Dump Heap When Out of Memory When set, generates heap dump file when java.lang.OutOfMemoryError is thrown. true oom_heap_dump_enabled true
Kill When Out of Memory When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown. true oom_sigkill_enabled true
Automatically Restart Process When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure. true process_auto_restart true
Enable Metric Collection Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process. true process_should_monitor true

Erasure Coding

Display Name Description Related Name Default Value API Name Required
DataNode Striped Read Reconstruction Threads The number of threads that a DataNode can use during background data reconstruction. dfs.datanode.ec.reconstruction.threads 20 erasure_coding_reconstruction_threads false
DataNode Striped Read Reconstruction Timeout The timeout for striped reads during background data reconstruction. dfs.datanode.ec.reconstruction.stripedread.timeout.millis 5 second(s) erasure_coding_reconstruction_timeout_millis false
Erasure Coding Reconstruction Weight The relative weight of resources used by EC for data recovery. The number of blocks that must be read is based on the EC policy used. For example, RS-6-3-1024k requires six blocks to be read. Replication only requires one block to be read. Higher values result in fewer reconstruction tasks being able to run concurrently. The number of blocks required to be read to recover data is multiplied by this weight to determine the total weight of the recovery task. The total weight of the recovery task counts against the limit set with the dfs.namenode.replication.max-streams property. dfs.datanode.ec.reconstruction.xmits.weight 0.5 erasure_coding_reconstruction_xmits_weight false

Logs

Display Name Description Related Name Default Value API Name Required
DataNode Log Directory Directory where DataNode will place its log files. hadoop.log.dir /var/log/hadoop-hdfs datanode_log_dir false
DataNode Logging Threshold The minimum log level for DataNode logs INFO log_threshold false
DataNode Maximum Log File Backups The maximum number of rolled log files to keep for DataNode logs. Typically used by log4j or logback. 10 max_log_backup_index false
DataNode Max Log Size The maximum size, in megabytes, per log file for DataNode logs. Typically used by log4j or logback. 200 MiB max_log_size false

Monitoring

Display Name Description Related Name Default Value API Name Required
DataNode Block Count Thresholds The health test thresholds of the number of blocks on a DataNode Warning: 1000000.0, Critical: Never datanode_block_count_thresholds false
DataNode Connectivity Health Test Enables the health test that verifies the DataNode is connected to the NameNode true datanode_connectivity_health_enabled false
DataNode Connectivity Tolerance at Startup The amount of time to wait for the DataNode to fully start up and connect to the NameNode before enforcing the connectivity check. 3 minute(s) datanode_connectivity_tolerance false
DataNode Data Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's DataNode Data Directory. Warning: 10 GiB, Critical: 5 GiB datanode_data_directories_free_space_absolute_thresholds false
DataNode Data Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's DataNode Data Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a DataNode Data Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never datanode_data_directories_free_space_percentage_thresholds false
File Descriptor Monitoring Thresholds The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit. Warning: 50.0 %, Critical: 70.0 % datanode_fd_thresholds false
DataNode Free Space Monitoring Thresholds The health test thresholds of free space in a DataNode. Specified as a percentage of the capacity on the DataNode. Warning: 20.0 %, Critical: 10.0 % datanode_free_space_thresholds false
DataNode Host Health Test When computing the overall DataNode health, consider the host's health. true datanode_host_health_enabled false
Pause Duration Thresholds The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time. Warning: 30.0, Critical: 60.0 datanode_pause_duration_thresholds false
Pause Duration Monitoring Period The period to review when computing the moving average of extra time the pause monitor spent paused. 5 minute(s) datanode_pause_duration_window false
DataNode Process Health Test Enables the health test that the DataNode's process state is consistent with the role configuration true datanode_scm_health_enabled false
DataNode Transceivers Usage Thresholds The health test thresholds of transceivers usage in a DataNode. Specified as a percentage of the total configured number of transceivers. Warning: 75.0 %, Critical: 95.0 % datanode_transceivers_usage_thresholds false
DataNode Volume Failures Thresholds The health test thresholds of failed volumes in a DataNode. Warning: Never, Critical: Any datanode_volume_failures_thresholds false
Web Metric Collection Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server. true datanode_web_metric_collection_enabled false
Web Metric Collection Duration The health test thresholds on the duration of the metrics request to the web server. Warning: 10 second(s), Critical: Never datanode_web_metric_collection_thresholds false
Enable Health Alerts for this Role When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold false enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Heap Dump Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Warning: 10 GiB, Critical: 5 GiB heap_dump_directory_free_space_absolute_thresholds false
Heap Dump Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never heap_dump_directory_free_space_percentage_thresholds false
Log Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Warning: 10 GiB, Critical: 5 GiB log_directory_free_space_absolute_thresholds false
Log Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never log_directory_free_space_percentage_thresholds false
Rules to Extract Events from Log Files This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.. Each rule has some or all of the following fields:
  • alert - whether or not events generated from this rule should be promoted to alerts. A value of "true" will cause alerts to be generated. If not specified, the default is "false".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is "WARN" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.
Example:
  • {"alert": false, "rate": 10, "exceptiontype": "java.lang.StringIndexOutOfBoundsException"}This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {"alert": false, "rate": 1, "periodminutes": 1, "exceptiontype": ".*"}, {"alert": true, "rate": 1, "periodminutes": 1, "threshold":"ERROR"}In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
version: 0, rules: [ alert: false, rate: 1, periodminutes: 1, threshold: FATAL , alert: false, rate: 0, threshold: WARN, content: .* is deprecated. Instead, use .* , alert: false, rate: 0, threshold: WARN, content: .* is deprecated. Use .* instead , alert: false, rate: 0, exceptiontype: java.io.IOException , alert: false, rate: 0, exceptiontype: java.net.SocketException , alert: false, rate: 0, exceptiontype: java.net.SocketClosedException , alert: false, rate: 0, exceptiontype: java.io.EOFException , alert: false, rate: 0, exceptiontype: java.nio.channels.CancelledKeyException , alert: false, rate: 1, periodminutes: 5, content: Datanode registration failed , alert: false, rate: 1, periodminutes: 2, exceptiontype: .* , alert: false, rate: 0, threshold: WARN, content: Got a command from standby NN - ignoring command:.* , alert: false, rate: 0, threshold: WARN, content: Unknown job [^ ]+ being deleted.* , alert: false, rate: 0, threshold: WARN, content: Error executing shell command .+ No such process.+ , alert: false, rate: 0, threshold: WARN, content: .*attempt to override final parameter.+ , alert: false, rate: 0, threshold: WARN, content: [^ ]+ is a deprecated filesystem name. Use.* , alert: false, rate: 1, periodminutes: 1, threshold: WARN ] log_event_whitelist false
Navigator Audit Failure Thresholds The health test thresholds for failures encountered when monitoring audits within a recent period specified by the mgmt_navigator_failure_window configuration for the role. The value that can be specified for this threshold is the number of bytes of audits data that is left to be sent to audit server. mgmt.navigator.failure.thresholds Warning: Never, Critical: Any mgmt_navigator_failure_thresholds false
Monitoring Period For Audit Failures The period to review when checking if audits are blocked and not getting processed. mgmt.navigator.failure.window 20 minute(s) mgmt_navigator_failure_window false
Navigator Audit Pipeline Health Check Enable test of audit events processing pipeline. This will test if audit events are not getting processed by Audit Server for a role that generates audit. mgmt.navigator.status.check.enabled true mgmt_navigator_status_check_enabled false
Process Swap Memory Thresholds The health test thresholds on the swap memory usage of the process. This takes precedence over the host level threshold. Warning: 200 B, Critical: Never process_swap_memory_thresholds false
Role Triggers The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.
For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.
[] role_triggers true
Unexpected Exits Thresholds The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role. Warning: Never, Critical: Any unexpected_exits_thresholds false
Unexpected Exits Monitoring Period The period to review when computing unexpected exits. 5 minute(s) unexpected_exits_window false

Other

Display Name Description Related Name Default Value API Name Required
DataNode Data Directory Comma-delimited list of directories on the local file system where the DataNode stores HDFS block data. Typical values are /data/N/dfs/dn for N = 1, 2, 3.... In CDH 5.7 and higher, these directories can be optionally tagged with their storage types, for example, [SSD]/data/1/dns/dn. HDFS supports the following storage types: [DISK], [SSD], [ARCHIVE], [RAM_DISK]. The default storage type of a directory will be [DISK] if it does not have a storage type tagged explicitly. These directories should be mounted using the noatime option, and the disks should be configured using JBOD. RAID is not recommended. Warning: Be very careful when modifying this property. Removing or changing entries can result in data loss. To hot swap drives in CDH 5.4 and higher, override the value of this property for the specific DataNode role instance that has the drive to be hot-swapped; do not modify the property value in the role group. See Configuring Hot Swap for DataNodes for more information. dfs.datanode.data.dir dfs_data_dir_list true
Reserved Space for Non DFS Use Reserved space in bytes per volume for non Distributed File System (DFS) use. dfs.datanode.du.reserved 10 GiB dfs_datanode_du_reserved false
DataNode Failed Volumes Tolerated The number of volumes that are allowed to fail before a DataNode stops offering service. By default, any volume failure will cause a DataNode to shutdown. dfs.datanode.failed.volumes.tolerated 0 dfs_datanode_failed_volumes_tolerated false

Performance

Display Name Description Related Name Default Value API Name Required
DataNode Balancing Bandwidth Maximum amount of bandwidth that each DataNode can use for balancing. Specified in bytes per second. dfs.datanode.balance.bandwidthPerSec 10 MiB dfs_balance_bandwidthPerSec false
Enable purging cache after reads In some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This may improve performance for some workloads by freeing buffer cache spare usage for more cacheable data. This behavior will always be disabled for workloads that read only short sections of a block (e.g HBase random-IO workloads). This property is supported in CDH3u3 or later deployments. dfs.datanode.drop.cache.behind.reads false dfs_datanode_drop_cache_behind_reads false
Enable purging cache after writes In some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache spare usage for more cacheable data. This property is supported in CDH3u3 or later deployments. dfs.datanode.drop.cache.behind.writes false dfs_datanode_drop_cache_behind_writes false
Handler Count The number of server threads for the DataNode. dfs.datanode.handler.count 3 dfs_datanode_handler_count false
Maximum Number of Transfer Threads Specifies the maximum number of threads to use for transferring data in and out of the DataNode. dfs.datanode.max.transfer.threads 4096 dfs_datanode_max_xcievers false
Number of read ahead bytes While reading block files, the DataNode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader's position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the DataNode will attempt to read ahead. A value of 0 disables this feature. This property is supported in CDH3u3 or later deployments. dfs.datanode.readahead.bytes 4 MiB dfs_datanode_readahead_bytes false
Enable immediate enqueuing of data to disk after writes If this configuration is enabled, the DataNode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. This property is supported in CDH3u3 or later deployments. dfs.datanode.sync.behind.writes false dfs_datanode_sync_behind_writes false
HDFS Thrift Server Max Threadcount Maximum number of running threads for the HDFS Thrift server running on each DataNode dfs.thrift.threads.max 20 dfs_thrift_threads_max false
HDFS Thrift Server Min Threadcount Minimum number of running threads for the HDFS Thrift server running on each DataNode dfs.thrift.threads.min 10 dfs_thrift_threads_min false
HDFS Thrift Server Timeout Timeout in seconds for the HDFS Thrift server running on each DataNode dfs.thrift.timeout 60 dfs_thrift_timeout false
Maximum Process File Descriptors If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value. rlimit_fds false

Ports and Addresses

Display Name Description Related Name Default Value API Name Required
Bind DataNode to Wildcard Address If enabled, the DataNode binds to the wildcard address ("0.0.0.0") on all of its ports. false dfs_datanode_bind_wildcard false
DataNode HTTP Web UI Port Port for the DataNode HTTP web UI. Combined with the DataNode's hostname to build its HTTP address. dfs.datanode.http.address 9864 dfs_datanode_http_port false
Secure DataNode Web UI Port (TLS/SSL) The base port where the secure DataNode web UI listens. Combined with the DataNode's hostname to build its secure web UI address. dfs.datanode.https.address 9865 dfs_datanode_https_port false
DataNode Protocol Port Port for the various DataNode Protocols. Combined with the DataNode's hostname to build its IPC port address. dfs.datanode.ipc.address 9867 dfs_datanode_ipc_port false
DataNode Transceiver Port Port for DataNode's XCeiver Protocol. Combined with the DataNode's hostname to build its address. dfs.datanode.address 9866 dfs_datanode_port false
Use DataNode Hostname Whether DataNodes should use DataNode hostnames when connecting to DataNodes for data transfer. This property is supported in CDH3u4 or later deployments. dfs.datanode.use.datanode.hostname false dfs_datanode_use_datanode_hostname false

Resource Management

Display Name Description Related Name Default Value API Name Required
Java Heap Size of DataNode in Bytes Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx. 1 GiB datanode_java_heapsize false
Maximum Memory Used for Caching The maximum amount of memory a DataNode may use to cache data blocks in memory. Setting it to zero will disable caching. dfs.datanode.max.locked.memory 4 GiB dfs_datanode_max_locked_memory false
Cgroup CPU Shares Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager. cpu.shares 1024 rm_cpu_shares true
Cgroup I/O Weight Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager. blkio.weight 500 rm_io_weight true
Cgroup Memory Hard Limit Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.limit_in_bytes -1 MiB rm_memory_hard_limit true
Cgroup Memory Soft Limit Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.soft_limit_in_bytes -1 MiB rm_memory_soft_limit true

Security

Display Name Description Related Name Default Value API Name Required
DataNode Data Directory Permissions Permissions for the directories on the local file system where the DataNode stores its blocks. The permissions must be octal. 755 and 700 are typical values. dfs.datanode.data.dir.perm 700 dfs_datanode_data_dir_perm false

Stacks Collection

Display Name Description Related Name Default Value API Name Required
Stacks Collection Data Retention The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted. stacks_collection_data_retention 100 MiB stacks_collection_data_retention false
Stacks Collection Directory The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory. stacks_collection_directory stacks_collection_directory false
Stacks Collection Enabled Whether or not periodic stacks collection is enabled. stacks_collection_enabled false stacks_collection_enabled true
Stacks Collection Frequency The frequency with which stacks are collected. stacks_collection_frequency 5.0 second(s) stacks_collection_frequency false
Stacks Collection Method The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped. stacks_collection_method jstack stacks_collection_method false

Suppressions

Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: Version Validator Whether to suppress configuration warnings produced by the Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter. false role_config_suppression_datanode_config_safety_valve true
Suppress Configuration Validator: DataNode Failed Volumes Tolerated Validator Whether to suppress configuration warnings produced by the DataNode Failed Volumes Tolerated Validator configuration validator. false role_config_suppression_datanode_failed_volumes_validator true
Suppress Parameter Validation: Java Heap Size of DataNode in Bytes Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of DataNode in Bytes parameter. false role_config_suppression_datanode_java_heapsize true
Suppress Parameter Validation: Java Configuration Options for DataNode Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for DataNode parameter. false role_config_suppression_datanode_java_opts true
Suppress Parameter Validation: DataNode Log Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Log Directory parameter. false role_config_suppression_datanode_log_dir true
Suppress Configuration Validator: DataNode Reserved Space Validator Whether to suppress configuration warnings produced by the DataNode Reserved Space Validator configuration validator. false role_config_suppression_datanode_reserved_space_validator true
Suppress Parameter Validation: DataNode Environment Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Environment Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_datanode_role_env_safety_valve true
Suppress Parameter Validation: DataNode Data Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Data Directory parameter. false role_config_suppression_dfs_data_dir_list true
Suppress Parameter Validation: DataNode Data Directory Permissions Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Data Directory Permissions parameter. false role_config_suppression_dfs_datanode_data_dir_perm true
Suppress Parameter Validation: DataNode HTTP Web UI Port Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode HTTP Web UI Port parameter. false role_config_suppression_dfs_datanode_http_port true
Suppress Parameter Validation: Secure DataNode Web UI Port (TLS/SSL) Whether to suppress configuration warnings produced by the built-in parameter validation for the Secure DataNode Web UI Port (TLS/SSL) parameter. false role_config_suppression_dfs_datanode_https_port true
Suppress Parameter Validation: DataNode Protocol Port Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Protocol Port parameter. false role_config_suppression_dfs_datanode_ipc_port true
Suppress Parameter Validation: DataNode Transceiver Port Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Transceiver Port parameter. false role_config_suppression_dfs_datanode_port true
Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_hadoop_metrics2_safety_valve true
Suppress Parameter Validation: DataNode Logging Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Logging Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_log4j_safety_valve true
Suppress Parameter Validation: Rules to Extract Events from Log Files Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter. false role_config_suppression_log_event_whitelist true
Suppress Parameter Validation: Heap Dump Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter. false role_config_suppression_oom_heap_dump_dir true
Suppress Parameter Validation: Role Triggers Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter. false role_config_suppression_role_triggers true
Suppress Parameter Validation: Stacks Collection Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter. false role_config_suppression_stacks_collection_directory true
Suppress Health Test: Audit Pipeline Test Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_audit_health true
Suppress Health Test: File Descriptors Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_file_descriptor true
Suppress Health Test: Heap Dump Directory Free Space Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_heap_dump_directory_free_space true
Suppress Health Test: Host Health Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_host_health true
Suppress Health Test: Log Directory Free Space Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_log_directory_free_space true
Suppress Health Test: Pause Duration Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_pause_duration true
Suppress Health Test: Process Status Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_scm_health true
Suppress Health Test: Swap Memory Usage Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_swap_memory_usage true
Suppress Health Test: Transceiver Usage Whether to suppress the results of the Transceiver Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_transceivers_usage true
Suppress Health Test: Unexpected Exits Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_unexpected_exits true
Suppress Health Test: Web Server Status Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_data_node_web_metric_collection true
Suppress Health Test: DataNode Data Directory Free Space Whether to suppress the results of the DataNode Data Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_datanode_data_directories_free_space true

Failover Controller

Categories:

Advanced

Display Name Description Related Name Default Value API Name Required
Java Configuration Options for Failover Controller These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here. Note: When CM version is 6.3.0 or greater, {{JAVA_GC_ARGS}} will be replaced by JVM Garbage Collection arguments based on the runtime Java JVM version. failover_controller_java_opts false
Failover Controller Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration. FAILOVERCONTROLLER_role_env_safety_valve false
Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only. A string to be inserted into hdfs-site.xml for this role only. fc_config_safety_valve false
HA Health Monitor RPC Timeout The RPC timeout for the HA health monitor. ha.health-monitor.rpc-timeout.ms 45 second(s) ha_health_monitor_rpc_timeout_ms false
Failover Controller Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false
Heap Dump Directory Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role. oom_heap_dump_dir /tmp oom_heap_dump_dir false
Dump Heap When Out of Memory When set, generates heap dump file when java.lang.OutOfMemoryError is thrown. true oom_heap_dump_enabled true
Kill When Out of Memory When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown. true oom_sigkill_enabled true
Automatically Restart Process When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure. false process_auto_restart true
Enable Metric Collection Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process. true process_should_monitor true

Logs

Display Name Description Related Name Default Value API Name Required
Failover Controller Log Directory Directory where Failover Controller will place its log files. hadoop.log.dir /var/log/hadoop-hdfs failover_controller_log_dir false
Failover Controller Logging Threshold The minimum log level for Failover Controller logs INFO log_threshold false
Failover Controller Maximum Log File Backups The maximum number of rolled log files to keep for Failover Controller logs. Typically used by log4j or logback. 10 max_log_backup_index false
Failover Controller Max Log Size The maximum size, in megabytes, per log file for Failover Controller logs. Typically used by log4j or logback. 200 MiB max_log_size false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Health Alerts for this Role When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
File Descriptor Monitoring Thresholds The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit. Warning: 50.0 %, Critical: 70.0 % failovercontroller_fd_thresholds false
Failover Controller Host Health Test When computing the overall Failover Controller health, consider the host's health. true failovercontroller_host_health_enabled false
Failover Controller Process Health Test Enables the health test that the Failover Controller's process state is consistent with the role configuration true failovercontroller_scm_health_enabled false
Heap Dump Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Warning: 10 GiB, Critical: 5 GiB heap_dump_directory_free_space_absolute_thresholds false
Heap Dump Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never heap_dump_directory_free_space_percentage_thresholds false
Log Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Warning: 10 GiB, Critical: 5 GiB log_directory_free_space_absolute_thresholds false
Log Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never log_directory_free_space_percentage_thresholds false
Rules to Extract Events from Log Files This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.. Each rule has some or all of the following fields:
  • alert - whether or not events generated from this rule should be promoted to alerts. A value of "true" will cause alerts to be generated. If not specified, the default is "false".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is "WARN" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.
Example:
  • {"alert": false, "rate": 10, "exceptiontype": "java.lang.StringIndexOutOfBoundsException"}This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {"alert": false, "rate": 1, "periodminutes": 1, "exceptiontype": ".*"}, {"alert": true, "rate": 1, "periodminutes": 1, "threshold":"ERROR"}In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
version: 0, rules: [ alert: false, rate: 1, periodminutes: 1, threshold: FATAL , alert: false, rate: 1, periodminutes: 2, exceptiontype: .* , alert: false, rate: 1, periodminutes: 1, threshold: WARN ] log_event_whitelist false
Navigator Audit Failure Thresholds The health test thresholds for failures encountered when monitoring audits within a recent period specified by the mgmt_navigator_failure_window configuration for the role. The value that can be specified for this threshold is the number of bytes of audits data that is left to be sent to audit server. mgmt.navigator.failure.thresholds Warning: Never, Critical: Any mgmt_navigator_failure_thresholds false
Monitoring Period For Audit Failures The period to review when checking if audits are blocked and not getting processed. mgmt.navigator.failure.window 20 minute(s) mgmt_navigator_failure_window false
Navigator Audit Pipeline Health Check Enable test of audit events processing pipeline. This will test if audit events are not getting processed by Audit Server for a role that generates audit. mgmt.navigator.status.check.enabled true mgmt_navigator_status_check_enabled false
Process Swap Memory Thresholds The health test thresholds on the swap memory usage of the process. This takes precedence over the host level threshold. Warning: 200 B, Critical: Never process_swap_memory_thresholds false
Role Triggers The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.
For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.
[] role_triggers true
Unexpected Exits Thresholds The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role. Warning: Never, Critical: Any unexpected_exits_thresholds false
Unexpected Exits Monitoring Period The period to review when computing unexpected exits. 5 minute(s) unexpected_exits_window false

Performance

Display Name Description Related Name Default Value API Name Required
Maximum Process File Descriptors If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value. rlimit_fds false

Resource Management

Display Name Description Related Name Default Value API Name Required
Java Heap Size of Failover Controller in Bytes Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx. 256 MiB failover_controller_java_heapsize false
Cgroup CPU Shares Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager. cpu.shares 1024 rm_cpu_shares true
Cgroup I/O Weight Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager. blkio.weight 500 rm_io_weight true
Cgroup Memory Hard Limit Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.limit_in_bytes -1 MiB rm_memory_hard_limit true
Cgroup Memory Soft Limit Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.soft_limit_in_bytes -1 MiB rm_memory_soft_limit true

Stacks Collection

Display Name Description Related Name Default Value API Name Required
Stacks Collection Data Retention The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted. stacks_collection_data_retention 100 MiB stacks_collection_data_retention false
Stacks Collection Directory The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory. stacks_collection_directory stacks_collection_directory false
Stacks Collection Enabled Whether or not periodic stacks collection is enabled. stacks_collection_enabled false stacks_collection_enabled true
Stacks Collection Frequency The frequency with which stacks are collected. stacks_collection_frequency 5.0 second(s) stacks_collection_frequency false
Stacks Collection Method The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped. stacks_collection_method jstack stacks_collection_method false

Suppressions

Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: Version Validator Whether to suppress configuration warnings produced by the Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: Java Configuration Options for Failover Controller Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Failover Controller parameter. false role_config_suppression_failover_controller_java_opts true
Suppress Parameter Validation: Failover Controller Log Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Log Directory parameter. false role_config_suppression_failover_controller_log_dir true
Suppress Parameter Validation: Failover Controller Environment Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Environment Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_failovercontroller_role_env_safety_valve true
Suppress Parameter Validation: Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter. false role_config_suppression_fc_config_safety_valve true
Suppress Parameter Validation: Failover Controller Logging Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Logging Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_log4j_safety_valve true
Suppress Parameter Validation: Rules to Extract Events from Log Files Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter. false role_config_suppression_log_event_whitelist true
Suppress Parameter Validation: Heap Dump Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter. false role_config_suppression_oom_heap_dump_dir true
Suppress Parameter Validation: Role Triggers Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter. false role_config_suppression_role_triggers true
Suppress Parameter Validation: Stacks Collection Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter. false role_config_suppression_stacks_collection_directory true
Suppress Health Test: Audit Pipeline Test Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_hdfs_failovercontroller_audit_health true
Suppress Health Test: File Descriptors Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_hdfs_failovercontroller_file_descriptor true
Suppress Health Test: Heap Dump Directory Free Space Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_hdfs_failovercontroller_heap_dump_directory_free_space true
Suppress Health Test: Host Health Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_hdfs_failovercontroller_host_health true
Suppress Health Test: Log Directory Free Space Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_hdfs_failovercontroller_log_directory_free_space true
Suppress Health Test: Process Status Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_hdfs_failovercontroller_scm_health true
Suppress Health Test: Swap Memory Usage Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_hdfs_failovercontroller_swap_memory_usage true
Suppress Health Test: Unexpected Exits Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_hdfs_failovercontroller_unexpected_exits true

Gateway

Categories:

Advanced

Display Name Description Related Name Default Value API Name Required
Deploy Directory The directory where the client configs will be deployed /etc/hadoop client_config_root_dir true
HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only, a string to be inserted into the client configuration for hdfs-site.xml. hdfs_client_config_safety_valve false
HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hadoop-env.sh hdfs_client_env_safety_valve false
Client Java Configuration Options These are Java command-line arguments. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here. -Djava.net.preferIPv4Stack=true hdfs_client_java_opts false
Gateway Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false

Logs

Display Name Description Related Name Default Value API Name Required
Gateway Logging Threshold The minimum log level for Gateway logs INFO log_threshold false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false

Other

Display Name Description Related Name Default Value API Name Required
Alternatives Priority The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others. 90 client_config_priority true
Use Trash Move deleted files to the trash so that they can be recovered if necessary. This client side configuration takes effect only if the HDFS service-wide trash is disabled (NameNode Filesystem Trash Interval set to 0) and is ignored otherwise. The trash is not automatically emptied when enabled with this configuration. false dfs_client_use_trash false

Performance

Display Name Description Related Name Default Value API Name Required
Enable HDFS Short-Circuit Read Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality. dfs.client.read.shortcircuit false dfs_client_read_shortcircuit false

Resource Management

Display Name Description Related Name Default Value API Name Required
Client Java Heap Size in Bytes Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx. 256 MiB hdfs_client_java_heapsize false

Suppressions

Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: Version Validator Whether to suppress configuration warnings produced by the Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: Deploy Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter. false role_config_suppression_client_config_root_dir true
Suppress Parameter Validation: HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter. false role_config_suppression_hdfs_client_config_safety_valve true
Suppress Parameter Validation: HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter. false role_config_suppression_hdfs_client_env_safety_valve true
Suppress Parameter Validation: Client Java Configuration Options Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter. false role_config_suppression_hdfs_client_java_opts true
Suppress Configuration Validator: HDFS Trash Enabled Validator Whether to suppress configuration warnings produced by the HDFS Trash Enabled Validator configuration validator. false role_config_suppression_hdfs_trash_disabled_validator true
Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_log4j_safety_valve true

HttpFS

Categories:

Advanced

Display Name Description Related Name Default Value API Name Required
HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml For advanced use only. A string to be inserted into httpfs-site.xml for this role only. httpfs_config_safety_valve false
HttpFS Advanced Configuration Snippet (Safety Valve) for core-site.xml For advanced use only. A string to be inserted into core-site.xml for this role only. httpfs_core_site_safety_valve false
HttpFS Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only. A string to be inserted into hdfs-site.xml for this role only. httpfs_hdfs_site_safety_valve false
Java Configuration Options for HttpFS These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here. Note: When CM version is 6.3.0 or greater, {{JAVA_GC_ARGS}} will be replaced by JVM Garbage Collection arguments based on the runtime Java JVM version. httpfs_java_opts false
System Group The group that the HttpFS server process should run as. httpfs httpfs_process_groupname true
System User The user that the HttpFS server process should run as. httpfs httpfs_process_username true
HttpFS Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration. HTTPFS_role_env_safety_valve false
HttpFS Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false
Heap Dump Directory Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role. oom_heap_dump_dir /tmp oom_heap_dump_dir false
Dump Heap When Out of Memory When set, generates heap dump file when java.lang.OutOfMemoryError is thrown. true oom_heap_dump_enabled true
Kill When Out of Memory When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown. true oom_sigkill_enabled true
Automatically Restart Process When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure. false process_auto_restart true
Enable Metric Collection Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process. true process_should_monitor true

Logs

Display Name Description Related Name Default Value API Name Required
HttpFS Log Directory Directory where HttpFS will place its log files. hadoop.log.dir /var/log/hadoop-httpfs httpfs_log_dir false
HttpFS Logging Threshold The minimum log level for HttpFS logs INFO log_threshold false
HttpFS Maximum Log File Backups The maximum number of rolled log files to keep for HttpFS logs. Typically used by log4j or logback. 10 max_log_backup_index false
HttpFS Max Log Size The maximum size, in megabytes, per log file for HttpFS logs. Typically used by log4j or logback. 200 MiB max_log_size false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Health Alerts for this Role When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Heap Dump Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Warning: 10 GiB, Critical: 5 GiB heap_dump_directory_free_space_absolute_thresholds false
Heap Dump Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never heap_dump_directory_free_space_percentage_thresholds false
File Descriptor Monitoring Thresholds The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit. Warning: 50.0 %, Critical: 70.0 % httpfs_fd_thresholds false
HttpFS Host Health Test When computing the overall HttpFS health, consider the host's health. true httpfs_host_health_enabled false
HttpFS Process Health Test Enables the health test that the HttpFS's process state is consistent with the role configuration true httpfs_scm_health_enabled false
Log Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Warning: 10 GiB, Critical: 5 GiB log_directory_free_space_absolute_thresholds false
Log Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never log_directory_free_space_percentage_thresholds false
Navigator Audit Failure Thresholds The health test thresholds for failures encountered when monitoring audits within a recent period specified by the mgmt_navigator_failure_window configuration for the role. The value that can be specified for this threshold is the number of bytes of audits data that is left to be sent to audit server. mgmt.navigator.failure.thresholds Warning: Never, Critical: Any mgmt_navigator_failure_thresholds false
Monitoring Period For Audit Failures The period to review when checking if audits are blocked and not getting processed. mgmt.navigator.failure.window 20 minute(s) mgmt_navigator_failure_window false
Navigator Audit Pipeline Health Check Enable test of audit events processing pipeline. This will test if audit events are not getting processed by Audit Server for a role that generates audit. mgmt.navigator.status.check.enabled true mgmt_navigator_status_check_enabled false
Process Swap Memory Thresholds The health test thresholds on the swap memory usage of the process. This takes precedence over the host level threshold. Warning: 200 B, Critical: Never process_swap_memory_thresholds false
Role Triggers The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.
For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.
[] role_triggers true
Unexpected Exits Thresholds The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role. Warning: Never, Critical: Any unexpected_exits_thresholds false
Unexpected Exits Monitoring Period The period to review when computing unexpected exits. 5 minute(s) unexpected_exits_window false

Other

Display Name Description Related Name Default Value API Name Required
HttpFS Load Balancer Address of the load balancer used for HttpFS roles. Should be specified in host:port format. Note: Changing this property will regenerate Kerberos keytabs for all HttpFS roles. httpfs_load_balancer false

Performance

Display Name Description Related Name Default Value API Name Required
Maximum Process File Descriptors If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value. rlimit_fds false

Ports and Addresses

Display Name Description Related Name Default Value API Name Required
Administration Port The port for the administration interface. hdfs.httpfs.admin.port 14001 hdfs_httpfs_admin_port false
REST Port The port where the REST interface to HDFS is available. The REST interface is served over HTTPS if TLS/SSL is enabled for HttpFS, or over HTTP otherwise. hdfs.httpfs.http.port 14000 hdfs_httpfs_http_port false

Resource Management

Display Name Description Related Name Default Value API Name Required
Java Heap Size of HttpFS in Bytes Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx. 256 MiB httpfs_java_heapsize false
Cgroup CPU Shares Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager. cpu.shares 1024 rm_cpu_shares true
Cgroup I/O Weight Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager. blkio.weight 500 rm_io_weight true
Cgroup Memory Hard Limit Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.limit_in_bytes -1 MiB rm_memory_hard_limit true
Cgroup Memory Soft Limit Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.soft_limit_in_bytes -1 MiB rm_memory_soft_limit true

Security

Display Name Description Related Name Default Value API Name Required
Signature Secret The secret to use for signing client authentication tokens. hdfs.httpfs.signature.secret ****** hdfs_httpfs_signature_secret true
HttpFS TLS/SSL Server JKS Keystore File Location The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HttpFS is acting as a TLS/SSL server. The keystore must be in JKS format. httpfs_https_keystore_file false
HttpFS TLS/SSL Server JKS Keystore File Password The password for the HttpFS JKS keystore file. httpfs_https_keystore_password false
HttpFS TLS/SSL Client Trust Store File The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HttpFS might connect to. This is used when HttpFS is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead. httpfs_https_truststore_file false
HttpFS TLS/SSL Client Trust Store Password The password for the HttpFS TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information. httpfs_https_truststore_password false
Enable TLS/SSL for HttpFS Encrypt communication between clients and HttpFS using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)). false httpfs_use_ssl false
Role-Specific Kerberos Principal Kerberos principal used by the HttpFS roles. httpfs kerberos_role_princ_name true

Stacks Collection

Display Name Description Related Name Default Value API Name Required
Stacks Collection Data Retention The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted. stacks_collection_data_retention 100 MiB stacks_collection_data_retention false
Stacks Collection Directory The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory. stacks_collection_directory stacks_collection_directory false
Stacks Collection Enabled Whether or not periodic stacks collection is enabled. stacks_collection_enabled false stacks_collection_enabled true
Stacks Collection Frequency The frequency with which stacks are collected. stacks_collection_frequency 5.0 second(s) stacks_collection_frequency false
Stacks Collection Method The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped. stacks_collection_method jstack stacks_collection_method false

Suppressions

Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: Version Validator Whether to suppress configuration warnings produced by the Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: Administration Port Whether to suppress configuration warnings produced by the built-in parameter validation for the Administration Port parameter. false role_config_suppression_hdfs_httpfs_admin_port true
Suppress Parameter Validation: REST Port Whether to suppress configuration warnings produced by the built-in parameter validation for the REST Port parameter. false role_config_suppression_hdfs_httpfs_http_port true
Suppress Parameter Validation: Signature Secret Whether to suppress configuration warnings produced by the built-in parameter validation for the Signature Secret parameter. false role_config_suppression_hdfs_httpfs_signature_secret true
Suppress Parameter Validation: HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml parameter. false role_config_suppression_httpfs_config_safety_valve true
Suppress Parameter Validation: HttpFS Advanced Configuration Snippet (Safety Valve) for core-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter. false role_config_suppression_httpfs_core_site_safety_valve true
Suppress Parameter Validation: HttpFS Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter. false role_config_suppression_httpfs_hdfs_site_safety_valve true
Suppress Parameter Validation: HttpFS TLS/SSL Server JKS Keystore File Location Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Server JKS Keystore File Location parameter. false role_config_suppression_httpfs_https_keystore_file true
Suppress Parameter Validation: HttpFS TLS/SSL Server JKS Keystore File Password Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Server JKS Keystore File Password parameter. false role_config_suppression_httpfs_https_keystore_password true
Suppress Parameter Validation: HttpFS TLS/SSL Client Trust Store File Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Client Trust Store File parameter. false role_config_suppression_httpfs_https_truststore_file true
Suppress Parameter Validation: HttpFS TLS/SSL Client Trust Store Password Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Client Trust Store Password parameter. false role_config_suppression_httpfs_https_truststore_password true
Suppress Parameter Validation: Java Configuration Options for HttpFS Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HttpFS parameter. false role_config_suppression_httpfs_java_opts true
Suppress Parameter Validation: HttpFS Load Balancer Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Load Balancer parameter. false role_config_suppression_httpfs_load_balancer true
Suppress Parameter Validation: HttpFS Log Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Log Directory parameter. false role_config_suppression_httpfs_log_dir true
Suppress Parameter Validation: System Group Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter. false role_config_suppression_httpfs_process_groupname true
Suppress Parameter Validation: System User Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter. false role_config_suppression_httpfs_process_username true
Suppress Parameter Validation: HttpFS Environment Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Environment Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_httpfs_role_env_safety_valve true
Suppress Parameter Validation: Role-Specific Kerberos Principal Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter. false role_config_suppression_kerberos_role_princ_name true
Suppress Parameter Validation: HttpFS Logging Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Logging Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_log4j_safety_valve true
Suppress Parameter Validation: Heap Dump Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter. false role_config_suppression_oom_heap_dump_dir true
Suppress Parameter Validation: Role Triggers Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter. false role_config_suppression_role_triggers true
Suppress Parameter Validation: Stacks Collection Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter. false role_config_suppression_stacks_collection_directory true
Suppress Health Test: Audit Pipeline Test Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_httpfs_audit_health true
Suppress Health Test: File Descriptors Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_httpfs_file_descriptor true
Suppress Health Test: Heap Dump Directory Free Space Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_httpfs_heap_dump_directory_free_space true
Suppress Health Test: Host Health Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_httpfs_host_health true
Suppress Health Test: Log Directory Free Space Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_httpfs_log_directory_free_space true
Suppress Health Test: Process Status Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_httpfs_scm_health true
Suppress Health Test: Swap Memory Usage Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_httpfs_swap_memory_usage true
Suppress Health Test: Unexpected Exits Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_httpfs_unexpected_exits true

JournalNode

Categories:

Advanced

Display Name Description Related Name Default Value API Name Required
Enable JournalNode Syncer When enabled, a JournalNode will periodically sync edit logs with other JournalNodes. dfs.journalnode.enable.sync true dfs_journalnode_enable_sync false
JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only. A string to be inserted into hdfs-site.xml for this role only. jn_config_safety_valve false
Java Configuration Options for JournalNode These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here. Note: When CM version is 6.3.0 or greater, {{JAVA_GC_ARGS}} will be replaced by JVM Garbage Collection arguments based on the runtime Java JVM version. journalNode_java_opts false
JournalNode Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration. JOURNALNODE_role_env_safety_valve false
JournalNode Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false
Heap Dump Directory Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role. oom_heap_dump_dir /tmp oom_heap_dump_dir false
Dump Heap When Out of Memory When set, generates heap dump file when java.lang.OutOfMemoryError is thrown. true oom_heap_dump_enabled true
Kill When Out of Memory When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown. true oom_sigkill_enabled true
Automatically Restart Process When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure. true process_auto_restart true
Enable Metric Collection Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process. true process_should_monitor true

Logs

Display Name Description Related Name Default Value API Name Required
JournalNode Log Directory Directory where JournalNode will place its log files. hadoop.log.dir /var/log/hadoop-hdfs journalnode_log_dir false
JournalNode Logging Threshold The minimum log level for JournalNode logs INFO log_threshold false
JournalNode Maximum Log File Backups The maximum number of rolled log files to keep for JournalNode logs. Typically used by log4j or logback. 10 max_log_backup_index false
JournalNode Max Log Size The maximum size, in megabytes, per log file for JournalNode logs. Typically used by log4j or logback. 200 MiB max_log_size false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Health Alerts for this Role When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Heap Dump Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Warning: 10 GiB, Critical: 5 GiB heap_dump_directory_free_space_absolute_thresholds false
Heap Dump Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never heap_dump_directory_free_space_percentage_thresholds false
JournalNode Edits Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's JournalNode Edits Directory. Warning: 10 GiB, Critical: 5 GiB journalnode_edits_directory_free_space_absolute_thresholds false
JournalNode Edits Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's JournalNode Edits Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a JournalNode Edits Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never journalnode_edits_directory_free_space_percentage_thresholds false
File Descriptor Monitoring Thresholds The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit. Warning: 50.0 %, Critical: 70.0 % journalnode_fd_thresholds false
JournalNode Fsync Latency Thresholds The health test thresholds for JournalNode fsync latency. Warning: 1 second(s), Critical: 3 second(s) journalnode_fsync_latency_thresholds false
Garbage Collection Duration Thresholds The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time. Warning: 30.0, Critical: 60.0 journalnode_gc_duration_thresholds false
Garbage Collection Duration Monitoring Period The period to review when computing the moving average of garbage collection time. 5 minute(s) journalnode_gc_duration_window false
JournalNode Host Health Test When computing the overall JournalNode health, consider the host's health. true journalnode_host_health_enabled false
JournalNode Process Health Test Enables the health test that the JournalNode's process state is consistent with the role configuration true journalnode_scm_health_enabled false
Active NameNode Sync Status Health Check Enables the health check that verifies the active NameNode's sync status to the JournalNode true journalnode_sync_status_enabled false
Active NameNode Sync Status Startup Tolerance The amount of time at JournalNode startup allowed for the active NameNode to get in sync with the JournalNode. 3 minute(s) journalnode_sync_status_startup_tolerance false
Web Metric Collection Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server. true journalnode_web_metric_collection_enabled false
Web Metric Collection Duration The health test thresholds on the duration of the metrics request to the web server. Warning: 10 second(s), Critical: Never journalnode_web_metric_collection_thresholds false
Log Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Warning: 10 GiB, Critical: 5 GiB log_directory_free_space_absolute_thresholds false
Log Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never log_directory_free_space_percentage_thresholds false
Rules to Extract Events from Log Files This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.. Each rule has some or all of the following fields:
  • alert - whether or not events generated from this rule should be promoted to alerts. A value of "true" will cause alerts to be generated. If not specified, the default is "false".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is "WARN" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.
Example:
  • {"alert": false, "rate": 10, "exceptiontype": "java.lang.StringIndexOutOfBoundsException"}This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {"alert": false, "rate": 1, "periodminutes": 1, "exceptiontype": ".*"}, {"alert": true, "rate": 1, "periodminutes": 1, "threshold":"ERROR"}In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
version: 0, rules: [ alert: false, rate: 1, periodminutes: 1, threshold: FATAL , alert: false, rate: 1, periodminutes: 2, exceptiontype: .* , alert: false, rate: 1, periodminutes: 1, threshold: WARN ] log_event_whitelist false
Navigator Audit Failure Thresholds The health test thresholds for failures encountered when monitoring audits within a recent period specified by the mgmt_navigator_failure_window configuration for the role. The value that can be specified for this threshold is the number of bytes of audits data that is left to be sent to audit server. mgmt.navigator.failure.thresholds Warning: Never, Critical: Any mgmt_navigator_failure_thresholds false
Monitoring Period For Audit Failures The period to review when checking if audits are blocked and not getting processed. mgmt.navigator.failure.window 20 minute(s) mgmt_navigator_failure_window false
Navigator Audit Pipeline Health Check Enable test of audit events processing pipeline. This will test if audit events are not getting processed by Audit Server for a role that generates audit. mgmt.navigator.status.check.enabled true mgmt_navigator_status_check_enabled false
Process Swap Memory Thresholds The health test thresholds on the swap memory usage of the process. This takes precedence over the host level threshold. Warning: 200 B, Critical: Never process_swap_memory_thresholds false
Role Triggers The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.
For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.
[] role_triggers true
Unexpected Exits Thresholds The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role. Warning: Never, Critical: Any unexpected_exits_thresholds false
Unexpected Exits Monitoring Period The period to review when computing unexpected exits. 5 minute(s) unexpected_exits_window false

Other

Display Name Description Related Name Default Value API Name Required
JournalNode Edits Directory Directory on the local file system where NameNode edits are written. dfs.journalnode.edits.dir dfs_journalnode_edits_dir true

Performance

Display Name Description Related Name Default Value API Name Required
Maximum Process File Descriptors If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value. rlimit_fds false

Ports and Addresses

Display Name Description Related Name Default Value API Name Required
JournalNode HTTP Port Port for the JournalNode HTTP web UI. Combined with the JournalNode hostname to build its HTTP address. dfs.journalnode.http-address 8480 dfs_journalnode_http_port false
Secure JournalNode Web UI Port (TLS/SSL) The base port where the secure JournalNode web UI listens. Combined with the JournalNode's hostname to build its secure web UI address. dfs.journalnode.https-address 8481 dfs_journalnode_https_port false
JournalNode RPC Port Port for the JournalNode's RPC. Combined with the JournalNode's hostname to build its RPC address. dfs.journalnode.rpc-address 8485 dfs_journalnode_rpc_port false
Bind JournalNode to Wildcard Address If enabled, the JournalNode binds to the wildcard address ("0.0.0.0") on all of its ports. false journalnode_bind_wildcard false

Resource Management

Display Name Description Related Name Default Value API Name Required
Java Heap Size of JournalNode in Bytes Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx. 512 MiB journalNode_java_heapsize false
Cgroup CPU Shares Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager. cpu.shares 1024 rm_cpu_shares true
Cgroup I/O Weight Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager. blkio.weight 500 rm_io_weight true
Cgroup Memory Hard Limit Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.limit_in_bytes -1 MiB rm_memory_hard_limit true
Cgroup Memory Soft Limit Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.soft_limit_in_bytes -1 MiB rm_memory_soft_limit true

Stacks Collection

Display Name Description Related Name Default Value API Name Required
Stacks Collection Data Retention The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted. stacks_collection_data_retention 100 MiB stacks_collection_data_retention false
Stacks Collection Directory The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory. stacks_collection_directory stacks_collection_directory false
Stacks Collection Enabled Whether or not periodic stacks collection is enabled. stacks_collection_enabled false stacks_collection_enabled true
Stacks Collection Frequency The frequency with which stacks are collected. stacks_collection_frequency 5.0 second(s) stacks_collection_frequency false
Stacks Collection Method The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped. stacks_collection_method jstack stacks_collection_method false

Suppressions

Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: Version Validator Whether to suppress configuration warnings produced by the Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: JournalNode Edits Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Edits Directory parameter. false role_config_suppression_dfs_journalnode_edits_dir true
Suppress Parameter Validation: JournalNode HTTP Port Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode HTTP Port parameter. false role_config_suppression_dfs_journalnode_http_port true
Suppress Parameter Validation: Secure JournalNode Web UI Port (TLS/SSL) Whether to suppress configuration warnings produced by the built-in parameter validation for the Secure JournalNode Web UI Port (TLS/SSL) parameter. false role_config_suppression_dfs_journalnode_https_port true
Suppress Parameter Validation: JournalNode RPC Port Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode RPC Port parameter. false role_config_suppression_dfs_journalnode_rpc_port true
Suppress Parameter Validation: JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter. false role_config_suppression_jn_config_safety_valve true
Suppress Parameter Validation: Java Configuration Options for JournalNode Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for JournalNode parameter. false role_config_suppression_journalnode_java_opts true
Suppress Parameter Validation: JournalNode Log Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Log Directory parameter. false role_config_suppression_journalnode_log_dir true
Suppress Parameter Validation: JournalNode Environment Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Environment Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_journalnode_role_env_safety_valve true
Suppress Parameter Validation: JournalNode Logging Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Logging Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_log4j_safety_valve true
Suppress Parameter Validation: Rules to Extract Events from Log Files Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter. false role_config_suppression_log_event_whitelist true
Suppress Parameter Validation: Heap Dump Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter. false role_config_suppression_oom_heap_dump_dir true
Suppress Parameter Validation: Role Triggers Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter. false role_config_suppression_role_triggers true
Suppress Parameter Validation: Stacks Collection Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter. false role_config_suppression_stacks_collection_directory true
Suppress Health Test: Audit Pipeline Test Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_audit_health true
Suppress Health Test: JournalNode Edits Directory Free Space Whether to suppress the results of the JournalNode Edits Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_edits_directory_free_space true
Suppress Health Test: File Descriptors Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_file_descriptor true
Suppress Health Test: Fsync Latency Whether to suppress the results of the Fsync Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_fsync_latency true
Suppress Health Test: GC Duration Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_gc_duration true
Suppress Health Test: Heap Dump Directory Free Space Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_heap_dump_directory_free_space true
Suppress Health Test: Host Health Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_host_health true
Suppress Health Test: Log Directory Free Space Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_log_directory_free_space true
Suppress Health Test: Process Status Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_scm_health true
Suppress Health Test: Swap Memory Usage Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_swap_memory_usage true
Suppress Health Test: Unexpected Exits Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_unexpected_exits true
Suppress Health Test: Web Server Status Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_journal_node_web_metric_collection true

NameNode

Categories:

Advanced

Display Name Description Related Name Default Value API Name Required
Enable Automatic Failover Enable Automatic Failover to maintain High Availability. Requires a ZooKeeper service and a High Availability NameNode partner. dfs.ha.automatic-failover.enabled false autofailover_enabled false
NameNode Nameservice Nameservice of this NameNode. The Nameservice represents the interface to this NameNode and its High Availability partner. The Nameservice also represents the namespace associated with a federated NameNode. dfs_federation_namenode_nameservice false
Avoid Reading Stale DataNode Indicate whether or not to avoid reading from stale DataNodes for which heartbeat messages have not been received by the NameNode for more than Stale DataNode Time Interval. Stale DataNodes are moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes. dfs.namenode.avoid.read.stale.datanode false dfs_namenode_avoid_read_stale_datanode false
Avoid Writing Stale DataNode Indicate whether or not to avoid writing to stale DataNodes for which heartbeat messages have not been received by the NameNode for more than Stale DataNode Time Interval. Writes avoid using stale DataNodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of DataNodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads. dfs.namenode.avoid.write.stale.datanode false dfs_namenode_avoid_write_stale_datanode false
Invalidate Work Percentage Per Iteration This determines the percentage amount of block invalidations (deletes) to do over a single DataNode heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DataNode. dfs.namenode.invalidate.work.pct.per.iteration 0.32 dfs_namenode_invalidate_work_pct_per_iteration false
Quorum-based Storage Journal name Name of the journal located on each JournalNode filesystem. dfs_namenode_quorum_journal_name false
Maximum Number of Replication Threads on a DataNode The maximum number of outgoing replication threads a node can have at one time. This limit is waived for the highest priority replications. Configure dfs.namenode.replication.max-streams-hard-limit to set the absolute limit, including the highest-priority replications. dfs.namenode.replication.max-streams 20 dfs_namenode_replication_max_streams false
Hard Limit on the Number of Replication Threads on a Datanode The absolute maximum number of outgoing replication threads a given node can have at one time. The regular limit (dfs.namenode.replication.max-streams) is waived for highest-priority block replications. Highest replication priority is for blocks that are at a very high risk of loss if the disk or server on which they remain fails. These are usually blocks with only one copy, or blocks with zero live copies but a copy in a node being decommissioned. dfs.namenode.replication.max-streams-hard-limit provides a limit on the total number of outgoing replication threads, including threads of all priorities. dfs.namenode.replication.max-streams-hard-limit 40 dfs_namenode_replication_max_streams_hard_limit false
Replication Work Multiplier Per Iteration This determines the total amount of block transfers to begin in parallel at a DataNode for replication, when such a command list is being sent over a DataNode heartbeat by the NameNode. The actual number is obtained by multiplying this value by the total number of live nodes in the cluster. The result number is the number of blocks to transfer immediately, per DataNode heartbeat. dfs.namenode.replication.work.multiplier.per.iteration 10 dfs_namenode_replication_work_multiplier_per_iteration false
Enable Immutable Snapshots When enabled, HDFS snapshots will capture point-in-time copies of open files. dfs.namenode.snapshot.capture.openfiles true dfs_namenode_snapshot_capture_openfiles false
Stale DataNode Time Interval Default time interval for marking a DataNode as "stale". If the NameNode has not received heartbeat messages from a DataNode for more than this time interval, the DataNode is marked and treated as "stale" by default. dfs.namenode.stale.datanode.interval 30 second(s) dfs_namenode_stale_datanode_interval false
Write Stale DataNode Ratio When the ratio of number stale DataNodes to total DataNodes marked is greater than this ratio, permit writing to stale nodes to prevent causing hotspots. dfs.namenode.write.stale.datanode.ratio 0.5 dfs_namenode_write_stale_datanode_ratio false
JournalNode Accept Recovery Timeout Timeout when accepting recovery of an edit segment from JournalNodes. This only applies when NameNode high availability is enabled. dfs.qjournal.accept-recovery.timeout.ms 2 minute(s) dfs_qjournal_accept_recovery_timeout_ms false
JournalNode Finalize Segment Timeout Timeout when finalizing current edit segment with JournalNodes. This only applies when NameNode high availability is enabled. dfs.qjournal.finalize-segment.timeout.ms 2 minute(s) dfs_qjournal_finalize_segment_timeout_ms false
JournalNode Get State Timeout Timeout when getting current states from JournalNodes. This only applies when NameNode high availability is enabled. dfs.qjournal.get-journal-state.timeout.ms 2 minute(s) dfs_qjournal_get_journal_state_timeout_ms false
JournalNode New Epoch Timeout Timeout when creating new epoch number with JournalNodes. This only applies when NameNode high availability is enabled. dfs.qjournal.new-epoch.timeout.ms 2 minute(s) dfs_qjournal_new_epoch_timeout_ms false
JournalNode Prepare Recovery Timeout Timeout when preparing recovery of an edit segment with JournalNodes. This only applies when NameNode high availability is enabled. dfs.qjournal.prepare-recovery.timeout.ms 2 minute(s) dfs_qjournal_prepare_recovery_timeout_ms false
JournalNode Select Input Streams Timeout Timeout when selecting input streams on JournalNodes. This only applies when NameNode high availability is enabled. dfs.qjournal.select-input-streams.timeout.ms 20 second(s) dfs_qjournal_select_input_streams_timeout_ms false
JournalNode Start Segment Timeout Timeout when starting a new edit segment with JournalNodes. This only applies when NameNode high availability is enabled. dfs.qjournal.start-segment.timeout.ms 20 second(s) dfs_qjournal_start_segment_timeout_ms false
JournalNode Write Transactions Timeout Timeout when writing edits to a JournalNode. This only applies when NameNode high availability is enabled. dfs.qjournal.write-txns.timeout.ms 20 second(s) dfs_qjournal_write_txns_timeout_ms false
Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties. hadoop_metrics2_safety_valve false
NameNode Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false
NameNode Advanced Configuration Snippet (Safety Valve) for dfs_all_hosts.txt For advanced use only. A string to be inserted into dfs_all_hosts.txt for this role only. namenode_all_hosts_safety_valve false
NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only. A string to be inserted into hdfs-site.xml for this role only. namenode_config_safety_valve false
NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt For advanced use only. A string to be inserted into dfs_hosts_allow.txt for this role only. namenode_hosts_allow_safety_valve false
NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt For advanced use only. A string to be inserted into dfs_hosts_exclude.txt for this role only. namenode_hosts_exclude_safety_valve false
Java Configuration Options for NameNode These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here. Note: When CM version is 6.3.0 or greater, {{JAVA_GC_ARGS}} will be replaced by JVM Garbage Collection arguments based on the runtime Java JVM version. JAVA_GC_ARGS namenode_java_opts false
NameNode Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration. NAMENODE_role_env_safety_valve false
Mount Points Mount points that are mapped to this NameNode's nameservice. / nameservice_mountpoints false
Heap Dump Directory Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role. oom_heap_dump_dir /tmp oom_heap_dump_dir false
Dump Heap When Out of Memory When set, generates heap dump file when java.lang.OutOfMemoryError is thrown. true oom_heap_dump_enabled true
Kill When Out of Memory When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown. true oom_sigkill_enabled true
Automatically Restart Process When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure. false process_auto_restart true
Enable Metric Collection Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process. true process_should_monitor true

Checkpointing

Display Name Description Related Name Default Value API Name Required
Filesystem Checkpoint Period The time between two periodic file system checkpoints. dfs.namenode.checkpoint.period 1 hour(s) fs_checkpoint_period false
Filesystem Checkpoint Transaction Threshold The number of transactions after which the NameNode or SecondaryNameNode will create a checkpoint of the namespace, regardless of whether the checkpoint period has expired. dfs.namenode.checkpoint.txns 1000000 fs_checkpoint_txns false

Erasure Coding

Display Name Description Related Name Default Value API Name Required
Fallback Erasure Coding Policy The fallback Erasure Coding policy that HDFS uses if no policy is specified when you run the -setPolicy command. dfs.namenode.ec.system.default.policy RS-6-3-1024k erasure_coding_default_policy false

Logs

Display Name Description Related Name Default Value API Name Required
NameNode Logging Threshold The minimum log level for NameNode logs INFO log_threshold false
NameNode Maximum Log File Backups The maximum number of rolled log files to keep for NameNode logs. Typically used by log4j or logback. 10 max_log_backup_index false
NameNode Max Log Size The maximum size, in megabytes, per log file for NameNode logs. Typically used by log4j or logback. 200 MiB max_log_size false
NameNode Block State Change Logging Threshold The minimum log level for NameNode block state change log messages. Setting this to WARN or higher greatly reduces the amount of log output related to block state changes. log4j.logger.BlockStateChange INFO namenode_blockstatechange_log_threshold false
NameNode Log Directory Directory where NameNode will place its log files. hadoop.log.dir /var/log/hadoop-hdfs namenode_log_dir false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Health Alerts for this Role When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Heap Dump Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Warning: 10 GiB, Critical: 5 GiB heap_dump_directory_free_space_absolute_thresholds false
Heap Dump Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never heap_dump_directory_free_space_percentage_thresholds false
Log Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Warning: 10 GiB, Critical: 5 GiB log_directory_free_space_absolute_thresholds false
Log Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never log_directory_free_space_percentage_thresholds false
Rules to Extract Events from Log Files This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.. Each rule has some or all of the following fields:
  • alert - whether or not events generated from this rule should be promoted to alerts. A value of "true" will cause alerts to be generated. If not specified, the default is "false".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is "WARN" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.
Example:
  • {"alert": false, "rate": 10, "exceptiontype": "java.lang.StringIndexOutOfBoundsException"}This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {"alert": false, "rate": 1, "periodminutes": 1, "exceptiontype": ".*"}, {"alert": true, "rate": 1, "periodminutes": 1, "threshold":"ERROR"}In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
version: 0, rules: [ alert: false, rate: 1, periodminutes: 1, threshold: FATAL , alert: false, rate: 0, threshold: WARN, content: .* is deprecated. Instead, use .* , alert: false, rate: 0, threshold: WARN, content: .* is deprecated. Use .* instead , alert: false, rate: 0, exceptiontype: java.io.IOException , alert: false, rate: 0, exceptiontype: java.net.SocketException , alert: false, rate: 0, exceptiontype: java.net.SocketClosedException , alert: false, rate: 0, exceptiontype: java.io.EOFException , alert: false, rate: 0, exceptiontype: java.nio.channels.CancelledKeyException , alert: false, rate: 1, periodminutes: 2, exceptiontype: .* , alert: false, rate: 0, threshold: WARN, content: Unknown job [^ ]+ being deleted.* , alert: false, rate: 0, threshold: WARN, content: Error executing shell command .+ No such process.+ , alert: false, rate: 0, threshold: WARN, content: .*attempt to override final parameter.+ , alert: false, rate: 0, threshold: WARN, content: [^ ]+ is a deprecated filesystem name. Use.* , alert: false, rate: 1, periodminutes: 1, threshold: WARN , alert: false, rate: 1, threshold: INFO, content: Triggering checkpoint.* ] log_event_whitelist false
Navigator Audit Failure Thresholds The health test thresholds for failures encountered when monitoring audits within a recent period specified by the mgmt_navigator_failure_window configuration for the role. The value that can be specified for this threshold is the number of bytes of audits data that is left to be sent to audit server. mgmt.navigator.failure.thresholds Warning: Never, Critical: Any mgmt_navigator_failure_thresholds false
Monitoring Period For Audit Failures The period to review when checking if audits are blocked and not getting processed. mgmt.navigator.failure.window 20 minute(s) mgmt_navigator_failure_window false
Navigator Audit Pipeline Health Check Enable test of audit events processing pipeline. This will test if audit events are not getting processed by Audit Server for a role that generates audit. mgmt.navigator.status.check.enabled true mgmt_navigator_status_check_enabled false
Filesystem Checkpoint Age Monitoring Thresholds The health test thresholds of the age of the HDFS namespace checkpoint. Specified as a percentage of the configured checkpoint interval. Warning: 200.0 %, Critical: 400.0 % namenode_checkpoint_age_thresholds false
Filesystem Checkpoint Transactions Monitoring Thresholds The health test thresholds of the number of transactions since the last HDFS namespace checkpoint. Specified as a percentage of the configured checkpointing transaction limit. Warning: 200.0 %, Critical: 400.0 % namenode_checkpoint_transactions_thresholds false
NameNode Data Directories Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's NameNode Data Directories. Warning: 10 GiB, Critical: 5 GiB namenode_data_directories_free_space_absolute_thresholds false
NameNode Data Directories Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's NameNode Data Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NameNode Data Directories Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never namenode_data_directories_free_space_percentage_thresholds false
NameNode Directory Failures Thresholds The health test thresholds of failed status directories in a NameNode. Warning: Never, Critical: Any namenode_directory_failures_thresholds false
File Descriptor Monitoring Thresholds The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit. Warning: 50.0 %, Critical: 70.0 % namenode_fd_thresholds false
NameNode Host Health Test When computing the overall NameNode health, consider the host's health. true namenode_host_health_enabled false
NameNode Out-Of-Sync JournalNodes Thresholds The health check thresholds for the number of out-of-sync JournalNodes for this NameNode. Warning: Never, Critical: Any namenode_out_of_sync_journal_nodes_thresholds false
Pause Duration Thresholds The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time. Warning: 30.0, Critical: 60.0 namenode_pause_duration_thresholds false
Pause Duration Monitoring Period The period to review when computing the moving average of extra time the pause monitor spent paused. 5 minute(s) namenode_pause_duration_window false
HDFS Rolling Metadata Upgrade Status Health Test Enables the health test of the rolling metadata upgrade status of the NameNode. This covers rolling metadata upgrades. Nonrolling metadata upgrades are covered in a separate health test. true namenode_rolling_upgrade_status_enabled false
NameNode RPC Latency Thresholds The health check thresholds of the NameNode's RPC latency. Warning: 1 second(s), Critical: 5 second(s) namenode_rpc_latency_thresholds false
NameNode RPC Latency Monitoring Window The period to review when computing the moving average of the NameNode's RPC latency. 5 minute(s) namenode_rpc_latency_window false
NameNode Safemode Health Test Enables the health test that the NameNode is not in safemode true namenode_safe_mode_enabled false
NameNode Process Health Test Enables the health test that the NameNode's process state is consistent with the role configuration true namenode_scm_health_enabled false
Health Test Startup Tolerance The amount of time allowed after this role is started that failures of health tests that rely on communication with this role will be tolerated. 5 minute(s) namenode_startup_tolerance false
HDFS Metadata Upgrade Status Health Test Enables the health test of the metadata upgrade status of the NameNode. This covers nonrolling metadata upgrades. Rolling metadata upgrades are covered in a separate health test. true namenode_upgrade_status_enabled false
Web Metric Collection Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server. true namenode_web_metric_collection_enabled false
Web Metric Collection Duration The health test thresholds on the duration of the metrics request to the web server. Warning: 10 second(s), Critical: Never namenode_web_metric_collection_thresholds false
Process Swap Memory Thresholds The health test thresholds on the swap memory usage of the process. This takes precedence over the host level threshold. Warning: 200 B, Critical: Never process_swap_memory_thresholds false
Role Triggers The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.
For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.
[] role_triggers true
Unexpected Exits Thresholds The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role. Warning: Never, Critical: Any unexpected_exits_thresholds false
Unexpected Exits Monitoring Period The period to review when computing unexpected exits. 5 minute(s) unexpected_exits_window false

Other

Display Name Description Related Name Default Value API Name Required
Backup and Disaster Log Retention Maximum age of log files related to backup and disaster recovery. 90 day(s) bdr_log_expiration_days false
Access Time Precision The access time for HDFS file is precise upto this value. Setting the value of 0 disables access times for HDFS. When using the NFS Gateway role, make sure this property is enabled. dfs.access.time.precision 1 hour(s) dfs_access_time_precision false
Decommissioning blocks per interval The approximate number of blocks to process per decommission interval, as defined in dfs.namenode.decommission.interval. dfs.namenode.decommission.blocks.per.interval 500000 dfs_decommission_blocks_per_interval false
Decommissioning max tracked nodes The maximum number of decommission-in-progress datanodes nodes that will be tracked at one time by the namenode. Tracking a decommission-in-progress datanode consumes additional NN memory proportional to the number of blocks on the datnode. A value of 0 means no limit will be enforced. dfs.namenode.decommission.max.concurrent.tracked.nodes 100 dfs_decommission_max_concurrent_tracked_nodes false
NameNode Data Directories Determines where on the local file system the NameNode should store the name table (fsimage). For redundancy, enter a comma-delimited list of directories to replicate the name table in all of the directories. Typical values are /data/N/dfs/nn where N=1..3. dfs.namenode.name.dir dfs_name_dir_list true
Restore NameNode Directories at Checkpoint Time If set to false and if one of the replicas of the NameNode storage fails, such as temporarily failure of NFS, this directory is not used until the NameNode restarts. If enabled, failed storage is re-checked on every checkpoint and, if it becomes valid, the NameNode will try to restore the edits and fsimage. dfs.namenode.name.dir.restore false dfs_name_dir_restore false
NameNode Edits Directories Directories on the local file system to store the NameNode edits. If not set, the edits are stored in the NameNode's Data Directories. The value of this configuration is automatically generated to be the Quorum-based Storage URI if there are JournalNodes and this NameNode is not Highly Available. dfs.namenode.edits.dir dfs_namenode_edits_dir false
Shared Edits Directory Directory on a shared storage device, such as a Quorum-based Storage URI or a local directory that is an NFS mount from a NAS, to store the NameNode edits. The value of this configuration is automatically generated to be the Quourm Journal URI if there are JournalNodes and this NameNode is Highly Available. dfs.namenode.shared.edits.dir dfs_namenode_shared_edits_dir false
Safemode Extension Determines extension of safemode in milliseconds after the threshold level is reached. dfs.namenode.safemode.extension 30 second(s) dfs_safemode_extension false
Safemode Minimum DataNodes Specifies the number of DataNodes that must be live before the name node exits safemode. Enter a value less than or equal to 0 to take the number of live DataNodes into account when deciding whether to remain in safemode during startup. Values greater than the number of DataNodes in the cluster will make safemode permanent. dfs.namenode.safemode.min.datanodes 1 dfs_safemode_min_datanodes false
Filesystem Trash Checkpoint Interval Number of minutes between trash checkpoints. After a .Trash directory checkpoint is created, the Filesystem Trash Interval will define the time until permanent deletion. If set to 0, the value will be considered equal to the Filesytem Trash Interval value, which can cause the permanent deletion of entries in Trash to take over twice as long. The value for this must not exceed the Filesystem Trash Interval value. fs.trash.checkpoint.interval 1 hour(s) fs_trash_checkpoint_interval false
Filesystem Trash Interval Controls the number of minutes after which a trash checkpoint directory is deleted permanently. To disable the trash feature, enter 0. The checkpointing frequency of .Trash directory contents is separately controlled by Filesystem Trash Checkpoint Interval. fs.trash.interval 1 day(s) fs_trash_interval false
Topology Script File Name Full path to a custom topology script on the host file system. The topology script is used to determine the rack location of nodes. If left blank, a topology script will be provided that uses your hosts' rack information, visible in the "Hosts" page. net.topology.script.file.name topology_script_file_name false

Performance

Display Name Description Related Name Default Value API Name Required
NameNode Handler Count The number of server threads for the NameNode. dfs.namenode.handler.count 30 dfs_namenode_handler_count false
NameNode Service Handler Count The number of server threads for the NameNode used for service calls. Only used when NameNode Service RPC Port is configured. dfs.namenode.service.handler.count 30 dfs_namenode_service_handler_count false
HDFS Thrift Server Max Threadcount Maximum number of running threads for the HDFS Thrift server running on the NameNode dfs.thrift.threads.max 20 dfs_thrift_threads_max false
HDFS Thrift Server Min Threadcount Minimum number of running threads for the HDFS Thrift server running on the NameNode dfs.thrift.threads.min 10 dfs_thrift_threads_min false
HDFS Thrift Server Timeout Timeout in seconds for the HDFS Thrift server running on the NameNode dfs.thrift.timeout 60 dfs_thrift_timeout false
Maximum Process File Descriptors If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value. rlimit_fds false

Ports and Addresses

Display Name Description Related Name Default Value API Name Required
NameNode Web UI Port The base port where the DFS NameNode web UI listens. If the port number is 0, then the server starts on a free port. Combined with the NameNode's hostname to build its HTTP address. dfs.namenode.http-address 9870 dfs_http_port false
Secure NameNode Web UI Port (TLS/SSL) The base port where the secure NameNode web UI listens. dfs.https.port 9871 dfs_https_port false
NameNode Service RPC Port Optional port for the service-rpc address which can be used by HDFS daemons instead of sharing the RPC address used by the clients. dfs.namenode.servicerpc-address dfs_namenode_servicerpc_address false
Bind NameNode to Wildcard Address If enabled, the NameNode binds to the wildcard address ("0.0.0.0") on all of its ports. false namenode_bind_wildcard false
NameNode Port The port where the NameNode runs the HDFS protocol. Combined with the NameNode's hostname to build its address. fs.defaultFS 8020 namenode_port false

Replication

Display Name Description Related Name Default Value API Name Required
Safemode Threshold Percentage Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.replication.min. Enter a value less than or equal to 0 to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safemode permanent. dfs.namenode.safemode.threshold-pct 0.999 dfs_safemode_threshold_pct false

Resource Management

Display Name Description Related Name Default Value API Name Required
Java Heap Size of NameNode in Bytes Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx. 4 GiB namenode_java_heapsize false
Cgroup CPU Shares Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager. cpu.shares 1024 rm_cpu_shares true
Cgroup I/O Weight Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager. blkio.weight 500 rm_io_weight true
Cgroup Memory Hard Limit Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.limit_in_bytes -1 MiB rm_memory_hard_limit true
Cgroup Memory Soft Limit Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.soft_limit_in_bytes -1 MiB rm_memory_soft_limit true

Security

Display Name Description Related Name Default Value API Name Required
HDFS NameNode TLS/SSL Client Trust Store File The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HDFS NameNode might connect to. This is used when HDFS NameNode is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead. namenode_truststore_file false
HDFS NameNode TLS/SSL Client Trust Store Password The password for the HDFS NameNode TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information. namenode_truststore_password false

Stacks Collection

Display Name Description Related Name Default Value API Name Required
Stacks Collection Data Retention The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted. stacks_collection_data_retention 100 MiB stacks_collection_data_retention false
Stacks Collection Directory The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory. stacks_collection_directory stacks_collection_directory false
Stacks Collection Enabled Whether or not periodic stacks collection is enabled. stacks_collection_enabled false stacks_collection_enabled true
Stacks Collection Frequency The frequency with which stacks are collected. stacks_collection_frequency 5.0 second(s) stacks_collection_frequency false
Stacks Collection Method The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped. stacks_collection_method jstack stacks_collection_method false

Suppressions

Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: Version Validator Whether to suppress configuration warnings produced by the Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: NameNode Nameservice Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Nameservice parameter. false role_config_suppression_dfs_federation_namenode_nameservice true
Suppress Parameter Validation: NameNode Web UI Port Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Web UI Port parameter. false role_config_suppression_dfs_http_port true
Suppress Parameter Validation: Secure NameNode Web UI Port (TLS/SSL) Whether to suppress configuration warnings produced by the built-in parameter validation for the Secure NameNode Web UI Port (TLS/SSL) parameter. false role_config_suppression_dfs_https_port true
Suppress Parameter Validation: NameNode Data Directories Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Data Directories parameter. false role_config_suppression_dfs_name_dir_list true
Suppress Parameter Validation: NameNode Edits Directories Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Edits Directories parameter. false role_config_suppression_dfs_namenode_edits_dir true
Suppress Configuration Validator: NameNode Handler Count Minimum Validator Whether to suppress configuration warnings produced by the NameNode Handler Count Minimum Validator configuration validator. false role_config_suppression_dfs_namenode_handler_count_minimum_validator true
Suppress Parameter Validation: Quorum-based Storage Journal name Whether to suppress configuration warnings produced by the built-in parameter validation for the Quorum-based Storage Journal name parameter. false role_config_suppression_dfs_namenode_quorum_journal_name true
Suppress Configuration Validator: NameNode Service Handler Count Minimum Validator Whether to suppress configuration warnings produced by the NameNode Service Handler Count Minimum Validator configuration validator. false role_config_suppression_dfs_namenode_service_handler_count_minimum_validator true
Suppress Parameter Validation: NameNode Service RPC Port Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Service RPC Port parameter. false role_config_suppression_dfs_namenode_servicerpc_address true
Suppress Parameter Validation: Shared Edits Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Shared Edits Directory parameter. false role_config_suppression_dfs_namenode_shared_edits_dir true
Suppress Configuration Validator: Filesystem Trash Interval On Validator Whether to suppress configuration warnings produced by the Filesystem Trash Interval On Validator configuration validator. false role_config_suppression_fs_trash_interval_minimum_validator true
Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_hadoop_metrics2_safety_valve true
Suppress Parameter Validation: NameNode Logging Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Logging Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_log4j_safety_valve true
Suppress Parameter Validation: Rules to Extract Events from Log Files Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter. false role_config_suppression_log_event_whitelist true
Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_all_hosts.txt Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_all_hosts.txt parameter. false role_config_suppression_namenode_all_hosts_safety_valve true
Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter. false role_config_suppression_namenode_config_safety_valve true
Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt parameter. false role_config_suppression_namenode_hosts_allow_safety_valve true
Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt parameter. false role_config_suppression_namenode_hosts_exclude_safety_valve true
Suppress Configuration Validator: Java Heap Size of NameNode in Bytes Minimum Validator Whether to suppress configuration warnings produced by the Java Heap Size of NameNode in Bytes Minimum Validator configuration validator. false role_config_suppression_namenode_java_heapsize_minimum_validator true
Suppress Parameter Validation: Java Configuration Options for NameNode Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NameNode parameter. false role_config_suppression_namenode_java_opts true
Suppress Parameter Validation: NameNode Log Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Log Directory parameter. false role_config_suppression_namenode_log_dir true
Suppress Parameter Validation: NameNode Port Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Port parameter. false role_config_suppression_namenode_port true
Suppress Parameter Validation: NameNode Environment Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Environment Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_namenode_role_env_safety_valve true
Suppress Parameter Validation: HDFS NameNode TLS/SSL Client Trust Store File Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS NameNode TLS/SSL Client Trust Store File parameter. false role_config_suppression_namenode_truststore_file true
Suppress Parameter Validation: HDFS NameNode TLS/SSL Client Trust Store Password Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS NameNode TLS/SSL Client Trust Store Password parameter. false role_config_suppression_namenode_truststore_password true
Suppress Configuration Validator: Validates Nameservices do not conflict between base and compute clusters. Whether to suppress configuration warnings produced by the Validates Nameservices do not conflict between base and compute clusters. configuration validator. false role_config_suppression_nameservice_conflict_validator true
Suppress Parameter Validation: Mount Points Whether to suppress configuration warnings produced by the built-in parameter validation for the Mount Points parameter. false role_config_suppression_nameservice_mountpoints true
Suppress Parameter Validation: Heap Dump Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter. false role_config_suppression_oom_heap_dump_dir true
Suppress Parameter Validation: Role Triggers Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter. false role_config_suppression_role_triggers true
Suppress Parameter Validation: Stacks Collection Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter. false role_config_suppression_stacks_collection_directory true
Suppress Parameter Validation: Topology Script File Name Whether to suppress configuration warnings produced by the built-in parameter validation for the Topology Script File Name parameter. false role_config_suppression_topology_script_file_name true
Suppress Health Test: Audit Pipeline Test Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_audit_health true
Suppress Health Test: NameNode Data Directories Free Space Whether to suppress the results of the NameNode Data Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_data_directories_free_space true
Suppress Health Test: File Descriptors Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_file_descriptor true
Suppress Health Test: Heap Dump Directory Free Space Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_heap_dump_directory_free_space true
Suppress Health Test: Host Health Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_host_health true
Suppress Health Test: Log Directory Free Space Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_log_directory_free_space true
Suppress Health Test: Pause Duration Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_pause_duration true
Suppress Health Test: Rolling Upgrade Status Whether to suppress the results of the Rolling Upgrade Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_rolling_upgrade_status true
Suppress Health Test: Process Status Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_scm_health true
Suppress Health Test: Swap Memory Usage Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_swap_memory_usage true
Suppress Health Test: Unexpected Exits Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_unexpected_exits true
Suppress Health Test: Web Server Status Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_name_node_web_metric_collection true

NFS Gateway

Categories:

Advanced

Display Name Description Related Name Default Value API Name Required
NFS Gateway Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false
NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only. A string to be inserted into hdfs-site.xml for this role only. nfsgateway_config_safety_valve false
Java Configuration Options for NFS Gateway These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here. Note: When CM version is 6.3.0 or greater, {{JAVA_GC_ARGS}} will be replaced by JVM Garbage Collection arguments based on the runtime Java JVM version. nfsgateway_java_opts false
NFS Gateway Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration. NFSGATEWAY_role_env_safety_valve false
Heap Dump Directory Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role. oom_heap_dump_dir /tmp oom_heap_dump_dir false
Dump Heap When Out of Memory When set, generates heap dump file when java.lang.OutOfMemoryError is thrown. true oom_heap_dump_enabled true
Kill When Out of Memory When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown. true oom_sigkill_enabled true
Automatically Restart Process When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure. false process_auto_restart true
Enable Metric Collection Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process. true process_should_monitor true

Logs

Display Name Description Related Name Default Value API Name Required
NFS Gateway Logging Threshold The minimum log level for NFS Gateway logs INFO log_threshold false
NFS Gateway Maximum Log File Backups The maximum number of rolled log files to keep for NFS Gateway logs. Typically used by log4j or logback. 10 max_log_backup_index false
NFS Gateway Max Log Size The maximum size, in megabytes, per log file for NFS Gateway logs. Typically used by log4j or logback. 200 MiB max_log_size false
NFS Gateway Log Directory Directory where NFS Gateway will place its log files. hadoop.log.dir /var/log/hadoop-hdfs nfsgateway_log_dir false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Health Alerts for this Role When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Heap Dump Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Warning: 10 GiB, Critical: 5 GiB heap_dump_directory_free_space_absolute_thresholds false
Heap Dump Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never heap_dump_directory_free_space_percentage_thresholds false
Log Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Warning: 10 GiB, Critical: 5 GiB log_directory_free_space_absolute_thresholds false
Log Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never log_directory_free_space_percentage_thresholds false
Rules to Extract Events from Log Files This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.. Each rule has some or all of the following fields:
  • alert - whether or not events generated from this rule should be promoted to alerts. A value of "true" will cause alerts to be generated. If not specified, the default is "false".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is "WARN" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.
Example:
  • {"alert": false, "rate": 10, "exceptiontype": "java.lang.StringIndexOutOfBoundsException"}This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {"alert": false, "rate": 1, "periodminutes": 1, "exceptiontype": ".*"}, {"alert": true, "rate": 1, "periodminutes": 1, "threshold":"ERROR"}In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
version: 0, rules: [ alert: false, rate: 1, periodminutes: 1, threshold: FATAL , alert: false, rate: 1, periodminutes: 2, exceptiontype: .* , alert: false, rate: 1, periodminutes: 1, threshold: WARN ] log_event_whitelist false
Navigator Audit Failure Thresholds The health test thresholds for failures encountered when monitoring audits within a recent period specified by the mgmt_navigator_failure_window configuration for the role. The value that can be specified for this threshold is the number of bytes of audits data that is left to be sent to audit server. mgmt.navigator.failure.thresholds Warning: Never, Critical: Any mgmt_navigator_failure_thresholds false
Monitoring Period For Audit Failures The period to review when checking if audits are blocked and not getting processed. mgmt.navigator.failure.window 20 minute(s) mgmt_navigator_failure_window false
Navigator Audit Pipeline Health Check Enable test of audit events processing pipeline. This will test if audit events are not getting processed by Audit Server for a role that generates audit. mgmt.navigator.status.check.enabled true mgmt_navigator_status_check_enabled false
Temporary Dump Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's Temporary Dump Directory. Warning: 10 GiB, Critical: 5 GiB nfsgateway_dump_directory_free_space_absolute_thresholds false
Temporary Dump Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's Temporary Dump Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Temporary Dump Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never nfsgateway_dump_directory_free_space_percentage_thresholds false
File Descriptor Monitoring Thresholds The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit. Warning: 50.0 %, Critical: 70.0 % nfsgateway_fd_thresholds false
NFS Gateway Host Health Test When computing the overall NFS Gateway health, consider the host's health. true nfsgateway_host_health_enabled false
NFS Gateway Process Health Test Enables the health test that the NFS Gateway's process state is consistent with the role configuration true nfsgateway_scm_health_enabled false
Process Swap Memory Thresholds The health test thresholds on the swap memory usage of the process. This takes precedence over the host level threshold. Warning: 200 B, Critical: Never process_swap_memory_thresholds false
Role Triggers The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.
For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.
[] role_triggers true
Unexpected Exits Thresholds The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role. Warning: Never, Critical: Any unexpected_exits_thresholds false
Unexpected Exits Monitoring Period The period to review when computing unexpected exits. 5 minute(s) unexpected_exits_window false

Other

Display Name Description Related Name Default Value API Name Required
Temporary Dump Directory NFS clients often reorder writes. As a result, sequential writes can arrive at the NFS Gateway in random order. This directory is used to temporarily save out-of-order writes before writing to HDFS. For each file, the out-of-order writes are dumped after they are accumulated to exceed certain threshold (e.g., 1MB) in memory. Please make sure this directory has enough space. For example, if the application uploads 10 files with each having 100MB, it is recommended that this directory have roughly 1GB of space in case write reorder happens (in the worst case) to every file. dfs.nfs3.dump.dir /tmp/.hdfs-nfs dfs_nfs3_dump_dir false
Allowed Hosts and Privileges By default, NFS Gateway exported directories can be mounted by any client. For better access control, update this property with a list of host names and access privileges separated by whitespace characters. Host name format can be a single host, a Java regular expression, or an IPv4 address. The access privilege uses rw to specify readwrite and ro to specify readonly access. If the access privilege is not provided, the default is read-only. Examples of host name format and access privilege: "192.168.0.0/22 rw", "host.*.example.com", "host1.test.org ro". dfs.nfs.exports.allowed.hosts * rw dfs_nfs_exports_allowed_hosts false
NFS Gateway Export Point The NFS Gateway export point(s). Full path is required. In federated clusters, multiple export points can be configured, and at most 1 export point per federated nameservice is allowed. nfs.export.point nfs_export_point false

Performance

Display Name Description Related Name Default Value API Name Required
Maximum Process File Descriptors If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value. rlimit_fds false

Ports and Addresses

Display Name Description Related Name Default Value API Name Required
NFS Gateway Web UI Port The base port where the NFS Gateway server web UI listens. Combined with the NFS Gateway server hostname to build its HTTP address. nfs.http.port 50079 nfs3_http_port false
Secure NFS Gateway Web UI Port (TLS/SSL) The base port where the secure NFS Gateway server web UI listens. Combined with the NFS Gateway server's hostname to build its secure web UI address. nfs.https.port 50579 nfs3_https_port false
NFS Gateway MountD Port The port number of the mount daemon implemented inside the NFS Gateway server role. nfs3.mountd.port 4242 nfs3_mountd_port false
Portmap (or Rpcbind) Port The port number of the system portmap or rpcbind service. This configuration is used by Cloudera Manager to verify if the system portmap or rpcbind service is running before starting NFS Gateway role. Cloudera Manager does not manage the system portmap or rpcbind service. 111 nfs3_portmap_port false
NFS Gateway Server Port The NFS Gateway server port. nfs3.server.port 2049 nfs3_server_port false

Resource Management

Display Name Description Related Name Default Value API Name Required
Java Heap Size of NFS Gateway in Bytes Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx. 256 MiB nfsgateway_java_heapsize false
Cgroup CPU Shares Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager. cpu.shares 1024 rm_cpu_shares true
Cgroup I/O Weight Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager. blkio.weight 500 rm_io_weight true
Cgroup Memory Hard Limit Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.limit_in_bytes -1 MiB rm_memory_hard_limit true
Cgroup Memory Soft Limit Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.soft_limit_in_bytes -1 MiB rm_memory_soft_limit true

Stacks Collection

Display Name Description Related Name Default Value API Name Required
Stacks Collection Data Retention The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted. stacks_collection_data_retention 100 MiB stacks_collection_data_retention false
Stacks Collection Directory The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory. stacks_collection_directory stacks_collection_directory false
Stacks Collection Enabled Whether or not periodic stacks collection is enabled. stacks_collection_enabled false stacks_collection_enabled true
Stacks Collection Frequency The frequency with which stacks are collected. stacks_collection_frequency 5.0 second(s) stacks_collection_frequency false
Stacks Collection Method The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped. stacks_collection_method jstack stacks_collection_method false

Suppressions

Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: Version Validator Whether to suppress configuration warnings produced by the Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: Temporary Dump Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Temporary Dump Directory parameter. false role_config_suppression_dfs_nfs3_dump_dir true
Suppress Parameter Validation: Allowed Hosts and Privileges Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed Hosts and Privileges parameter. false role_config_suppression_dfs_nfs_exports_allowed_hosts true
Suppress Parameter Validation: NFS Gateway Logging Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_log4j_safety_valve true
Suppress Parameter Validation: Rules to Extract Events from Log Files Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter. false role_config_suppression_log_event_whitelist true
Suppress Parameter Validation: NFS Gateway Web UI Port Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Web UI Port parameter. false role_config_suppression_nfs3_http_port true
Suppress Parameter Validation: Secure NFS Gateway Web UI Port (TLS/SSL) Whether to suppress configuration warnings produced by the built-in parameter validation for the Secure NFS Gateway Web UI Port (TLS/SSL) parameter. false role_config_suppression_nfs3_https_port true
Suppress Parameter Validation: NFS Gateway MountD Port Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway MountD Port parameter. false role_config_suppression_nfs3_mountd_port true
Suppress Parameter Validation: Portmap (or Rpcbind) Port Whether to suppress configuration warnings produced by the built-in parameter validation for the Portmap (or Rpcbind) Port parameter. false role_config_suppression_nfs3_portmap_port true
Suppress Parameter Validation: NFS Gateway Server Port Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Server Port parameter. false role_config_suppression_nfs3_server_port true
Suppress Parameter Validation: NFS Gateway Export Point Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Export Point parameter. false role_config_suppression_nfs_export_point true
Suppress Parameter Validation: NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter. false role_config_suppression_nfsgateway_config_safety_valve true
Suppress Parameter Validation: Java Configuration Options for NFS Gateway Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NFS Gateway parameter. false role_config_suppression_nfsgateway_java_opts true
Suppress Parameter Validation: NFS Gateway Log Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Log Directory parameter. false role_config_suppression_nfsgateway_log_dir true
Suppress Parameter Validation: NFS Gateway Environment Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Environment Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_nfsgateway_role_env_safety_valve true
Suppress Parameter Validation: Heap Dump Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter. false role_config_suppression_oom_heap_dump_dir true
Suppress Parameter Validation: Role Triggers Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter. false role_config_suppression_role_triggers true
Suppress Parameter Validation: Stacks Collection Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter. false role_config_suppression_stacks_collection_directory true
Suppress Health Test: Audit Pipeline Test Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_nfsgateway_audit_health true
Suppress Health Test: Temporary Dump Directory Free Space Whether to suppress the results of the Temporary Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_nfsgateway_dump_directory_free_space true
Suppress Health Test: File Descriptors Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_nfsgateway_file_descriptor true
Suppress Health Test: Heap Dump Directory Free Space Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_nfsgateway_heap_dump_directory_free_space true
Suppress Health Test: Host Health Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_nfsgateway_host_health true
Suppress Health Test: Log Directory Free Space Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_nfsgateway_log_directory_free_space true
Suppress Health Test: Process Status Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_nfsgateway_scm_health true
Suppress Health Test: Swap Memory Usage Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_nfsgateway_swap_memory_usage true
Suppress Health Test: Unexpected Exits Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_nfsgateway_unexpected_exits true

SecondaryNameNode

Categories:

Advanced

Display Name Description Related Name Default Value API Name Required
SecondaryNameNode Nameservice Nameservice of this SecondaryNameNode dfs_secondarynamenode_nameservice false
Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties. hadoop_metrics2_safety_valve false
SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false
Heap Dump Directory Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role. oom_heap_dump_dir /tmp oom_heap_dump_dir false
Dump Heap When Out of Memory When set, generates heap dump file when java.lang.OutOfMemoryError is thrown. true oom_heap_dump_enabled true
Kill When Out of Memory When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown. true oom_sigkill_enabled true
Automatically Restart Process When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure. false process_auto_restart true
Enable Metric Collection Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process. true process_should_monitor true
SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only. A string to be inserted into hdfs-site.xml for this role only. secondarynamenode_config_safety_valve false
Java Configuration Options for Secondary NameNode These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here. Note: When CM version is 6.3.0 or greater, {{JAVA_GC_ARGS}} will be replaced by JVM Garbage Collection arguments based on the runtime Java JVM version. JAVA_GC_ARGS secondarynamenode_java_opts false
SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration. SECONDARYNAMENODE_role_env_safety_valve false

Checkpointing

Display Name Description Related Name Default Value API Name Required
Filesystem Checkpoint Period The time between two periodic file system checkpoints. dfs.namenode.checkpoint.period 1 hour(s) fs_checkpoint_period false
Filesystem Checkpoint Transaction Threshold The number of transactions after which the NameNode or SecondaryNameNode will create a checkpoint of the namespace, regardless of whether the checkpoint period has expired. dfs.namenode.checkpoint.txns 1000000 fs_checkpoint_txns false

Logs

Display Name Description Related Name Default Value API Name Required
SecondaryNameNode Logging Threshold The minimum log level for SecondaryNameNode logs INFO log_threshold false
SecondaryNameNode Maximum Log File Backups The maximum number of rolled log files to keep for SecondaryNameNode logs. Typically used by log4j or logback. 10 max_log_backup_index false
SecondaryNameNode Max Log Size The maximum size, in megabytes, per log file for SecondaryNameNode logs. Typically used by log4j or logback. 200 MiB max_log_size false
SecondaryNameNode Log Directory Directory where SecondaryNameNode will place its log files. hadoop.log.dir /var/log/hadoop-hdfs secondarynamenode_log_dir false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Health Alerts for this Role When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Heap Dump Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Warning: 10 GiB, Critical: 5 GiB heap_dump_directory_free_space_absolute_thresholds false
Heap Dump Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never heap_dump_directory_free_space_percentage_thresholds false
Log Directory Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Warning: 10 GiB, Critical: 5 GiB log_directory_free_space_absolute_thresholds false
Log Directory Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never log_directory_free_space_percentage_thresholds false
Rules to Extract Events from Log Files This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.. Each rule has some or all of the following fields:
  • alert - whether or not events generated from this rule should be promoted to alerts. A value of "true" will cause alerts to be generated. If not specified, the default is "false".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is "WARN" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.
Example:
  • {"alert": false, "rate": 10, "exceptiontype": "java.lang.StringIndexOutOfBoundsException"}This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {"alert": false, "rate": 1, "periodminutes": 1, "exceptiontype": ".*"}, {"alert": true, "rate": 1, "periodminutes": 1, "threshold":"ERROR"}In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
version: 0, rules: [ alert: false, rate: 1, periodminutes: 1, threshold: FATAL , alert: false, rate: 0, threshold: WARN, content: .* is deprecated. Instead, use .* , alert: false, rate: 0, threshold: WARN, content: .* is deprecated. Use .* instead , alert: false, rate: 0, exceptiontype: java.io.IOException , alert: false, rate: 0, exceptiontype: java.net.SocketException , alert: false, rate: 0, exceptiontype: java.net.SocketClosedException , alert: false, rate: 0, exceptiontype: java.io.EOFException , alert: false, rate: 0, exceptiontype: java.nio.channels.CancelledKeyException , alert: false, rate: 1, periodminutes: 2, exceptiontype: .* , alert: false, rate: 0, threshold: WARN, content: Unknown job [^ ]+ being deleted.* , alert: false, rate: 0, threshold: WARN, content: Error executing shell command .+ No such process.+ , alert: false, rate: 0, threshold: WARN, content: .*attempt to override final parameter.+ , alert: false, rate: 0, threshold: WARN, content: [^ ]+ is a deprecated filesystem name. Use.* , alert: false, rate: 1, periodminutes: 1, threshold: WARN , alert: false, rate: 1, threshold: INFO, content: Triggering checkpoint.* ] log_event_whitelist false
Navigator Audit Failure Thresholds The health test thresholds for failures encountered when monitoring audits within a recent period specified by the mgmt_navigator_failure_window configuration for the role. The value that can be specified for this threshold is the number of bytes of audits data that is left to be sent to audit server. mgmt.navigator.failure.thresholds Warning: Never, Critical: Any mgmt_navigator_failure_thresholds false
Monitoring Period For Audit Failures The period to review when checking if audits are blocked and not getting processed. mgmt.navigator.failure.window 20 minute(s) mgmt_navigator_failure_window false
Navigator Audit Pipeline Health Check Enable test of audit events processing pipeline. This will test if audit events are not getting processed by Audit Server for a role that generates audit. mgmt.navigator.status.check.enabled true mgmt_navigator_status_check_enabled false
Process Swap Memory Thresholds The health test thresholds on the swap memory usage of the process. This takes precedence over the host level threshold. Warning: 200 B, Critical: Never process_swap_memory_thresholds false
Role Triggers The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.
For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.
[] role_triggers true
HDFS Checkpoint Directories Free Space Monitoring Absolute Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's HDFS Checkpoint Directories. Warning: 10 GiB, Critical: 5 GiB secondarynamenode_checkpoint_directories_free_space_absolute_thresholds false
HDFS Checkpoint Directories Free Space Monitoring Percentage Thresholds The health test thresholds for monitoring of free space on the filesystem that contains this role's HDFS Checkpoint Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a HDFS Checkpoint Directories Free Space Monitoring Absolute Thresholds setting is configured. Warning: Never, Critical: Never secondarynamenode_checkpoint_directories_free_space_percentage_thresholds false
File Descriptor Monitoring Thresholds The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit. Warning: 50.0 %, Critical: 70.0 % secondarynamenode_fd_thresholds false
Garbage Collection Duration Thresholds The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time. Warning: 30.0, Critical: 60.0 secondarynamenode_gc_duration_thresholds false
Garbage Collection Duration Monitoring Period The period to review when computing the moving average of garbage collection time. 5 minute(s) secondarynamenode_gc_duration_window false
SecondaryNameNode Host Health Test When computing the overall SecondaryNameNode health, consider the host's health. true secondarynamenode_host_health_enabled false
SecondaryNameNode Process Health Test Enables the health test that the SecondaryNameNode's process state is consistent with the role configuration true secondarynamenode_scm_health_enabled false
Web Metric Collection Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server. true secondarynamenode_web_metric_collection_enabled false
Web Metric Collection Duration The health test thresholds on the duration of the metrics request to the web server. Warning: 10 second(s), Critical: Never secondarynamenode_web_metric_collection_thresholds false
Unexpected Exits Thresholds The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role. Warning: Never, Critical: Any unexpected_exits_thresholds false
Unexpected Exits Monitoring Period The period to review when computing unexpected exits. 5 minute(s) unexpected_exits_window false

Other

Display Name Description Related Name Default Value API Name Required
HDFS Checkpoint Directories Determines where on the local file system the HDFS SecondaryNameNode should store the temporary images to merge. For redundancy, enter a comma-delimited list of directories to replicate the image in all of the directories. Typical values are /data/N/dfs/snn for N = 1, 2, 3... dfs.namenode.checkpoint.dir fs_checkpoint_dir_list true

Performance

Display Name Description Related Name Default Value API Name Required
Maximum Process File Descriptors If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value. rlimit_fds false

Ports and Addresses

Display Name Description Related Name Default Value API Name Required
SecondaryNameNode Web UI Port The SecondaryNameNode HTTP port. If the port is 0, then the server starts on a free port. Combined with the SecondaryNameNode's hostname to build its HTTP address. dfs.namenode.secondary.http-address 9868 dfs_secondary_http_port false
Secure SecondaryNameNode Web UI Port (TLS/SSL) The base port where the secure SecondaryNameNode web UI listens. dfs.secondary.https.port 9869 dfs_secondary_https_port false
Bind SecondaryNameNode to Wildcard Address If enabled, the SecondaryNameNode binds to the wildcard address ("0.0.0.0") on all of its ports. false secondary_namenode_bind_wildcard false

Resource Management

Display Name Description Related Name Default Value API Name Required
Cgroup CPU Shares Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager. cpu.shares 1024 rm_cpu_shares true
Cgroup I/O Weight Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager. blkio.weight 500 rm_io_weight true
Cgroup Memory Hard Limit Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.limit_in_bytes -1 MiB rm_memory_hard_limit true
Cgroup Memory Soft Limit Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 to specify no limit. By default processes not managed by Cloudera Manager will have no limit. If the value is -1, Cloudera Manager will not monitor Cgroup memory usage therefore some of the charts will show 'No Data' memory.soft_limit_in_bytes -1 MiB rm_memory_soft_limit true
Java Heap Size of Secondary NameNode in Bytes Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx. 4 GiB secondary_namenode_java_heapsize false

Stacks Collection

Display Name Description Related Name Default Value API Name Required
Stacks Collection Data Retention The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted. stacks_collection_data_retention 100 MiB stacks_collection_data_retention false
Stacks Collection Directory The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory. stacks_collection_directory stacks_collection_directory false
Stacks Collection Enabled Whether or not periodic stacks collection is enabled. stacks_collection_enabled false stacks_collection_enabled true
Stacks Collection Frequency The frequency with which stacks are collected. stacks_collection_frequency 5.0 second(s) stacks_collection_frequency false
Stacks Collection Method The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped. stacks_collection_method jstack stacks_collection_method false

Suppressions

Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: Version Validator Whether to suppress configuration warnings produced by the Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: SecondaryNameNode Web UI Port Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Web UI Port parameter. false role_config_suppression_dfs_secondary_http_port true
Suppress Parameter Validation: Secure SecondaryNameNode Web UI Port (TLS/SSL) Whether to suppress configuration warnings produced by the built-in parameter validation for the Secure SecondaryNameNode Web UI Port (TLS/SSL) parameter. false role_config_suppression_dfs_secondary_https_port true
Suppress Parameter Validation: SecondaryNameNode Nameservice Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Nameservice parameter. false role_config_suppression_dfs_secondarynamenode_nameservice true
Suppress Parameter Validation: HDFS Checkpoint Directories Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Checkpoint Directories parameter. false role_config_suppression_fs_checkpoint_dir_list true
Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_hadoop_metrics2_safety_valve true
Suppress Parameter Validation: SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_log4j_safety_valve true
Suppress Parameter Validation: Rules to Extract Events from Log Files Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter. false role_config_suppression_log_event_whitelist true
Suppress Parameter Validation: Heap Dump Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter. false role_config_suppression_oom_heap_dump_dir true
Suppress Parameter Validation: Role Triggers Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter. false role_config_suppression_role_triggers true
Suppress Parameter Validation: SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter. false role_config_suppression_secondarynamenode_config_safety_valve true
Suppress Parameter Validation: Java Configuration Options for Secondary NameNode Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Secondary NameNode parameter. false role_config_suppression_secondarynamenode_java_opts true
Suppress Parameter Validation: SecondaryNameNode Log Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Log Directory parameter. false role_config_suppression_secondarynamenode_log_dir true
Suppress Parameter Validation: SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve) parameter. false role_config_suppression_secondarynamenode_role_env_safety_valve true
Suppress Parameter Validation: Stacks Collection Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter. false role_config_suppression_stacks_collection_directory true
Suppress Health Test: Audit Pipeline Test Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_audit_health true
Suppress Health Test: HDFS Checkpoint Directories Free Space Whether to suppress the results of the HDFS Checkpoint Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_checkpoint_directories_free_space true
Suppress Health Test: File Descriptors Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_file_descriptor true
Suppress Health Test: GC Duration Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_gc_duration true
Suppress Health Test: Heap Dump Directory Free Space Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_heap_dump_directory_free_space true
Suppress Health Test: Host Health Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_host_health true
Suppress Health Test: Log Directory Free Space Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_log_directory_free_space true
Suppress Health Test: Process Status Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_scm_health true
Suppress Health Test: Swap Memory Usage Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_swap_memory_usage true
Suppress Health Test: Unexpected Exits Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_unexpected_exits true
Suppress Health Test: Web Server Status Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts. false role_health_suppression_secondary_name_node_web_metric_collection true

Service-Wide

Categories:

Advanced

Display Name Description Related Name Default Value API Name Required
Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml For advanced use only, a string to be inserted into core-site.xml. Applies to all roles and client configurations in this HDFS service as well as all its dependent services. Any configs added here will be overridden by their default values in HDFS (which can be found in hdfs-default.xml). core_site_safety_valve false
Enable HDFS Block Metadata API Enables DataNode support for the experimental DistributedFileSystem.getFileVBlockStorageLocations API. Applicable to CDH 4.1 and onwards. dfs.datanode.hdfs-blocks-metadata.enabled true dfs_datanode_hdfs_blocks_metadata_enabled false
HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml For advanced use only, a string to be inserted into hadoop-policy.xml. Applies to configurations of all roles in this service except client configuration. hadoop_policy_config_safety_valve false
Shared Hadoop Group Name The name of the system group shared by all the core Hadoop services. hadoop hdfs_hadoop_group_name true
HDFS Replication Advanced Configuration Snippet (Safety Valve) for core-site.xml For advanced use only, a string to be inserted into core-site.xml. Applies to all HDFS Replication jobs. hdfs_replication_core_site_safety_valve false
HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into the environment of HDFS replication jobs. hdfs_replication_env_safety_valve false
HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh For advanced use only. Key-value pairs (one on each line) to be inserted into the HDFS replication configuration for hadoop-env.sh. hdfs_replication_haoop_env_sh_safety_valve false
HDFS Replication Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only, a string to be inserted into hdfs-site.xml. Applies to all HDFS Replication jobs. hdfs_replication_hdfs_site_safety_valve false
HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml For advanced use only, a string to be inserted into hdfs-site.xml. Applies to configurations of all roles in this service except client configuration. hdfs_service_config_safety_valve false
HDFS Service Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration. hdfs_service_env_safety_valve false
HDFS Snapshot Shell Command Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into the environment of HDFS snapshot shell command. hdfs_shell_cmd_env_safety_valve false
HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml For advanced use only, a string to be inserted into ssl-client.xml. Applies cluster-wide, but can be overridden by individual services. hdfs_ssl_client_safety_valve false
HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration. hdfs_ssl_server_safety_valve false
System User's Home Directory The home directory of the system user on the local filesystem. This setting must reflect the system's configured value - only changing it here will not change the actual home directory. /var/lib/hadoop-hdfs hdfs_user_home_dir true
HDFS Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties For advanced use only, a string to be inserted into the client configuration for navigator.client.properties. navigator_client_config_safety_valve false
System Group The group that this service's processes should run as (except the HttpFS server, which has its own group) hdfs process_groupname true
System User The user that this service's processes should run as (except the HttpFS server, which has its own user) hdfs process_username true
HDFS Service Advanced Configuration Snippet (Safety Valve) for ranger-hdfs-audit.xml For advanced use only, a string to be inserted into ranger-hdfs-audit.xml. Applies to configurations of all roles in this service except client configuration. ranger_audit_safety_valve false
HDFS Service Advanced Configuration Snippet (Safety Valve) for ranger-hdfs-security.xml For advanced use only, a string to be inserted into ranger-hdfs-security.xml. Applies to configurations of all roles in this service except client configuration. ranger_security_safety_valve false

Cloudera Navigator

Display Name Description Related Name Default Value API Name Required
Enable Audit Collection Enable collection of audit events from the service's roles. navigator.audit.enabled true navigator_audit_enabled false
Audit Event Filter Event filters are defined in a JSON object like the following: { "defaultAction" : ("accept", "discard"), "rules" : [ { "action" : ("accept", "discard"), "fields" : [ { "name" : "fieldName", "match" : "regex" } ] } ] } A filter has a default action and a list of rules, in order of precedence. Each rule defines an action, and a list of fields to match against the audit event. A rule is "accepted" if all the listed field entries match the audit event. At that point, the action declared by the rule is taken. If no rules match the event, the default action is taken. Actions default to "accept" if not defined in the JSON object. The following is the list of fields that can be filtered for HDFS events:
  • username: the user performing the action.
  • ipAddress: the IP from where the request originated.
  • command: the HDFS operation being performed.
  • src: the source path for the operation.
  • dest: the destination path for the operation.
  • permissions: the permissions associated with the operation.
The default HDFS audit event filter accepts all denied access, delete and rename events, and discards events that affects files in any of the staging directories (Hive, Spark, Impala), events that affect files in /tmp directory, events that affect files in Cloudera Hive Canary directory, events generated by the internal Cloudera and Hadoop users (cloudera-scm, dr.who, hbase, hive, impala, mapred, solr, and spark), and 'ls' actions performed by the hdfs user.
navigator.event.filter comment: [ The default HDFS audit event filter accepts all denied access, delete , and rename events, and discards events that affects files in any of the , staging directories (Hive, Spark, Impala), events that affect files in /tmp , directory, events that affect files in Cloudera Hive Canary directory, , events generated by the internal Cloudera and Hadoop users (cloudera-scm, , dr.who, hbase, hive, impala, mapred, solr, and spark), and \u0027ls\u0027 actions , performed by the hdfs user. ], defaultAction: accept, rules: [ action: accept, fields: [ name: allowed, match: (?:false) ] , action: discard, fields: [ name: src, match: (?:.*/\\.hive-staging($|.*)?|.*/\\.staging($|/.*)?|.*/\\.sparkStaging($|/.*)?|.*/_impala_insert_staging($|/.*)?|/user/history/done_intermediate(?:/.*)?|/user/spark/spark2ApplicationHistory($|/.*)|/user/spark/applicationHistory($|/.*)|/user/hue/\\.cloudera_manager_hive_metastore_canary(?:/.*)?|/user/hue/\\.Trash/Current/user/hue/\\.cloudera_manager_hive_metastore_canary(?:/.*)?|/tmp(?:/.*)?) ] , action: accept, fields: [ name: operation, match: delete|rename.* ] , action: discard, fields: [ name: username, match: (?:cloudera-scm|dr.who|hbase|hive|impala|mapred|solr|spark)(?:/.+)? ] , action: discard, fields: [ name: username, match: (?:hdfs)(?:/.+)? , name: operation, match: (?:listStatus|listCachePools|listCacheDirectives|getfileinfo) ] , action: accept, fields: [ name: operation, match: (?:getfileinfo) ] ] navigator_audit_event_filter false
Audit Queue Policy Action to take when the audit event queue is full. Drop the event or shutdown the affected process. navigator.batch.queue_policy DROP navigator_audit_queue_policy false
Audit Event Tracker Configures the rules for event tracking and coalescing. This feature is used to define equivalency between different audit events. When events match, according to a set of configurable parameters, only one entry in the audit list is generated for all the matching events. Tracking works by keeping a reference to events when they first appear, and comparing other incoming events against the "tracked" events according to the rules defined here. Event trackers are defined in a JSON object like the following: { "timeToLive" : [integer], "fields" : [ { "type" : [string], "name" : [string] } ] } Where:
  • timeToLive: maximum amount of time an event will be tracked, in milliseconds. Must be provided. This defines how long, since it's first seen, an event will be tracked. A value of 0 disables tracking.
  • fields: list of fields to compare when matching events against tracked events.
Each field has an evaluator type associated with it. The evaluator defines how the field data is to be compared. The following evaluators are available:
  • value: uses the field value for comparison.
  • username: treats the field value as a user name, and ignores any host-specific data. This is useful for environment using Kerberos, so that only the principal name and realm are compared.
The following is the list of fields that can be used to compare HDFS events:
  • operation: the HDFS operation being performed.
  • username: the user performing the action.
  • ipAddress: the IP from where the request originated.
  • allowed: whether the operation was allowed or denied.
  • src: the source path for the operation.
  • dest: the destination path for the operation.
  • permissions: the permissions associated with the operation.
The default event tracker for HDFS services defines equality by comparing the username, operation, and source path of the events.
navigator_event_tracker comment: [ The default event tracker for HDFS services defines equality by , comparing the username, operation, and source path of the events. ], timeToLive: 60000, fields: [ type: value, name: src , type: value, name: operation , type: username, name: username ] navigator_event_tracker false

High Availability

Display Name Description Related Name Default Value API Name Required
Timeout for Cloudera Manager Fencing Strategy The timeout, in milliseconds, to use with the Cloudera Manager agent-based fencer. dfs.ha.fencing.cloudera_manager.timeout_millis 10000 dfs_ha_fencing_cloudera_manager_timeout_millis false
HDFS High Availability Fencing Methods List of fencing methods to use for service fencing. Setting this to shell(true) enables the built-in HDFS fencing mechanism, which causes the NameNode to exit if it attempts a write operation when it is not active. In almost all cases, this is the best choice. The sshfence method uses SSH. If using custom fencers (that may communicate with shared store, power units, or network switches), use the shell to invoke them. dfs.ha.fencing.methods shell(true) dfs_ha_fencing_methods false
Timeout for SSH Fencing Strategy SSH connection timeout, in milliseconds, to use with the built-in sshfence fencer. dfs.ha.fencing.ssh.connect-timeout 30 second(s) dfs_ha_fencing_ssh_connect_timeout false
Private Keys for SSH Fencing Strategy The SSH private key files to use with the built-in sshfence fencer. These are to be accessible to the hdfs user on the machines running the NameNodes. dfs.ha.fencing.ssh.private-key-files dfs_ha_fencing_ssh_private_key_files false
FailoverProxyProvider Class Enter a FailoverProxyProvider implementation to configure two URIs to connect to during fail-over. The first configured address is tried first, and on a fail-over event the other address is tried. dfs.client.failover.proxy.provider org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs_ha_proxy_provider true

Logs

Display Name Description Related Name Default Value API Name Required
Audit Log Directory Path to the directory where audit logs will be written. The directory will be created if it doesn't exist. audit_event_log_dir /var/log/hadoop-hdfs/audit audit_event_log_dir false
Maximum Audit Log File Size Maximum size of audit log file in MB before it is rolled over. navigator.audit_log_max_file_size 100 MiB navigator_audit_log_max_file_size false
Number of Audit Logs to Retain Maximum number of rolled-over audit logs to retain. The logs are not deleted if they contain audit events that have not yet been propagated to the Audit Server. navigator.client.max_num_audit_log 10 navigator_client_max_num_audit_log false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Log Event Capture When set, each role identifies important log events and forwards them to Cloudera Manager. true catch_events false
Enable Service Level Health Alerts When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Failover Controllers Healthy Enables the health check that verifies that the failover controllers associated with this service are healthy and running. true failover_controllers_healthy_enabled false
HDFS Health Canary Directory The service monitor will use this directory to create files to test if the hdfs service is healthy. The directory and files are created with permissions specified by 'HDFS Health Canary Directory Permissions' /tmp/.cloudera_health_monitoring_canary_files firehose_hdfs_canary_directory false
HDFS Health Canary Directory Permissions The service monitor will use these permissions to create the directory and files to test if the hdfs service is healthy. Permissions are specified using the 10-character unix-symbolic format e.g. '-rwxr-xr-x'. -rwxrwxrwx firehose_hdfs_canary_directory_permissions false
Active NameNode Detection Window The tolerance window that will be used in HDFS service tests that depend on detection of the active NameNode. 3 minute(s) hdfs_active_namenode_detection_window false
Blocks With Corrupt Replicas Monitoring Thresholds The health check thresholds of the number of blocks that have at least one corrupt replica. Specified as a percentage of the total number of blocks. Warning: 0.5 %, Critical: 1.0 % hdfs_blocks_with_corrupt_replicas_thresholds false
HDFS Canary Health Check Enables the health check that a client can create, read, write, and delete files true hdfs_canary_health_enabled false
Healthy DataNode Monitoring Thresholds The health test thresholds of the overall DataNode health. The check returns "Concerning" health if the percentage of "Healthy" DataNodes falls below the warning threshold. The check is unhealthy if the total percentage of "Healthy" and "Concerning" DataNodes falls below the critical threshold. Warning: 95.0 %, Critical: 90.0 % hdfs_datanodes_healthy_thresholds false
HDFS Free Space Monitoring Thresholds The health check thresholds of free space in HDFS. Specified as a percentage of total HDFS capacity. Warning: 20.0 %, Critical: 10.0 % hdfs_free_space_thresholds false
Missing Block Monitoring Thresholds The health check thresholds of the number of missing blocks. Specified as a percentage of the total number of blocks. Warning: Never, Critical: Any hdfs_missing_blocks_thresholds false
NameNode Activation Startup Tolerance The amount of time after NameNode(s) start that the lack of an active NameNode will be tolerated. This is intended to allow either the auto-failover daemon to make a NameNode active, or a specifically issued failover command to take effect. This is an advanced option that does not often need to be changed. 3 minute(s) hdfs_namenode_activation_startup_tolerance false
Active NameNode Role Health Check When computing the overall HDFS cluster health, consider the active NameNode's health true hdfs_namenode_health_enabled false
Standby NameNode Health Check When computing the overall HDFS cluster health, consider the health of the standby NameNode. true hdfs_standby_namenodes_health_enabled false
Under-replicated Block Monitoring Thresholds The health check thresholds of the number of under-replicated blocks. Specified as a percentage of the total number of blocks. Warning: 10.0 %, Critical: 40.0 % hdfs_under_replicated_blocks_thresholds false
Erasure Coding Policy Verification Health Check Enables the health test for verifying if the cluster topology supports all the enabled erasure coding policies. false hdfs_verify_ec_with_topology_enabled false
Log Event Retry Frequency The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds 30 log_event_retry_frequency false
Service Triggers The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.
For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad", "streamThreshold": 10, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.
[] service_triggers true
Service Monitor Client Config Overrides For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service. <property> <name>dfs.socket.timeout</name> <value>3000</value> </property> <property> <name>dfs.datanode.socket.write.timeout</name> <value>3000</value> </property> <property> <name>ipc.client.connect.max.retries</name> <value>1</value> </property> <property> <name>fs.permissions.umask-mode</name> <value>000</value> </property> smon_client_config_overrides false
Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones. smon_derived_configs_safety_valve false

Other

Display Name Description Related Name Default Value API Name Required
HDFS Block Size The default block size in bytes for new HDFS files. Note that this value is also used as the HBase Region Server HLog block size. dfs.blocksize 128 MiB dfs_block_size false
Check HDFS Permissions If false, permission checking is turned off for files in HDFS. dfs.permissions true dfs_permissions false
Default Umask Default umask for file and directory creation, specified in an octal value (with a leading 0) fs.permissions.umask-mode 022 dfs_umaskmode false
Enable WebHDFS Enable WebHDFS interface dfs.webhdfs.enabled true dfs_webhdfs_enabled false
Serve logs over HTTP Whether to serve logs over HTTP from HDFS web servers. This includes listing the logs directory at the /logs endpoint, which may be a security concern. hadoop.http.logs.enabled true http_logs_enabled false
Compression Codecs Comma-separated list of compression codecs that can be used in job or map compression. io.compression.codecs org.apache.hadoop.io.compress.DefaultCodec org.apache.hadoop.io.compress.GzipCodec org.apache.hadoop.io.compress.BZip2Codec org.apache.hadoop.io.compress.DeflateCodec org.apache.hadoop.io.compress.SnappyCodec org.apache.hadoop.io.compress.Lz4Codec io_compression_codecs false
KMS Service The Key Management Server used by HDFS. This must be set to use encryption for data at rest. kms_service false
Object Store Service Select an Object Store service to enable cloud storage support. Once enabled, the cloud storage can be used in Impala and Hue services, via fully-qualified URIs. object_store_service false
ZooKeeper Service Name of the ZooKeeper service that this HDFS service instance depends on zookeeper_service false

Performance

Display Name Description Related Name Default Value API Name Required
DataNode Local Path Access Users Comma separated list of users allowed to do short circuit read. A short circuit read allows a client co-located with the data to read HDFS file blocks directly from HDFS. If empty, will default to the DataNode process' user. dfs.block.local-path-access.user dfs_block_local_path_access_user false
HDFS File Block Storage Location Timeout Timeout in milliseconds for the parallel RPCs made in DistributedFileSystem#getFileBlockStorageLocations(). This value is only emitted for Impala. dfs.client.file-block-storage-locations.timeout.millis 10 second(s) dfs_client_file_block_storage_locations_timeout false
Enable HDFS Short-Circuit Read Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality. dfs.client.read.shortcircuit true dfs_datanode_read_shortcircuit false
UNIX Domain Socket path Path on the DataNode's local file system to a UNIX domain socket used for communication between the DataNode and local HDFS clients. This socket is used for Short Circuit Reads. Only the HDFS System User and "root" should have write access to the parent directory and all of its ancestors. This property is supported in CDH 4.2 or later deployments. dfs.domain.socket.path /var/run/hdfs-sockets/dn dfs_domain_socket_path false
FsImage Transfer Bandwidth Maximum bandwidth used for image transfer in bytes per second. This can help keep normal NameNode operations responsive during checkpointing. A default value of 0 indicates that throttling is disabled. dfs.image.transfer.bandwidthPerSec 0 B dfs_image_transfer_bandwidthPerSec false
FsImage Transfer Socket Timeout Socket timeout for the HttpURLConnection instance used in the image transfer. This is measured in milliseconds. This timeout prevents client hangs if the connection is idle for this configured timeout, during image transfer. dfs.image.transfer.timeout 1 minute(s) dfs_image_transfer_timeout false

Ports and Addresses

Display Name Description Related Name Default Value API Name Required
Use DataNode Hostname Typically, HDFS clients and servers communicate by opening sockets via an IP address. In certain networking configurations, it is preferable to open sockets after doing a DNS lookup on the hostname. Enable this property to open sockets after doing a DNS lookup on the hostname. This property is supported in CDH3u4 or later deployments. dfs.client.use.datanode.hostname false dfs_client_use_datanode_hostname false

Proxy

Display Name Description Related Name Default Value API Name Required
HDFS Proxy User Groups Comma-delimited list of groups to allow the HDFS user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'. hadoop.proxyuser.hdfs.groups * hdfs_proxy_user_groups_list false
HDFS Proxy User Hosts Comma-delimited list of hosts where you want to allow the HDFS user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. hadoop.proxyuser.hdfs.hosts * hdfs_proxy_user_hosts_list false
Hive Proxy User Groups Comma-delimited list of groups that you want to allow the Hive user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. hadoop.proxyuser.hive.groups * hive_proxy_user_groups_list false
Hive Proxy User Hosts Comma-delimited list of hosts where you want to allow the Hive user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. hadoop.proxyuser.hive.hosts * hive_proxy_user_hosts_list false
HTTP Proxy User Groups Comma-delimited list of groups that you want to allow the HTTP user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. This is used by WebHCat. hadoop.proxyuser.HTTP.groups * HTTP_proxy_user_groups_list false
HTTP Proxy User Hosts Comma-delimited list of hosts where you want to allow the HTTP user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. This is used by WebHCat. hadoop.proxyuser.HTTP.hosts * HTTP_proxy_user_hosts_list false
HttpFS Proxy User Groups Comma-delimited list of groups to allow the HttpFS user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'. hadoop.proxyuser.httpfs.groups * httpfs_proxy_user_groups_list false
HttpFS Proxy User Hosts Comma-delimited list of hosts where you allow the HttpFS user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. hadoop.proxyuser.httpfs.hosts * httpfs_proxy_user_hosts_list false
Hue Proxy User Groups Comma-delimited list of groups that you want to allow the Hue user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. hadoop.proxyuser.hue.groups * hue_proxy_user_groups_list false
Hue Proxy User Hosts Comma-delimited list of hosts where you want to allow the Hue user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. hadoop.proxyuser.hue.hosts * hue_proxy_user_hosts_list false
Impala Proxy User Groups Comma-delimited list of groups that you want to allow the Impala user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. hadoop.proxyuser.impala.groups * impala_proxy_user_groups_list false
Impala Proxy User Hosts Comma-delimited list of hosts where you want to allow the Impala user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. hadoop.proxyuser.impala.hosts * impala_proxy_user_hosts_list false
Knox Proxy User Groups Comma-delimited list of groups that you want to allow the Knox user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. hadoop.proxyuser.knox.groups * knox_proxy_user_groups_list false
Knox Proxy User Hosts Comma-delimited list of hosts where you want to allow the Knox user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. hadoop.proxyuser.knox.hosts * knox_proxy_user_hosts_list false
Livy Proxy User Groups Comma-delimited list of groups that you want to allow the Livy user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. hadoop.proxyuser.livy.groups * livy_proxy_user_groups_list false
Livy Proxy User Hosts Comma-delimited list of hosts where you want to allow the Livy user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. hadoop.proxyuser.livy.hosts * livy_proxy_user_hosts_list false
Oozie Proxy User Groups Allows the oozie superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. hadoop.proxyuser.oozie.groups * oozie_proxy_user_groups_list false
Oozie Proxy User Hosts Comma-delimited list of hosts where you want to allow the oozie user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. hadoop.proxyuser.oozie.hosts * oozie_proxy_user_hosts_list false
Service Monitor Proxy User Groups Allows the Cloudera Service Monitor user to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. This property is used only if Service Monitor is using a different Kerberos principal than the Hue service. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'. hadoop.proxyuser.smon.groups * smon_proxy_user_groups_list false
Service Monitor Proxy User Hosts