Spark (Standalone) Properties in CDH 5.4.0

gatewaydefaultgroup

Advanced

Display Name	Description	Related Name	Default Value	API Name	Required
Deploy Directory	The directory where the client configs will be deployed		/etc/spark	client_config_root_dir	true
Spark (Standalone) Client Advanced Configuration Snippet (Safety Valve) for spark-conf/log4j.properties	For advanced use only, a string to be inserted into the client configuration for spark-conf/log4j.properties.			spark-conf/log4j.properties_client_config_safety_valve	false
Spark (Standalone) Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf	For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-defaults.conf.			spark-conf/spark-defaults.conf_client_config_safety_valve	false
Spark (Standalone) Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh	For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-env.sh.			spark-conf/spark-env.sh_client_config_safety_valve	false

Monitoring

Display Name	Description	Related Name	Default Value	API Name	Required
Enable Configuration Change Alerts	When set, Cloudera Manager will send alerts when this entity's configuration changes.		false	enable_config_alerts	false

Other

Display Name	Description	Related Name	Default Value	API Name	Required
Alternatives Priority	The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.		50	client_config_priority	true
Enable History	Write Spark application history logs to HDFS.	spark.eventLog.enabled	true	spark_history_enabled	false

historyserverdefaultgroup

Advanced

Display Name	Description	Related Name	Default Value	API Name	Required
History Server Environment Advanced Configuration Snippet (Safety Valve)	For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.			SPARK_HISTORY_SERVER_role_env_safety_valve	false
History Server Advanced Configuration Snippet (Safety Valve) for config/spark-env.sh	For advanced use only, a string to be inserted into config/spark-env.sh for this role only.			config/spark-env.sh_role_safety_valve	false
History Server Logging Advanced Configuration Snippet (Safety Valve)	For advanced use only, a string to be inserted into log4j.properties for this role only.			log4j_safety_valve	false
Automatically Restart Process	When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.		false	process_auto_restart	true

Logs

Display Name	Description	Related Name	Default Value	API Name	Required
History Server Log Directory	The log directory for log files of the role History Server.	log_dir	/var/log/spark	log_dir	false
History Server Logging Threshold	The minimum log level for History Server logs		INFO	log_threshold	false
History Server Maximum Log File Backups	The maximum number of rolled log files to keep for History Server logs. Typically used by log4j or logback.		10	max_log_backup_index	false
History Server Max Log Size	The maximum size, in megabytes, per log file for History Server logs. Typically used by log4j or logback.		200 MiB	max_log_size	false

Monitoring

Display Name	Description	Related Name	Default Value	API Name	Required
Enable Health Alerts for this Role	When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold		true	enable_alerts	false
Enable Configuration Change Alerts	When set, Cloudera Manager will send alerts when this entity's configuration changes.		false	enable_config_alerts	false
Process Swap Memory Thresholds	The health test thresholds on the swap memory usage of the process.		Warning: Any, Critical: Never	process_swap_memory_thresholds	false
Role Triggers	The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has all of the following fields: triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role. triggerExpression (mandatory) - A tsquery expression representing the trigger. streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire. enabled (optional) - By default set to 'true'. If set to 'false', the trigger will not be evaluated. expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here may lead to inconsistencies. For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file-descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change in the future and, as a result, backward compatibility is not guaranteed between releases at this time.		[]	role_triggers	true
File Descriptor Monitoring Thresholds	The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.		Warning: 50.0 %, Critical: 70.0 %	spark_history_server_fd_thresholds	false
History Server Host Health Test	When computing the overall History Server health, consider the host's health.		true	spark_history_server_host_health_enabled	false
History Server Process Health Test	Enables the health test that the History Server's process state is consistent with the role configuration		true	spark_history_server_scm_health_enabled	false
Unexpected Exits Thresholds	The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.		Warning: Never, Critical: Any	unexpected_exits_thresholds	false
Unexpected Exits Monitoring Period	The period to review when computing unexpected exits.		5 minute(s)	unexpected_exits_window	false

Other

Display Name	Description	Related Name	Default Value	API Name	Required
Java Heap Size of History Server in Bytes	Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in bytes.	history_server_max_heapsize	256 MiB	history_server_max_heapsize	true

Performance

Display Name	Description	Related Name	Default Value	API Name	Required
Maximum Process File Descriptors	If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.			rlimit_fds	false

Ports and Addresses

Display Name	Description	Related Name	Default Value	API Name	Required
History Server WebUI Port	The port of the history server WebUI	history.port	18088	history_server_web_port	true

Resource Management

Display Name	Description	Related Name	Default Value	API Name	Required
Cgroup CPU Shares	Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.	cpu.shares	1024	rm_cpu_shares	true
Cgroup I/O Weight	Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.	blkio.weight	500	rm_io_weight	true
Cgroup Memory Hard Limit	Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.	memory.limit_in_bytes	-1 MiB	rm_memory_hard_limit	true
Cgroup Memory Soft Limit	Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.	memory.soft_limit_in_bytes	-1 MiB	rm_memory_soft_limit	true

masterdefaultgroup

Advanced

Display Name	Description	Related Name	Default Value	API Name	Required
Master Environment Advanced Configuration Snippet (Safety Valve)	For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.			SPARK_MASTER_role_env_safety_valve	false
Master Advanced Configuration Snippet (Safety Valve) for config/spark-env.sh	For advanced use only, a string to be inserted into config/spark-env.sh for this role only.			config/spark-env.sh_role_safety_valve	false
Master Logging Advanced Configuration Snippet (Safety Valve)	For advanced use only, a string to be inserted into log4j.properties for this role only.			log4j_safety_valve	false
Automatically Restart Process	When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.		false	process_auto_restart	true

Logs

Display Name	Description	Related Name	Default Value	API Name	Required
Master Log Directory	The log directory for log files of the role Master.	log.dir	/var/log/spark	log_dir	false
Master Logging Threshold	The minimum log level for Master logs		INFO	log_threshold	false
Master Maximum Log File Backups	The maximum number of rolled log files to keep for Master logs. Typically used by log4j or logback.		10	max_log_backup_index	false
Master Max Log Size	The maximum size, in megabytes, per log file for Master logs. Typically used by log4j or logback.		200 MiB	max_log_size	false

Monitoring

Display Name	Description	Related Name	Default Value	API Name	Required
Enable Health Alerts for this Role	When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold		true	enable_alerts	false
Enable Configuration Change Alerts	When set, Cloudera Manager will send alerts when this entity's configuration changes.		false	enable_config_alerts	false
Process Swap Memory Thresholds	The health test thresholds on the swap memory usage of the process.		Warning: Any, Critical: Never	process_swap_memory_thresholds	false
Role Triggers	The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has all of the following fields: triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role. triggerExpression (mandatory) - A tsquery expression representing the trigger. streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire. enabled (optional) - By default set to 'true'. If set to 'false', the trigger will not be evaluated. expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here may lead to inconsistencies. For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file-descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change in the future and, as a result, backward compatibility is not guaranteed between releases at this time.		[]	role_triggers	true
File Descriptor Monitoring Thresholds	The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.		Warning: 50.0 %, Critical: 70.0 %	spark_master_fd_thresholds	false
Master Host Health Test	When computing the overall Master health, consider the host's health.		true	spark_master_host_health_enabled	false
Master Process Health Test	Enables the health test that the Master's process state is consistent with the role configuration		true	spark_master_scm_health_enabled	false
Unexpected Exits Thresholds	The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.		Warning: Never, Critical: Any	unexpected_exits_thresholds	false
Unexpected Exits Monitoring Period	The period to review when computing unexpected exits.		5 minute(s)	unexpected_exits_window	false

Other

Display Name	Description	Related Name	Default Value	API Name	Required
Additional Master args	Additional arguments for the master	additional.args		master_additional_args	false
Master Address	Overrides the address where the Master will listen for connections. Note: Spark will not work if the wildcard (0.0.0.0) address is used here.	server.address		master_address	false
Java Heap Size of Master in Bytes	Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in bytes.	master_max_heapsize	512 MiB	master_max_heapsize	true

Performance

Display Name	Description	Related Name	Default Value	API Name	Required
Maximum Process File Descriptors	If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.			rlimit_fds	false

Ports and Addresses

Display Name	Description	Related Name	Default Value	API Name	Required
Master Port	The port of the master	server.port	7077	master_port	true
Master WebUI Port	The port of the master WebUI	webui.port	18080	master_webui_port	true

Resource Management

Display Name	Description	Related Name	Default Value	API Name	Required
Cgroup CPU Shares	Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.	cpu.shares	1024	rm_cpu_shares	true
Cgroup I/O Weight	Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.	blkio.weight	500	rm_io_weight	true
Cgroup Memory Hard Limit	Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.	memory.limit_in_bytes	-1 MiB	rm_memory_hard_limit	true
Cgroup Memory Soft Limit	Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.	memory.soft_limit_in_bytes	-1 MiB	rm_memory_soft_limit	true

service_wide

Advanced

Display Name	Description	Related Name	Default Value	API Name	Required
Spark (Standalone) Service Environment Advanced Configuration Snippet (Safety Valve)	For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.			SPARK_service_env_safety_valve	false
Spark (Standalone) Service Advanced Configuration Snippet (Safety Valve) for config/spark-env.sh	For advanced use only, a string to be inserted into config/spark-env.sh. Applies to configurations of all roles in this service except client configuration.			config/spark-env.sh_service_safety_valve	false
System Group	The group that this service's processes should run as.		spark	process_groupname	true
System User	The user that this service's processes should run as.		spark	process_username	true

Monitoring

Display Name	Description	Related Name	Default Value	API Name	Required
Enable Service Level Health Alerts	When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold		true	enable_alerts	false
Enable Configuration Change Alerts	When set, Cloudera Manager will send alerts when this entity's configuration changes.		false	enable_config_alerts	false
Service Triggers	The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has all of the following fields: triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service. triggerExpression (mandatory) - A tsquery expression representing the trigger. streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire. enabled (optional) - By default set to 'true'. If set to 'false', the trigger will not be evaluated. expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here may lead to inconsistencies. For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file-descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad", "streamThreshold": 10, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change in the future and, as a result, backward compatibility is not guaranteed between releases at this time.		[]	service_triggers	true
Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)	For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.			smon_derived_configs_safety_valve	false

Other

Display Name	Description	Related Name	Default Value	API Name	Required
HDFS Service	Name of the HDFS service that this Spark (Standalone) service instance depends on			hdfs_service	false
Identity String	The identity string of the Spark instance	identity.string		identity_string	false
Spark History Location (HDFS)	The location of Spark application history logs in HDFS. Changing this value will not move existing logs to the new location.	spark.eventLog.dir	/user/spark/applicationHistory	spark_history_log_dir	false
Spark Jar Location (HDFS)	The location of the Spark jar in HDFS	spark_jar_hdfs_path	/user/spark/share/lib/spark-assembly.jar	spark_jar_hdfs_path	false

workerdefaultgroup

Advanced

Display Name	Description	Related Name	Default Value	API Name	Required
Worker Environment Advanced Configuration Snippet (Safety Valve)	For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.			SPARK_WORKER_role_env_safety_valve	false
Worker Advanced Configuration Snippet (Safety Valve) for config/spark-env.sh	For advanced use only, a string to be inserted into config/spark-env.sh for this role only.			config/spark-env.sh_role_safety_valve	false
Worker Logging Advanced Configuration Snippet (Safety Valve)	For advanced use only, a string to be inserted into log4j.properties for this role only.			log4j_safety_valve	false
Automatically Restart Process	When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.		false	process_auto_restart	true

Logs

Display Name	Description	Related Name	Default Value	API Name	Required
Worker Log Directory	The log directory for log files of the role Worker.	log.dir	/var/log/spark	log_dir	false
Worker Logging Threshold	The minimum log level for Worker logs		INFO	log_threshold	false
Worker Maximum Log File Backups	The maximum number of rolled log files to keep for Worker logs. Typically used by log4j or logback.		10	max_log_backup_index	false
Worker Max Log Size	The maximum size, in megabytes, per log file for Worker logs. Typically used by log4j or logback.		200 MiB	max_log_size	false

Monitoring

Display Name	Description	Related Name	Default Value	API Name	Required
Enable Health Alerts for this Role	When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold		true	enable_alerts	false
Enable Configuration Change Alerts	When set, Cloudera Manager will send alerts when this entity's configuration changes.		false	enable_config_alerts	false
Process Swap Memory Thresholds	The health test thresholds on the swap memory usage of the process.		Warning: Any, Critical: Never	process_swap_memory_thresholds	false
Role Triggers	The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has all of the following fields: triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role. triggerExpression (mandatory) - A tsquery expression representing the trigger. streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire. enabled (optional) - By default set to 'true'. If set to 'false', the trigger will not be evaluated. expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here may lead to inconsistencies. For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file-descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change in the future and, as a result, backward compatibility is not guaranteed between releases at this time.		[]	role_triggers	true
File Descriptor Monitoring Thresholds	The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.		Warning: 50.0 %, Critical: 70.0 %	spark_worker_fd_thresholds	false
Worker Host Health Test	When computing the overall Worker health, consider the host's health.		true	spark_worker_host_health_enabled	false
Worker Process Health Test	Enables the health test that the Worker's process state is consistent with the role configuration		true	spark_worker_scm_health_enabled	false
Unexpected Exits Thresholds	The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.		Warning: Never, Critical: Any	unexpected_exits_thresholds	false
Unexpected Exits Monitoring Period	The period to review when computing unexpected exits.		5 minute(s)	unexpected_exits_window	false

Other

Display Name	Description	Related Name	Default Value	API Name	Required
Total Java Heap Sizes of Worker's Executors in Bytes	Memory available to the Worker's Executors. This is the maximum sum total of all the Executors' Java heap sizes on this Worker node. Passed to Java -Xmx. Measured in bytes.	executor_total_max_heapsize	8 GiB	executor_total_max_heapsize	true
Work directory	The work directory for temporary files	work.dir	/var/run/spark/work	work_directory	true
Additional Worker args	Additional arguments for the workers	additional.args		worker_additional_args	false
Java Heap Size of Worker in Bytes	Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in bytes.	worker_max_heapsize	512 MiB	worker_max_heapsize	true

Performance

Display Name	Description	Related Name	Default Value	API Name	Required
Maximum Process File Descriptors	If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.			rlimit_fds	false

Ports and Addresses

Display Name	Description	Related Name	Default Value	API Name	Required
Worker Port	The port of the worker	server.port	7078	worker_port	true
Worker WebUI Port	The port of the worker WebUI	webui.port	18081	worker_webui_port	true

Resource Management

Display Name	Description	Related Name	Default Value	API Name	Required
Cgroup CPU Shares	Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.	cpu.shares	1024	rm_cpu_shares	true
Cgroup I/O Weight	Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.	blkio.weight	500	rm_io_weight	true
Cgroup Memory Hard Limit	Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.	memory.limit_in_bytes	-1 MiB	rm_memory_hard_limit	true
Cgroup Memory Soft Limit	Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.	memory.soft_limit_in_bytes	-1 MiB	rm_memory_soft_limit	true