Gateway
Advanced
Deploy Directory
- Description
- The directory where the client configs will be deployed
- Related Name
- Default Value
- /etc/spark
- API Name
-
client_config_root_dir
- Required
- true
Gateway Logging Advanced Configuration Snippet (Safety Valve)
- Description
- For advanced use only, a string to be inserted into log4j.properties for this role only.
- Related Name
- Default Value
- API Name
-
log4j_safety_valve
- Required
- false
Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf
- Description
- For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-defaults.conf.
- Related Name
- Default Value
- API Name
-
spark-conf/spark-defaults.conf_client_config_safety_valve
- Required
- false
Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh
- Description
- For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-env.sh.
- Related Name
- Default Value
- API Name
-
spark-conf/spark-env.sh_client_config_safety_valve
- Required
- false
Logs
Gateway Logging Threshold
- Description
- The minimum log level for Gateway logs
- Related Name
- Default Value
- INFO
- API Name
-
log_threshold
- Required
- false
Monitoring
Enable Configuration Change Alerts
- Description
- When set, Cloudera Manager will send alerts when this entity's configuration changes.
- Related Name
- Default Value
- false
- API Name
-
enable_config_alerts
- Required
- false
Other
Alternatives Priority
- Description
- The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.
- Related Name
- Default Value
- 51
- API Name
-
client_config_priority
- Required
- true
Spark Data Serializer
- Description
- Name of class implementing org.apache.spark.serializer.Serializer to use in Spark applications.
- Related Name
-
spark.serializer
- Default Value
- org.apache.spark.serializer.KryoSerializer
- API Name
-
spark_data_serializer
- Required
- true
Default Application Deploy Mode
- Description
- Which deploy mode to use by default. Can be overridden by users when launching applications.
- Related Name
-
spark_deploy_mode
- Default Value
- client
- API Name
-
spark_deploy_mode
- Required
- false
Caching Executor Idle Timeout
- Description
- When dynamic allocation is enabled, time after which idle executors with cached RDDs blocks will be stopped. By default, they're never stopped.
- Related Name
-
spark.dynamicAllocation.cachedExecutorIdleTimeout
- Default Value
- API Name
-
spark_dynamic_allocation_cached_idle_timeout
- Required
- false
Enable Dynamic Allocation
- Description
- Enable dynamic allocation of executors in Spark applications.
- Related Name
-
spark.dynamicAllocation.enabled
- Default Value
- true
- API Name
-
spark_dynamic_allocation_enabled
- Required
- false
Executor Idle Timeout
- Description
- When dynamic allocation is enabled, time after which idle executors will be stopped.
- Related Name
-
spark.dynamicAllocation.executorIdleTimeout
- Default Value
- 1 minute(s)
- API Name
-
spark_dynamic_allocation_idle_timeout
- Required
- false
Initial Executor Count
- Description
- When dynamic allocation is enabled, number of executors to allocate when the application starts. By default, this is the same value as the minimum number of executors.
- Related Name
-
spark.dynamicAllocation.initialExecutors
- Default Value
- API Name
-
spark_dynamic_allocation_initial_executors
- Required
- false
Maximum Executor Count
- Description
- When dynamic allocation is enabled, maximum number of executors to allocate. By default, Spark relies on YARN to control the maximum number of executors for the application.
- Related Name
-
spark.dynamicAllocation.maxExecutors
- Default Value
- API Name
-
spark_dynamic_allocation_max_executors
- Required
- false
Minimum Executor Count
- Description
- When dynamic allocation is enabled, minimum number of executors to keep alive while the application is running.
- Related Name
-
spark.dynamicAllocation.minExecutors
- Default Value
- 0
- API Name
-
spark_dynamic_allocation_min_executors
- Required
- false
Scheduler Backlog Timeout
- Description
- When dynamic allocation is enabled, timeout before requesting new executors when there are backlogged tasks.
- Related Name
-
spark.dynamicAllocation.schedulerBacklogTimeout
- Default Value
- 1 second(s)
- API Name
-
spark_dynamic_allocation_scheduler_backlog_timeout
- Required
- false
Sustained Scheduler Backlog Timeout
- Description
- When dynamic allocation is enabled, timeout before requesting new executors after the initial backlog timeout has already expired. By default this is the same value as the initial backlog timeout.
- Related Name
-
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout
- Default Value
- API Name
-
spark_dynamic_allocation_sustained_scheduler_backlog_timeout
- Required
- false
Shell Logging Threshold
- Description
- The minimum log level for the Spark shell.
- Related Name
-
spark_gateway_shell_logging_threshold
- Default Value
- WARN
- API Name
-
spark_gateway_shell_logging_threshold
- Required
- true
Enable Kill From UI
- Description
- Whether to allow users to kill running stages from the Spark Web UI.
- Related Name
-
spark.ui.killEnabled
- Default Value
- true
- API Name
-
spark_gateway_ui_kill_enabled
- Required
- true
Enable History
- Description
- Write Spark application history logs to HDFS.
- Related Name
-
spark.eventLog.enabled
- Default Value
- true
- API Name
-
spark_history_enabled
- Required
- false
Enable I/O Encryption
- Description
- Whether to encrypt temporary shuffle and cache files stored by Spark on the local disks.
- Related Name
-
spark.io.encryption.enabled
- Default Value
- false
- API Name
-
spark_io_encryption_enabled
- Required
- false
Enable Spark Lineage
- Description
- Whether to enable spark lineage support. If enabled, spark lineage is sent to Atlas.
- Related Name
-
spark.lineage.enabled
- Default Value
- true
- API Name
-
spark_lineage_enabled
- Required
- false
Enable Network Encryption
- Description
- Whether to encrypt communication between Spark processes belonging to the same application. Requires authentication (spark.authenticate) to be enabled.
- Related Name
-
spark.network.crypto.enabled
- Default Value
- false
- API Name
-
spark_network_encryption_enabled
- Required
- false
Enable Optimized S3 Committers
- Description
- Whether use optimized committers when writing data to S3.
- Related Name
-
spark.cloudera.s3_committers.enabled
- Default Value
- true
- API Name
-
spark_optimized_s3_committers_enabled
- Required
- false
Extra Python Path
- Description
- Python library paths to add to PySpark applications.
- Related Name
-
spark_python_path
- Default Value
- API Name
-
spark_python_path
- Required
- false
Enable Shuffle Service
- Description
- Enables the external shuffle service. The external shuffle service preserves shuffle files written by executors so that the executors can be deallocated without losing work. Must be enabled if Enable Dynamic Allocation is enabled. Recommended and enabled by default.
- Related Name
-
spark.shuffle.service.enabled
- Default Value
- true
- API Name
-
spark_shuffle_service_enabled
- Required
- true
Enable Spark Web UI
- Description
- Whether to enable the Spark Web UI on individual applications. It's recommended that the UI be disabled in secure clusters.
- Related Name
-
spark.ui.enabled
- Default Value
- true
- API Name
-
spark_ui_enabled
- Required
- false
Suppressions
Suppress Configuration Validator: CDH Version Validator
- Description
- Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_cdh_version_validator
- Required
- true
Suppress Parameter Validation: Deploy Directory
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_client_config_root_dir
- Required
- true
Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_log4j_safety_valve
- Required
- true
Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_spark-conf/spark-defaults.conf_client_config_safety_valve
- Required
- true
Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_spark-conf/spark-env.sh_client_config_safety_valve
- Required
- true
Suppress Parameter Validation: Spark Data Serializer
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Data Serializer parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_spark_data_serializer
- Required
- true
Suppress Parameter Validation: Extra Python Path
- Description
- Whether to suppress configuration warnings produced by the built-in parameter validation for the Extra Python Path parameter.
- Related Name
- Default Value
- false
- API Name
-
role_config_suppression_spark_python_path
- Required
- true