Tez Properties in Cloudera Runtime 7.1.2
Role groups:
Gateway
Categories:
Advanced
| Display Name | Description | Property Name | Default Value | API Name | Required |
|---|---|---|---|---|---|
| Deploy Directory | The directory where the client configs will be deployed | /etc/tez |
client_config_root_dir
|
true | |
| Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml | For advanced use only, a string to be inserted into the client configuration for tez-conf/tez-site.xml. |
tez-conf/tez-site.xml_client_config_safety_valve
|
false |
Monitoring
| Display Name | Description | Property Name | Default Value | API Name | Required |
|---|---|---|---|---|---|
| Enable Configuration Change Alerts | When set, Cloudera Manager will send alerts when this entity's configuration changes. | false |
enable_config_alerts
|
false |
Other
| Display Name | Description | Property Name | Default Value | API Name | Required |
|---|---|---|---|---|---|
| Alternatives Priority | The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others. | 50 |
client_config_priority
|
true |
Suppressions
| Display Name | Description | Property Name | Default Value | API Name | Required |
|---|---|---|---|---|---|
| Suppress Configuration Validator: CDH Version Validator | Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator. | false |
role_config_suppression_cdh_version_validator
|
true | |
| Suppress Parameter Validation: Deploy Directory | Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter. | false |
role_config_suppression_client_config_root_dir
|
true | |
| Suppress Parameter Validation: Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml parameter. | false |
role_config_suppression_tez-conf/tez-site.xml_client_config_safety_valve
|
true |
Service-Wide
Categories:
Advanced
| Display Name | Description | Property Name | Default Value | API Name | Required |
|---|---|---|---|---|---|
| System Group | The group that this service's processes should run as. | hadoop |
process_groupname
|
true | |
| System User | The user that this service's processes should run as. | tez |
process_username
|
true | |
| Tez Service Environment Advanced Configuration Snippet (Safety Valve) | For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration. |
TEZ_service_env_safety_valve
|
false |
Monitoring
| Display Name | Description | Property Name | Default Value | API Name | Required |
|---|---|---|---|---|---|
| Enable Service Level Health Alerts | When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold | true |
enable_alerts
|
false | |
| Enable Configuration Change Alerts | When set, Cloudera Manager will send alerts when this entity's configuration changes. | false |
enable_config_alerts
|
false | |
| Service Triggers | The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
[{"triggerName": "sample-trigger",
"triggerExpression": "IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad",
"streamThreshold": 10, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases. |
[] |
service_triggers
|
true | |
| Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) | For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones. |
smon_derived_configs_safety_valve
|
false |
Other
| Display Name | Description | Property Name | Default Value | API Name | Required |
|---|---|---|---|---|---|
| Maximum Heartbeat Interval | The maximum heartbeat interval between the Application Master and RM in milliseconds. |
tez.am.am-rm.heartbeat.interval-ms.max
|
250 millisecond(s) |
tez.am.am-rm.heartbeat.interval-ms.max
|
true |
| Maximum Timeout to Hold Idle Containers | The maximum amount of time to hold on to a container if no task can be assigned to it immediately. Only active when reuse is enabled. |
tez.am.container.idle.release-timeout-max.millis
|
20 second(s) |
tez.am.container.idle.release-timeout-max.millis
|
true |
| Minimum Timeout to Hold Idle Containers | The minimum amount of time to hold on to a container that is idle. Only active when reuse is enabled. |
tez.am.container.idle.release-timeout-min.millis
|
10 second(s) |
tez.am.container.idle.release-timeout-min.millis
|
true |
| Enable Container Reuse | Configuration to specify whether container should be reused. |
tez.am.container.reuse.enabled
|
true |
tez.am.container.reuse.enabled
|
true |
| Timeout Before Container Reuse | The amount of time to wait before assigning a container to the next level of locality. NODE > RACK > NON_LOCAL |
tez.am.container.reuse.locality.delay-allocation-millis
|
250 millisecond(s) |
tez.am.container.reuse.locality.delay-allocation-millis
|
true |
| Enable Container Reuse for Non-Local Tasks | Whether to reuse containers for non-local tasks. Active only if reuse is enabled. |
tez.am.container.reuse.non-local-fallback.enabled
|
false |
tez.am.container.reuse.non-local-fallback.enabled
|
true |
| Enable Container Reuse for Rack Local Tasks | Whether to reuse containers for rack local tasks. Active only if reuse is enabled. |
tez.am.container.reuse.rack-fallback.enabled
|
true |
tez.am.container.reuse.rack-fallback.enabled
|
true |
| Tez Application Master Default Command Line Options | Cluster default Java options for the Tez Application Master process. These will be prepended to the properties specified via tez.am.launch.cmd-opts. |
tez.am.launch.cluster-default.cmd-opts
|
-server -Djava.net.preferIPv4Stack=true |
tez.am.launch.cluster-default.cmd-opts
|
true |
| Tez Application Master Command Line Options | Java options for the Tez Application Master process. The Xmx value is derived based on tez.am.resource.memory.mb and is 80% of the value by default. Used only if the value is not specified explicitly by the DAG definition. |
tez.am.launch.cmd-opts
|
-XX:+PrintGCDetails -verbose:gc -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp |
tez.am.launch.cmd-opts
|
true |
| Tez Application Master Environment Settings | Additional execution environment entries for tez. This is not an additive property. You must preserve the original value if you want to have access to native libraries. Used only if the value is not specified explicitly by the DAG definition. |
tez.am.launch.env
|
LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native |
tez.am.launch.env
|
true |
| Log level for Application Masters | Root Logging level passed to the Tez Application Master. |
tez.am.log.level
|
INFO |
tez.am.log.level
|
true |
| Number of Recovery Runs | Specifies the total number of time the Application Master will run in case recovery is triggered. |
tez.am.max.app.attempts
|
2 |
tez.am.max.app.attempts
|
true |
| Maximum Task Attempts | The maximum number of allowed task attempt failures on a node before it gets marked as blacklisted. |
tez.am.maxtaskfailures.per.node
|
10 |
tez.am.maxtaskfailures.per.node
|
true |
| Tez Application Master Memory | The amount of memory to be used by the Application Master. Used only if the value is not specified explicitly by the DAG definition. |
tez.am.resource.memory.mb
|
2 GiB |
tez.am.resource.memory.mb
|
true |
| History URL Template | Template to generate the History URL for a particular Tez Application. Template replaces __APPLICATION_ID__ with the actual applicationId and __HISTORY_URL_BASE__ with the value from the tez.tez-ui.history-url.base config property |
tez.am.tez-ui.history-url.template
|
__HISTORY_URL_BASE__?viewPath=%2F%23%2Ftez-app%2F__APPLICATION_ID__ |
tez.am.tez-ui.history-url.template
|
true |
| Tez Application Master View ACLs | Application Master view ACLs. This allows the specified users/groups to view the status of the Application Master and all DAGs that run within this Appliation Master. Value format: Comma separated list of users, followed by whitespace, followed by a comma separated list of groups. |
tez.am.view-acls
|
* |
tez.am.view-acls
|
false |
| Tez Additional Classpath | Specify additional classpath information to be used for Tez AM and all containers. |
tez.cluster.additional.classpath.prefix
|
tez.cluster.additional.classpath.prefix
|
false | |
| Maximum Number of Counters | The number of allowed counters for the executing DAG. |
tez.counters.max
|
10000 |
tez.counters.max
|
true |
| Maximum Counter Groups | The number of allowed counter groups for the executing DAG. |
tez.counters.max.groups
|
3000 |
tez.counters.max.groups
|
true |
| Whether to generate debug artifacts | Generate debug artifacts such as a text representation of the submitted DAG plan. |
tez.generate.debug.artifacts
|
false |
tez.generate.debug.artifacts
|
true |
| Grouped Split Maximum Size | Upper bound on the size (in bytes) of a grouped split, to avoid generating excessively large split. |
tez.grouping.max-size
|
1 GiB |
tez.grouping.max-size
|
true |
| Grouped Split Minimum Size | Lower bound on the size (in bytes) of a grouped split, to avoid generating too many splits. |
tez.grouping.min-size
|
16 MiB |
tez.grouping.min-size
|
true |
| Queue Capacity Multiplier | The multiplier for available queue capacity when determining number of tasks for a Vertex. 1.7 with 100% queue available implies generating a number of tasks roughly equal to 170% of the available containers on the queue. |
tez.grouping.split-waves
|
1.7 |
tez.grouping.split-waves
|
true |
| Tez history events directory | Directory where proto logger writes the history events, should generally be sys.db database directory. |
tez.history.logging.proto-base-dir
|
/warehouse/tablespace/managed/hive/sys.db |
tez.history.logging.proto-base-dir
|
true |
| DAGs per Group | DAGs per group. |
tez.history.logging.timeline-cache-plugin.old-num-dags-per-group
|
5 |
tez.history.logging.timeline-cache-plugin.old-num-dags-per-group
|
true |
| Enable Intermediate Data Compression | Whether intermediate data should be compressed or not. |
tez.runtime.compress
|
true |
tez.runtime.compress
|
true |
| Codec for Compressing Intermediate Data | The codec to be used if compressing intermediate data. Only applicable if tez.runtime.compress is enabled. |
tez.runtime.compress.codec
|
org.apache.hadoop.io.compress.SnappyCodec |
tez.runtime.compress.codec
|
true |
| Publish Configuration Information | Whether to publish configuration information to History logger. |
tez.runtime.convert.user-payload.to.history-text
|
false |
tez.runtime.convert.user-payload.to.history-text
|
true |
| Sort Buffer Size | The size of the sort buffer when output needs to be sorted. |
tez.runtime.io.sort.mb
|
272 MiB |
tez.runtime.io.sort.mb
|
true |
| Enable Accessing the Local Files Directly | If the shuffle input is on the local host bypass the http fetch and access the files directly. |
tez.runtime.optimize.local.fetch
|
true |
tez.runtime.optimize.local.fetch
|
true |
| Pipeline Sorter Sort Threads | Tez runtime pipelined sorter sort threads. |
tez.runtime.pipelined.sorter.sort.threads
|
2 |
tez.runtime.pipelined.sorter.sort.threads
|
true |
| Fraction of Memory to Retain Shuffled Data | Fraction (0-1) of the available memory which can be used to retain shuffled data. |
tez.runtime.shuffle.fetch.buffer.percent
|
0.6 |
tez.runtime.shuffle.fetch.buffer.percent
|
true |
| Keep the Shuffle Connection Alive | This property determines if the shuffle connection should be kept alive. If not, then the connection needs to be reestablished. |
tez.runtime.shuffle.keep-alive.enabled
|
true |
tez.runtime.shuffle.keep-alive.enabled
|
true |
| Maximum Percent of Shuffle Segment | This property determines the maximum size of a shuffle segment which can be fetched to memory. Fraction (0-1) of shuffle memory (after applying tez.runtime.shuffle.fetch.buffer.percent). |
tez.runtime.shuffle.memory.limit.percent
|
0.25 |
tez.runtime.shuffle.memory.limit.percent
|
true |
| Buffer Size for Unordered Output | The size of the buffer when output does not require to be sorted. |
tez.runtime.unordered.output.buffer.size-mb
|
100 MiB |
tez.runtime.unordered.output.buffer.size-mb
|
true |
| Timeout for Application Master for a Task | Time (in seconds) for which the Tez Application Master should wait for a DAG to be submitted before shutting down. |
tez.session.am.dag.submit.timeout.secs
|
5 minute(s) |
tez.session.am.dag.submit.timeout.secs
|
true |
| Timeout for Application Master to Come up | Time (in seconds) to wait for Application Master to come up when trying to submit a DAG from the client. |
tez.session.client.timeout.secs
|
-1 second(s) |
tez.session.client.timeout.secs
|
true |
| ScatterGather Connection Maximum Fraction of Tasks | In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled. Number of tasks ready for scheduling on the current vertex scales linearly between min-fraction and max-fraction. |
tez.shuffle-vertex-manager.max-src-fraction
|
0.4 |
tez.shuffle-vertex-manager.max-src-fraction
|
true |
| ScatterGather Connection Minimum Fraction of Tasks | In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are schedule. |
tez.shuffle-vertex-manager.min-src-fraction
|
0.2 |
tez.shuffle-vertex-manager.min-src-fraction
|
true |
| TEZ Staging directory | The staging dir used while submitting DAGs. |
tez.staging-dir
|
/tmp/$user.name/staging |
tez.staging-dir
|
true |
| Heartbeat Interval | Time interval at which task counters are sent to the Application Master. |
tez.task.am.heartbeat.counter.interval-ms.max
|
4 second(s) |
tez.task.am.heartbeat.counter.interval-ms.max
|
true |
| Generate Counters on a Per-Edge Basis | Whether to generate counters on a per-edge basis for a Tez DAG. Helpful for in-depth analysis. |
tez.task.generate.counters.per.io
|
true |
tez.task.generate.counters.per.io
|
true |
| Maximum Time Between Tasks | The maximum amount of time, in seconds, to wait before a task asks an Application Master for another task. |
tez.task.get-task.sleep.interval-ms.max
|
200 millisecond(s) |
tez.task.get-task.sleep.interval-ms.max
|
true |
| Tez Task Default Command Line Options | Cluster default Java options for tasks. These will be prepended to the properties specified via tez.task.launch.cmd-opts. |
tez.task.launch.cluster-default.cmd-opts
|
-server -Djava.net.preferIPv4Stack=true |
tez.task.launch.cluster-default.cmd-opts
|
true |
| Tez Task Command Line Options | Java options for tasks. The Xmx value is derived based on tez.task.resource.memory.mb and is 80% of this value by default. Used only if the value is not specified explicitly by the DAG definition. |
tez.task.launch.cmd-opts
|
-XX:+PrintGCDetails -verbose:gc -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp |
tez.task.launch.cmd-opts
|
true |
| Tez Task Environment Settings | Additional execution environment entries for tez. This is not an additive property. You must preserve the original value if you want to have access to native libraries. Used only if the value is not specified explicitly by the DAG definition. |
tez.task.launch.env
|
LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native |
tez.task.launch.env
|
true |
| Maximum Number of Events in a Heartbeat | Maximum number of events to fetch from the Application Master by the tasks in a single heartbeat. |
tez.task.max-events-per-heartbeat
|
500 |
tez.task.max-events-per-heartbeat
|
true |
| Tez Task Memory | The amount of memory to be used by launched tasks. Used only if the value is not specified explicitly by the DAG definition. |
tez.task.resource.memory.mb
|
1536 MiB |
tez.task.resource.memory.mb
|
true |
| Tez UI URL Base | The base of the Tez UI URL. |
tez.tez-ui.history-url.base
|
tez.tez-ui.history-url.base
|
false | |
| Use Hadoop Libs | This being true implies that the deployment is relying on hadoop jars being available on the cluster on all nodes. |
tez.use.cluster.hadoop-libs
|
false |
tez.use.cluster.hadoop-libs
|
true |
| Enable Yarn Timeline-Service | Timeline service version we're currently using. |
yarn.timeline-service.enabled
|
false |
yarn.timeline-service.enabled
|
true |
| YARN Service | Name of the YARN service that this Tez service instance depends on |
yarn_service
|
true |
Suppressions
| Display Name | Description | Property Name | Default Value | API Name | Required |
|---|---|---|---|---|---|
| Suppress Configuration Validator: CDH Version Validator | Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator. | false |
role_config_suppression_cdh_version_validator
|
true | |
| Suppress Configuration Validator: Deploy Directory | Whether to suppress configuration warnings produced by the Deploy Directory configuration validator. | false |
role_config_suppression_client_config_root_dir
|
true | |
| Suppress Configuration Validator: Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml | Whether to suppress configuration warnings produced by the Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml configuration validator. | false |
role_config_suppression_tez-conf/tez-site.xml_client_config_safety_valve
|
true | |
| Suppress Configuration Validator: Gateway Count Validator | Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator. | false |
service_config_suppression_gateway_count_validator
|
true | |
| Suppress Parameter Validation: System Group | Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter. | false |
service_config_suppression_process_groupname
|
true | |
| Suppress Parameter Validation: System User | Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter. | false |
service_config_suppression_process_username
|
true | |
| Suppress Parameter Validation: Service Triggers | Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter. | false |
service_config_suppression_service_triggers
|
true | |
| Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) | Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter. | false |
service_config_suppression_smon_derived_configs_safety_valve
|
true | |
| Suppress Parameter Validation: Tez Application Master Default Command Line Options | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Application Master Default Command Line Options parameter. | false |
service_config_suppression_tez.am.launch.cluster-default.cmd-opts
|
true | |
| Suppress Parameter Validation: Tez Application Master Command Line Options | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Application Master Command Line Options parameter. | false |
service_config_suppression_tez.am.launch.cmd-opts
|
true | |
| Suppress Parameter Validation: Tez Application Master Environment Settings | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Application Master Environment Settings parameter. | false |
service_config_suppression_tez.am.launch.env
|
true | |
| Suppress Parameter Validation: Number of Recovery Runs | Whether to suppress configuration warnings produced by the built-in parameter validation for the Number of Recovery Runs parameter. | false |
service_config_suppression_tez.am.max.app.attempts
|
true | |
| Suppress Parameter Validation: History URL Template | Whether to suppress configuration warnings produced by the built-in parameter validation for the History URL Template parameter. | false |
service_config_suppression_tez.am.tez-ui.history-url.template
|
true | |
| Suppress Parameter Validation: Tez Application Master View ACLs | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Application Master View ACLs parameter. | false |
service_config_suppression_tez.am.view-acls
|
true | |
| Suppress Parameter Validation: Tez Additional Classpath | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Additional Classpath parameter. | false |
service_config_suppression_tez.cluster.additional.classpath.prefix
|
true | |
| Suppress Parameter Validation: Tez history events directory | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez history events directory parameter. | false |
service_config_suppression_tez.history.logging.proto-base-dir
|
true | |
| Suppress Parameter Validation: DAGs per Group | Whether to suppress configuration warnings produced by the built-in parameter validation for the DAGs per Group parameter. | false |
service_config_suppression_tez.history.logging.timeline-cache-plugin.old-num-dags-per-group
|
true | |
| Suppress Parameter Validation: Codec for Compressing Intermediate Data | Whether to suppress configuration warnings produced by the built-in parameter validation for the Codec for Compressing Intermediate Data parameter. | false |
service_config_suppression_tez.runtime.compress.codec
|
true | |
| Suppress Parameter Validation: TEZ Staging directory | Whether to suppress configuration warnings produced by the built-in parameter validation for the TEZ Staging directory parameter. | false |
service_config_suppression_tez.staging-dir
|
true | |
| Suppress Parameter Validation: Tez Task Default Command Line Options | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Task Default Command Line Options parameter. | false |
service_config_suppression_tez.task.launch.cluster-default.cmd-opts
|
true | |
| Suppress Parameter Validation: Tez Task Command Line Options | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Task Command Line Options parameter. | false |
service_config_suppression_tez.task.launch.cmd-opts
|
true | |
| Suppress Parameter Validation: Tez Task Environment Settings | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Task Environment Settings parameter. | false |
service_config_suppression_tez.task.launch.env
|
true | |
| Suppress Parameter Validation: Tez UI URL Base | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez UI URL Base parameter. | false |
service_config_suppression_tez.tez-ui.history-url.base
|
true | |
| Suppress Parameter Validation: Tez Service Environment Advanced Configuration Snippet (Safety Valve) | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Service Environment Advanced Configuration Snippet (Safety Valve) parameter. | false |
service_config_suppression_tez_service_env_safety_valve
|
true |
