Tez Properties in Cloudera Runtime 7.0.x
Role groups:
Gateway
Categories:
Advanced
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Deploy Directory | The directory where the client configs will be deployed | /etc/tez |
client_config_root_dir
|
true | |
Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml | For advanced use only, a string to be inserted into the client configuration for tez-conf/tez-site.xml. |
tez-conf/tez-site.xml_client_config_safety_valve
|
false |
Monitoring
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Enable Configuration Change Alerts | When set, Cloudera Manager will send alerts when this entity's configuration changes. | false |
enable_config_alerts
|
false |
Other
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Alternatives Priority | The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others. | 50 |
client_config_priority
|
true |
Suppressions
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Suppress Configuration Validator: Version Validator | Whether to suppress configuration warnings produced by the Version Validator configuration validator. | false |
role_config_suppression_cdh_version_validator
|
true | |
Suppress Parameter Validation: Deploy Directory | Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter. | false |
role_config_suppression_client_config_root_dir
|
true | |
Suppress Parameter Validation: Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml parameter. | false |
role_config_suppression_tez-conf/tez-site.xml_client_config_safety_valve
|
true |
Service-Wide
Categories:
Advanced
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
System Group | The group that this service's processes should run as. | hadoop |
process_groupname
|
true | |
System User | The user that this service's processes should run as. | tez |
process_username
|
true | |
Tez Service Environment Advanced Configuration Snippet (Safety Valve) | For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration. |
TEZ_service_env_safety_valve
|
false |
Monitoring
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Enable Service Level Health Alerts | When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold | true |
enable_alerts
|
false | |
Enable Configuration Change Alerts | When set, Cloudera Manager will send alerts when this entity's configuration changes. | false |
enable_config_alerts
|
false | |
Service Triggers | The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
[{"triggerName": "sample-trigger",
"triggerExpression": "IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad",
"streamThreshold": 10, "enabled": "true"}] See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases. |
[] |
service_triggers
|
true | |
Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) | For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones. |
smon_derived_configs_safety_valve
|
false |
Other
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
tez.am.am-rm.heartbeat.interval-ms.max | The maximum heartbeat interval between the Application Master and RM in milliseconds. |
tez.am.am-rm.heartbeat.interval-ms.max
|
250 millisecond(s) |
tez.am.am-rm.heartbeat.interval-ms.max
|
true |
tez.am.container.idle.release-timeout-max.millis | The maximum amount of time to hold on to a container if no task can be assigned to it immediately. Only active when reuse is enabled. |
tez.am.container.idle.release-timeout-max.millis
|
20 second(s) |
tez.am.container.idle.release-timeout-max.millis
|
true |
tez.am.container.idle.release-timeout-min.millis | The minimum amount of time to hold on to a container that is idle. Only active when reuse is enabled. |
tez.am.container.idle.release-timeout-min.millis
|
10 second(s) |
tez.am.container.idle.release-timeout-min.millis
|
true |
tez.am.container.reuse.enabled | Configuration to specify whether container should be reused. |
tez.am.container.reuse.enabled
|
true |
tez.am.container.reuse.enabled
|
true |
tez.am.container.reuse.locality.delay-allocation-millis | The amount of time to wait before assigning a container to the next level of locality. NODE > RACK > NON_LOCAL |
tez.am.container.reuse.locality.delay-allocation-millis
|
250 millisecond(s) |
tez.am.container.reuse.locality.delay-allocation-millis
|
true |
tez.am.container.reuse.non-local-fallback.enabled | Whether to reuse containers for non-local tasks. Active only if reuse is enabled. |
tez.am.container.reuse.non-local-fallback.enabled
|
false |
tez.am.container.reuse.non-local-fallback.enabled
|
true |
tez.am.container.reuse.rack-fallback.enabled | Whether to reuse containers for rack local tasks. Active only if reuse is enabled. |
tez.am.container.reuse.rack-fallback.enabled
|
true |
tez.am.container.reuse.rack-fallback.enabled
|
true |
tez.am.launch.cluster-default.cmd-opts | Cluster default Java options for the Tez Application Master process. These will be prepended to the properties specified via tez.am.launch.cmd-opts. |
tez.am.launch.cluster-default.cmd-opts
|
-server -Djava.net.preferIPv4Stack=true |
tez.am.launch.cluster-default.cmd-opts
|
true |
tez.am.launch.cmd-opts | Java options for the Tez Application Master process. The Xmx value is derived based on tez.am.resource.memory.mb and is 80% of the value by default. Used only if the value is not specified explicitly by the DAG definition. |
tez.am.launch.cmd-opts
|
-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp |
tez.am.launch.cmd-opts
|
true |
tez.am.launch.env | Additional execution environment entries for tez. This is not an additive property. You must preserve the original value if you want to have access to native libraries. Used only if the value is not specified explicitly by the DAG definition. |
tez.am.launch.env
|
LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native |
tez.am.launch.env
|
true |
Log level for Application Masters | Root Logging level passed to the Tez Application Master. |
tez.am.log.level
|
INFO |
tez.am.log.level
|
true |
tez.am.max.app.attempts | Specifies the total number of time the Application Master will run in case recovery is triggered. |
tez.am.max.app.attempts
|
2 |
tez.am.max.app.attempts
|
true |
tez.am.maxtaskfailures.per.node | The maximum number of allowed task attempt failures on a node before it gets marked as blacklisted. |
tez.am.maxtaskfailures.per.node
|
10 |
tez.am.maxtaskfailures.per.node
|
true |
tez.am.resource.memory.mb | The amount of memory to be used by the Application Master. Used only if the value is not specified explicitly by the DAG definition. |
tez.am.resource.memory.mb
|
2 GiB |
tez.am.resource.memory.mb
|
true |
tez.am.tez-ui.history-url.template | Template to generate the History URL for a particular Tez Application. Template replaces __APPLICATION_ID__ with the actual applicationId and __HISTORY_URL_BASE__ with the value from the tez.tez-ui.history-url.base config property |
tez.am.tez-ui.history-url.template
|
__HISTORY_URL_BASE__?viewPath=%2F%23%2Ftez-app%2F__APPLICATION_ID__ |
tez.am.tez-ui.history-url.template
|
true |
tez.am.view-acls | Application Master view ACLs. This allows the specified users/groups to view the status of the Application Master and all DAGs that run within this Appliation Master. Value format: Comma separated list of users, followed by whitespace, followed by a comma separated list of groups. |
tez.am.view-acls
|
* |
tez.am.view-acls
|
false |
tez.cluster.additional.classpath.prefix | tez.cluster.additional.classpath.prefix |
tez.cluster.additional.classpath.prefix
|
tez.cluster.additional.classpath.prefix
|
false | |
tez.counters.max | The number of allowed counters for the executing DAG. |
tez.counters.max
|
10000 |
tez.counters.max
|
true |
tez.counters.max.groups | The number of allowed counter groups for the executing DAG. |
tez.counters.max.groups
|
3000 |
tez.counters.max.groups
|
true |
Whether to generate debug artifacts | Generate debug artifacts such as a text representation of the submitted DAG plan. |
tez.generate.debug.artifacts
|
false |
tez.generate.debug.artifacts
|
true |
tez.grouping.max-size | Upper bound on the size (in bytes) of a grouped split, to avoid generating excessively large split. |
tez.grouping.max-size
|
1 GiB |
tez.grouping.max-size
|
true |
tez.grouping.min-size | Lower bound on the size (in bytes) of a grouped split, to avoid generating too many splits. |
tez.grouping.min-size
|
16 MiB |
tez.grouping.min-size
|
true |
tez.grouping.split-waves | The multiplier for available queue capacity when determining number of tasks for a Vertex. 1.7 with 100% queue available implies generating a number of tasks roughly equal to 170% of the available containers on the queue. |
tez.grouping.split-waves
|
1.7 |
tez.grouping.split-waves
|
true |
Tez history events directory | Directory where proto logger writes the history events, should generally be sys.db database directory. |
tez.history.logging.proto-base-dir
|
/warehouse/tablespace/managed/hive/sys.db |
tez.history.logging.proto-base-dir
|
true |
tez.history.logging.timeline-cache-plugin.old-num-dags-per-group | DAGs per group. |
tez.history.logging.timeline-cache-plugin.old-num-dags-per-group
|
5 |
tez.history.logging.timeline-cache-plugin.old-num-dags-per-group
|
true |
tez.runtime.compress | Whether intermediate data should be compressed or not. |
tez.runtime.compress
|
true |
tez.runtime.compress
|
true |
tez.runtime.compress.codec | The coded to be used if compressing intermediate data. Only applicable if tez.runtime.compress is enabled. |
tez.runtime.compress.codec
|
org.apache.hadoop.io.compress.SnappyCodec |
tez.runtime.compress.codec
|
true |
tez.runtime.convert.user-payload.to.history-text | Whether to publish configuration information to History logger. |
tez.runtime.convert.user-payload.to.history-text
|
false |
tez.runtime.convert.user-payload.to.history-text
|
true |
tez.runtime.io.sort.mb | The size of the sort buffer when output needs to be sorted. |
tez.runtime.io.sort.mb
|
272 MiB |
tez.runtime.io.sort.mb
|
true |
tez.runtime.optimize.local.fetch | If the shuffle input is on the local host bypass the http fetch and access the files directly. |
tez.runtime.optimize.local.fetch
|
true |
tez.runtime.optimize.local.fetch
|
true |
tez.runtime.pipelined.sorter.sort.threads | Tez runtime pipelined sorter sort threads. |
tez.runtime.pipelined.sorter.sort.threads
|
2 |
tez.runtime.pipelined.sorter.sort.threads
|
true |
tez.runtime.shuffle.fetch.buffer.percent | Fraction (0-1) of the available memory which can be used to retain shuffled data. |
tez.runtime.shuffle.fetch.buffer.percent
|
0.6 |
tez.runtime.shuffle.fetch.buffer.percent
|
true |
tez.runtime.shuffle.memory.limit.percent | This property determines the maximum size of a shuffle segment which can be fetched to memory. Fraction (0-1) of shuffle memory (after applying tez.runtime.shuffle.fetch.buffer.percent). |
tez.runtime.shuffle.memory.limit.percent
|
0.25 |
tez.runtime.shuffle.memory.limit.percent
|
true |
tez.runtime.unordered.output.buffer.size-mb | The size of the buffer when output does not require to be sorted. |
tez.runtime.unordered.output.buffer.size-mb
|
100 MiB |
tez.runtime.unordered.output.buffer.size-mb
|
true |
tez.session.am.dag.submit.timeout.secs | Time (in seconds) for which the Tez Application Master should wait for a DAG to be submitted before shutting down. |
tez.session.am.dag.submit.timeout.secs
|
5 minute(s) |
tez.session.am.dag.submit.timeout.secs
|
true |
tez.session.client.timeout.secs | Time (in seconds) to wait for Application Master to come up when trying to submit a DAG from the client. |
tez.session.client.timeout.secs
|
-1 second(s) |
tez.session.client.timeout.secs
|
true |
tez.shuffle-vertex-manager.max-src-fraction | In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled. Number of tasks ready for scheduling on the current vertex scales linearly between min-fraction and max-fraction. |
tez.shuffle-vertex-manager.max-src-fraction
|
0.4 |
tez.shuffle-vertex-manager.max-src-fraction
|
true |
tez.shuffle-vertex-manager.min-src-fraction | In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are schedule. |
tez.shuffle-vertex-manager.min-src-fraction
|
0.2 |
tez.shuffle-vertex-manager.min-src-fraction
|
true |
TEZ Staging directory | The staging dir used while submitting DAGs. |
tez.staging-dir
|
/tmp/$user.name/staging |
tez.staging-dir
|
true |
tez.task.am.heartbeat.counter.interval-ms.max | Time interval at which task counters are sent to the Application Master. |
tez.task.am.heartbeat.counter.interval-ms.max
|
4 second(s) |
tez.task.am.heartbeat.counter.interval-ms.max
|
true |
tez.task.generate.counters.per.io | Whether to generate counters on a per-edge basis for a Tez DAG. Helpful for in-depth analysis. |
tez.task.generate.counters.per.io
|
true |
tez.task.generate.counters.per.io
|
true |
tez.task.get-task.sleep.interval-ms.max | The maximum amount of time, in seconds, to wait before a task asks an Application Master for another task. |
tez.task.get-task.sleep.interval-ms.max
|
200 millisecond(s) |
tez.task.get-task.sleep.interval-ms.max
|
true |
tez.task.launch.cluster-default.cmd-opts | Cluster default Java options for tasks. These will be prepended to the properties specified via tez.task.launch.cmd-opts. |
tez.task.launch.cluster-default.cmd-opts
|
-server -Djava.net.preferIPv4Stack=true |
tez.task.launch.cluster-default.cmd-opts
|
true |
tez.task.launch.cmd-opts | Java options for tasks. The Xmx value is derived based on tez.task.resource.memory.mb and is 80% of this value by default. Used only if the value is not specified explicitly by the DAG definition. |
tez.task.launch.cmd-opts
|
-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp |
tez.task.launch.cmd-opts
|
true |
tez.task.launch.env | Additional execution environment entries for tez. This is not an additive property. You must preserve the original value if you want to have access to native libraries. Used only if the value is not specified explicitly by the DAG definition. |
tez.task.launch.env
|
LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native |
tez.task.launch.env
|
true |
tez.task.max-events-per-heartbeat | Maximum number of of events to fetch from the Application Master by the tasks in a single heartbeat. |
tez.task.max-events-per-heartbeat
|
500 |
tez.task.max-events-per-heartbeat
|
true |
tez.task.resource.memory.mb | The amount of memory to be used by launched tasks. Used only if the value is not specified explicitly by the DAG definition. |
tez.task.resource.memory.mb
|
1536 MiB |
tez.task.resource.memory.mb
|
true |
tez.tez-ui.history-url.base | tez.tez-ui.history-url.base |
tez.tez-ui.history-url.base
|
tez.tez-ui.history-url.base
|
false | |
tez.use.cluster.hadoop-libs | This being true implies that the deployment is relying on hadoop jars being available on the cluster on all nodes. |
tez.use.cluster.hadoop-libs
|
false |
tez.use.cluster.hadoop-libs
|
true |
yarn.timeline-service.enabled | Timeline service version we're currently using. |
yarn.timeline-service.enabled
|
false |
yarn.timeline-service.enabled
|
true |
YARN Service | Name of the YARN service that this Tez service instance depends on |
yarn_service
|
true |
Suppressions
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Suppress Configuration Validator: Gateway Count Validator | Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator. | false |
service_config_suppression_gateway_count_validator
|
true | |
Suppress Parameter Validation: System Group | Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter. | false |
service_config_suppression_process_groupname
|
true | |
Suppress Parameter Validation: System User | Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter. | false |
service_config_suppression_process_username
|
true | |
Suppress Parameter Validation: Service Triggers | Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter. | false |
service_config_suppression_service_triggers
|
true | |
Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) | Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter. | false |
service_config_suppression_smon_derived_configs_safety_valve
|
true | |
Suppress Parameter Validation: tez.am.launch.cluster-default.cmd-opts | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.launch.cluster-default.cmd-opts parameter. | false |
service_config_suppression_tez.am.launch.cluster-default.cmd-opts
|
true | |
Suppress Parameter Validation: tez.am.launch.cmd-opts | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.launch.cmd-opts parameter. | false |
service_config_suppression_tez.am.launch.cmd-opts
|
true | |
Suppress Parameter Validation: tez.am.launch.env | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.launch.env parameter. | false |
service_config_suppression_tez.am.launch.env
|
true | |
Suppress Parameter Validation: tez.am.max.app.attempts | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.max.app.attempts parameter. | false |
service_config_suppression_tez.am.max.app.attempts
|
true | |
Suppress Parameter Validation: tez.am.tez-ui.history-url.template | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.tez-ui.history-url.template parameter. | false |
service_config_suppression_tez.am.tez-ui.history-url.template
|
true | |
Suppress Parameter Validation: tez.am.view-acls | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.view-acls parameter. | false |
service_config_suppression_tez.am.view-acls
|
true | |
Suppress Parameter Validation: tez.cluster.additional.classpath.prefix | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.cluster.additional.classpath.prefix parameter. | false |
service_config_suppression_tez.cluster.additional.classpath.prefix
|
true | |
Suppress Parameter Validation: Tez history events directory | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez history events directory parameter. | false |
service_config_suppression_tez.history.logging.proto-base-dir
|
true | |
Suppress Parameter Validation: tez.history.logging.timeline-cache-plugin.old-num-dags-per-group | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.history.logging.timeline-cache-plugin.old-num-dags-per-group parameter. | false |
service_config_suppression_tez.history.logging.timeline-cache-plugin.old-num-dags-per-group
|
true | |
Suppress Parameter Validation: tez.runtime.compress.codec | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.runtime.compress.codec parameter. | false |
service_config_suppression_tez.runtime.compress.codec
|
true | |
Suppress Parameter Validation: TEZ Staging directory | Whether to suppress configuration warnings produced by the built-in parameter validation for the TEZ Staging directory parameter. | false |
service_config_suppression_tez.staging-dir
|
true | |
Suppress Parameter Validation: tez.task.launch.cluster-default.cmd-opts | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.task.launch.cluster-default.cmd-opts parameter. | false |
service_config_suppression_tez.task.launch.cluster-default.cmd-opts
|
true | |
Suppress Parameter Validation: tez.task.launch.cmd-opts | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.task.launch.cmd-opts parameter. | false |
service_config_suppression_tez.task.launch.cmd-opts
|
true | |
Suppress Parameter Validation: tez.task.launch.env | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.task.launch.env parameter. | false |
service_config_suppression_tez.task.launch.env
|
true | |
Suppress Parameter Validation: tez.tez-ui.history-url.base | Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.tez-ui.history-url.base parameter. | false |
service_config_suppression_tez.tez-ui.history-url.base
|
true | |
Suppress Parameter Validation: Tez Service Environment Advanced Configuration Snippet (Safety Valve) | Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Service Environment Advanced Configuration Snippet (Safety Valve) parameter. | false |
service_config_suppression_tez_service_env_safety_valve
|
true |