ReferencePDF version

Tez Properties in Cloudera Runtime 7.0.x

Role groups:

Categories:

Display Name Description Related Name Default Value API Name Required
Deploy Directory The directory where the client configs will be deployed /etc/tez client_config_root_dir true
Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml For advanced use only, a string to be inserted into the client configuration for tez-conf/tez-site.xml. tez-conf/tez-site.xml_client_config_safety_valve false
Display Name Description Related Name Default Value API Name Required
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Display Name Description Related Name Default Value API Name Required
Alternatives Priority The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others. 50 client_config_priority true
Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: Version Validator Whether to suppress configuration warnings produced by the Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: Deploy Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter. false role_config_suppression_client_config_root_dir true
Suppress Parameter Validation: Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml parameter. false role_config_suppression_tez-conf/tez-site.xml_client_config_safety_valve true

Categories:

Display Name Description Related Name Default Value API Name Required
System Group The group that this service's processes should run as. hadoop process_groupname true
System User The user that this service's processes should run as. tez process_username true
Tez Service Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration. TEZ_service_env_safety_valve false
Display Name Description Related Name Default Value API Name Required
Enable Service Level Health Alerts When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Service Triggers The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.
For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad", "streamThreshold": 10, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.		 [] service_triggers true
Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones. smon_derived_configs_safety_valve false
Display Name Description Related Name Default Value API Name Required
tez.am.am-rm.heartbeat.interval-ms.max The maximum heartbeat interval between the Application Master and RM in milliseconds. tez.am.am-rm.heartbeat.interval-ms.max 250 millisecond(s) tez.am.am-rm.heartbeat.interval-ms.max true
tez.am.container.idle.release-timeout-max.millis The maximum amount of time to hold on to a container if no task can be assigned to it immediately. Only active when reuse is enabled. tez.am.container.idle.release-timeout-max.millis 20 second(s) tez.am.container.idle.release-timeout-max.millis true
tez.am.container.idle.release-timeout-min.millis The minimum amount of time to hold on to a container that is idle. Only active when reuse is enabled. tez.am.container.idle.release-timeout-min.millis 10 second(s) tez.am.container.idle.release-timeout-min.millis true
tez.am.container.reuse.enabled Configuration to specify whether container should be reused. tez.am.container.reuse.enabled true tez.am.container.reuse.enabled true
tez.am.container.reuse.locality.delay-allocation-millis The amount of time to wait before assigning a container to the next level of locality. NODE > RACK > NON_LOCAL tez.am.container.reuse.locality.delay-allocation-millis 250 millisecond(s) tez.am.container.reuse.locality.delay-allocation-millis true
tez.am.container.reuse.non-local-fallback.enabled Whether to reuse containers for non-local tasks. Active only if reuse is enabled. tez.am.container.reuse.non-local-fallback.enabled false tez.am.container.reuse.non-local-fallback.enabled true
tez.am.container.reuse.rack-fallback.enabled Whether to reuse containers for rack local tasks. Active only if reuse is enabled. tez.am.container.reuse.rack-fallback.enabled true tez.am.container.reuse.rack-fallback.enabled true
tez.am.launch.cluster-default.cmd-opts Cluster default Java options for the Tez Application Master process. These will be prepended to the properties specified via tez.am.launch.cmd-opts. tez.am.launch.cluster-default.cmd-opts -server -Djava.net.preferIPv4Stack=true tez.am.launch.cluster-default.cmd-opts true
tez.am.launch.cmd-opts Java options for the Tez Application Master process. The Xmx value is derived based on tez.am.resource.memory.mb and is 80% of the value by default. Used only if the value is not specified explicitly by the DAG definition. tez.am.launch.cmd-opts -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp tez.am.launch.cmd-opts true
tez.am.launch.env Additional execution environment entries for tez. This is not an additive property. You must preserve the original value if you want to have access to native libraries. Used only if the value is not specified explicitly by the DAG definition. tez.am.launch.env LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native tez.am.launch.env true
Log level for Application Masters Root Logging level passed to the Tez Application Master. tez.am.log.level INFO tez.am.log.level true
tez.am.max.app.attempts Specifies the total number of time the Application Master will run in case recovery is triggered. tez.am.max.app.attempts 2 tez.am.max.app.attempts true
tez.am.maxtaskfailures.per.node The maximum number of allowed task attempt failures on a node before it gets marked as blacklisted. tez.am.maxtaskfailures.per.node 10 tez.am.maxtaskfailures.per.node true
tez.am.resource.memory.mb The amount of memory to be used by the Application Master. Used only if the value is not specified explicitly by the DAG definition. tez.am.resource.memory.mb 2 GiB tez.am.resource.memory.mb true
tez.am.tez-ui.history-url.template Template to generate the History URL for a particular Tez Application. Template replaces __APPLICATION_ID__ with the actual applicationId and __HISTORY_URL_BASE__ with the value from the tez.tez-ui.history-url.base config property tez.am.tez-ui.history-url.template __HISTORY_URL_BASE__?viewPath=%2F%23%2Ftez-app%2F__APPLICATION_ID__ tez.am.tez-ui.history-url.template true
tez.am.view-acls Application Master view ACLs. This allows the specified users/groups to view the status of the Application Master and all DAGs that run within this Appliation Master. Value format: Comma separated list of users, followed by whitespace, followed by a comma separated list of groups. tez.am.view-acls * tez.am.view-acls false
tez.cluster.additional.classpath.prefix tez.cluster.additional.classpath.prefix tez.cluster.additional.classpath.prefix tez.cluster.additional.classpath.prefix false
tez.counters.max The number of allowed counters for the executing DAG. tez.counters.max 10000 tez.counters.max true
tez.counters.max.groups The number of allowed counter groups for the executing DAG. tez.counters.max.groups 3000 tez.counters.max.groups true
Whether to generate debug artifacts Generate debug artifacts such as a text representation of the submitted DAG plan. tez.generate.debug.artifacts false tez.generate.debug.artifacts true
tez.grouping.max-size Upper bound on the size (in bytes) of a grouped split, to avoid generating excessively large split. tez.grouping.max-size 1 GiB tez.grouping.max-size true
tez.grouping.min-size Lower bound on the size (in bytes) of a grouped split, to avoid generating too many splits. tez.grouping.min-size 16 MiB tez.grouping.min-size true
tez.grouping.split-waves The multiplier for available queue capacity when determining number of tasks for a Vertex. 1.7 with 100% queue available implies generating a number of tasks roughly equal to 170% of the available containers on the queue. tez.grouping.split-waves 1.7 tez.grouping.split-waves true
Tez history events directory Directory where proto logger writes the history events, should generally be sys.db database directory. tez.history.logging.proto-base-dir /warehouse/tablespace/managed/hive/sys.db tez.history.logging.proto-base-dir true
tez.history.logging.timeline-cache-plugin.old-num-dags-per-group DAGs per group. tez.history.logging.timeline-cache-plugin.old-num-dags-per-group 5 tez.history.logging.timeline-cache-plugin.old-num-dags-per-group true
tez.runtime.compress Whether intermediate data should be compressed or not. tez.runtime.compress true tez.runtime.compress true
tez.runtime.compress.codec The coded to be used if compressing intermediate data. Only applicable if tez.runtime.compress is enabled. tez.runtime.compress.codec org.apache.hadoop.io.compress.SnappyCodec tez.runtime.compress.codec true
tez.runtime.convert.user-payload.to.history-text Whether to publish configuration information to History logger. tez.runtime.convert.user-payload.to.history-text false tez.runtime.convert.user-payload.to.history-text true
tez.runtime.io.sort.mb The size of the sort buffer when output needs to be sorted. tez.runtime.io.sort.mb 272 MiB tez.runtime.io.sort.mb true
tez.runtime.optimize.local.fetch If the shuffle input is on the local host bypass the http fetch and access the files directly. tez.runtime.optimize.local.fetch true tez.runtime.optimize.local.fetch true
tez.runtime.pipelined.sorter.sort.threads Tez runtime pipelined sorter sort threads. tez.runtime.pipelined.sorter.sort.threads 2 tez.runtime.pipelined.sorter.sort.threads true
tez.runtime.shuffle.fetch.buffer.percent Fraction (0-1) of the available memory which can be used to retain shuffled data. tez.runtime.shuffle.fetch.buffer.percent 0.6 tez.runtime.shuffle.fetch.buffer.percent true
tez.runtime.shuffle.memory.limit.percent This property determines the maximum size of a shuffle segment which can be fetched to memory. Fraction (0-1) of shuffle memory (after applying tez.runtime.shuffle.fetch.buffer.percent). tez.runtime.shuffle.memory.limit.percent 0.25 tez.runtime.shuffle.memory.limit.percent true
tez.runtime.unordered.output.buffer.size-mb The size of the buffer when output does not require to be sorted. tez.runtime.unordered.output.buffer.size-mb 100 MiB tez.runtime.unordered.output.buffer.size-mb true
tez.session.am.dag.submit.timeout.secs Time (in seconds) for which the Tez Application Master should wait for a DAG to be submitted before shutting down. tez.session.am.dag.submit.timeout.secs 5 minute(s) tez.session.am.dag.submit.timeout.secs true
tez.session.client.timeout.secs Time (in seconds) to wait for Application Master to come up when trying to submit a DAG from the client. tez.session.client.timeout.secs -1 second(s) tez.session.client.timeout.secs true
tez.shuffle-vertex-manager.max-src-fraction In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled. Number of tasks ready for scheduling on the current vertex scales linearly between min-fraction and max-fraction. tez.shuffle-vertex-manager.max-src-fraction 0.4 tez.shuffle-vertex-manager.max-src-fraction true
tez.shuffle-vertex-manager.min-src-fraction In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are schedule. tez.shuffle-vertex-manager.min-src-fraction 0.2 tez.shuffle-vertex-manager.min-src-fraction true
TEZ Staging directory The staging dir used while submitting DAGs. tez.staging-dir /tmp/$user.name/staging tez.staging-dir true
tez.task.am.heartbeat.counter.interval-ms.max Time interval at which task counters are sent to the Application Master. tez.task.am.heartbeat.counter.interval-ms.max 4 second(s) tez.task.am.heartbeat.counter.interval-ms.max true
tez.task.generate.counters.per.io Whether to generate counters on a per-edge basis for a Tez DAG. Helpful for in-depth analysis. tez.task.generate.counters.per.io true tez.task.generate.counters.per.io true
tez.task.get-task.sleep.interval-ms.max The maximum amount of time, in seconds, to wait before a task asks an Application Master for another task. tez.task.get-task.sleep.interval-ms.max 200 millisecond(s) tez.task.get-task.sleep.interval-ms.max true
tez.task.launch.cluster-default.cmd-opts Cluster default Java options for tasks. These will be prepended to the properties specified via tez.task.launch.cmd-opts. tez.task.launch.cluster-default.cmd-opts -server -Djava.net.preferIPv4Stack=true tez.task.launch.cluster-default.cmd-opts true
tez.task.launch.cmd-opts Java options for tasks. The Xmx value is derived based on tez.task.resource.memory.mb and is 80% of this value by default. Used only if the value is not specified explicitly by the DAG definition. tez.task.launch.cmd-opts -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp tez.task.launch.cmd-opts true
tez.task.launch.env Additional execution environment entries for tez. This is not an additive property. You must preserve the original value if you want to have access to native libraries. Used only if the value is not specified explicitly by the DAG definition. tez.task.launch.env LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native tez.task.launch.env true
tez.task.max-events-per-heartbeat Maximum number of of events to fetch from the Application Master by the tasks in a single heartbeat. tez.task.max-events-per-heartbeat 500 tez.task.max-events-per-heartbeat true
tez.task.resource.memory.mb The amount of memory to be used by launched tasks. Used only if the value is not specified explicitly by the DAG definition. tez.task.resource.memory.mb 1536 MiB tez.task.resource.memory.mb true
tez.tez-ui.history-url.base tez.tez-ui.history-url.base tez.tez-ui.history-url.base tez.tez-ui.history-url.base false
tez.use.cluster.hadoop-libs This being true implies that the deployment is relying on hadoop jars being available on the cluster on all nodes. tez.use.cluster.hadoop-libs false tez.use.cluster.hadoop-libs true
yarn.timeline-service.enabled Timeline service version we're currently using. yarn.timeline-service.enabled false yarn.timeline-service.enabled true
YARN Service Name of the YARN service that this Tez service instance depends on yarn_service true
Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: Gateway Count Validator Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator. false service_config_suppression_gateway_count_validator true
Suppress Parameter Validation: System Group Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter. false service_config_suppression_process_groupname true
Suppress Parameter Validation: System User Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter. false service_config_suppression_process_username true
Suppress Parameter Validation: Service Triggers Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter. false service_config_suppression_service_triggers true
Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter. false service_config_suppression_smon_derived_configs_safety_valve true
Suppress Parameter Validation: tez.am.launch.cluster-default.cmd-opts Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.launch.cluster-default.cmd-opts parameter. false service_config_suppression_tez.am.launch.cluster-default.cmd-opts true
Suppress Parameter Validation: tez.am.launch.cmd-opts Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.launch.cmd-opts parameter. false service_config_suppression_tez.am.launch.cmd-opts true
Suppress Parameter Validation: tez.am.launch.env Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.launch.env parameter. false service_config_suppression_tez.am.launch.env true
Suppress Parameter Validation: tez.am.max.app.attempts Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.max.app.attempts parameter. false service_config_suppression_tez.am.max.app.attempts true
Suppress Parameter Validation: tez.am.tez-ui.history-url.template Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.tez-ui.history-url.template parameter. false service_config_suppression_tez.am.tez-ui.history-url.template true
Suppress Parameter Validation: tez.am.view-acls Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.am.view-acls parameter. false service_config_suppression_tez.am.view-acls true
Suppress Parameter Validation: tez.cluster.additional.classpath.prefix Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.cluster.additional.classpath.prefix parameter. false service_config_suppression_tez.cluster.additional.classpath.prefix true
Suppress Parameter Validation: Tez history events directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez history events directory parameter. false service_config_suppression_tez.history.logging.proto-base-dir true
Suppress Parameter Validation: tez.history.logging.timeline-cache-plugin.old-num-dags-per-group Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.history.logging.timeline-cache-plugin.old-num-dags-per-group parameter. false service_config_suppression_tez.history.logging.timeline-cache-plugin.old-num-dags-per-group true
Suppress Parameter Validation: tez.runtime.compress.codec Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.runtime.compress.codec parameter. false service_config_suppression_tez.runtime.compress.codec true
Suppress Parameter Validation: TEZ Staging directory Whether to suppress configuration warnings produced by the built-in parameter validation for the TEZ Staging directory parameter. false service_config_suppression_tez.staging-dir true
Suppress Parameter Validation: tez.task.launch.cluster-default.cmd-opts Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.task.launch.cluster-default.cmd-opts parameter. false service_config_suppression_tez.task.launch.cluster-default.cmd-opts true
Suppress Parameter Validation: tez.task.launch.cmd-opts Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.task.launch.cmd-opts parameter. false service_config_suppression_tez.task.launch.cmd-opts true
Suppress Parameter Validation: tez.task.launch.env Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.task.launch.env parameter. false service_config_suppression_tez.task.launch.env true
Suppress Parameter Validation: tez.tez-ui.history-url.base Whether to suppress configuration warnings produced by the built-in parameter validation for the tez.tez-ui.history-url.base parameter. false service_config_suppression_tez.tez-ui.history-url.base true
Suppress Parameter Validation: Tez Service Environment Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Service Environment Advanced Configuration Snippet (Safety Valve) parameter. false service_config_suppression_tez_service_env_safety_valve true

We want your opinion

How can we improve this page?

What kind of feedback do you have?