ReferencePDF version

Tez Properties in Cloudera Runtime 7.0.3

Role groups:

Categories:

Display Name Description Related Name Default Value API Name Required
Deploy Directory The directory where the client configs will be deployed /etc/tez client_config_root_dir true
Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml For advanced use only, a string to be inserted into the client configuration for tez-conf/tez-site.xml. tez-conf/tez-site.xml_client_config_safety_valve false
Display Name Description Related Name Default Value API Name Required
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Display Name Description Related Name Default Value API Name Required
Alternatives Priority The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others. 50 client_config_priority true
Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: CDH Version Validator Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Parameter Validation: Deploy Directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter. false role_config_suppression_client_config_root_dir true
Suppress Parameter Validation: Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml parameter. false role_config_suppression_tez-conf/tez-site.xml_client_config_safety_valve true

Categories:

Display Name Description Related Name Default Value API Name Required
System Group The group that this service's processes should run as. hadoop process_groupname true
System User The user that this service's processes should run as. tez process_username true
Tez Service Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration. TEZ_service_env_safety_valve false
Display Name Description Related Name Default Value API Name Required
Enable Service Level Health Alerts When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Service Triggers The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.
For example, the following JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad", "streamThreshold": 10, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.		 [] service_triggers true
Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones. smon_derived_configs_safety_valve false
Display Name Description Related Name Default Value API Name Required
Maximum Heartbeat Interval The maximum heartbeat interval between the Application Master and RM in milliseconds. tez.am.am-rm.heartbeat.interval-ms.max 250 millisecond(s) tez.am.am-rm.heartbeat.interval-ms.max true
Maximum Timout to Hold Idle Containers The maximum amount of time to hold on to a container if no task can be assigned to it immediately. Only active when reuse is enabled. tez.am.container.idle.release-timeout-max.millis 20 second(s) tez.am.container.idle.release-timeout-max.millis true
Minimum Timout to Hold Idle Containers The minimum amount of time to hold on to a container that is idle. Only active when reuse is enabled. tez.am.container.idle.release-timeout-min.millis 10 second(s) tez.am.container.idle.release-timeout-min.millis true
Enable Container Reuse Configuration to specify whether container should be reused. tez.am.container.reuse.enabled true tez.am.container.reuse.enabled true
Timeout Before Container Reuse The amount of time to wait before assigning a container to the next level of locality. NODE > RACK > NON_LOCAL tez.am.container.reuse.locality.delay-allocation-millis 250 millisecond(s) tez.am.container.reuse.locality.delay-allocation-millis true
Enable Container Reuse for Non-Local Tasks Whether to reuse containers for non-local tasks. Active only if reuse is enabled. tez.am.container.reuse.non-local-fallback.enabled false tez.am.container.reuse.non-local-fallback.enabled true
Enable Container Reuse for Rack Local Tasks Whether to reuse containers for rack local tasks. Active only if reuse is enabled. tez.am.container.reuse.rack-fallback.enabled true tez.am.container.reuse.rack-fallback.enabled true
Tez Application Master Default Command Line Options Cluster default Java options for the Tez Application Master process. These will be prepended to the properties specified via tez.am.launch.cmd-opts. tez.am.launch.cluster-default.cmd-opts -server -Djava.net.preferIPv4Stack=true tez.am.launch.cluster-default.cmd-opts true
Tez Application Master Command Line Options Java options for the Tez Application Master process. The Xmx value is derived based on tez.am.resource.memory.mb and is 80% of the value by default. Used only if the value is not specified explicitly by the DAG definition. tez.am.launch.cmd-opts -XX:+PrintGCDetails -verbose:gc -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp tez.am.launch.cmd-opts true
Tez Application Master Environment Settings Additional execution environment entries for tez. This is not an additive property. You must preserve the original value if you want to have access to native libraries. Used only if the value is not specified explicitly by the DAG definition. tez.am.launch.env LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native tez.am.launch.env true
Log level for Application Masters Root Logging level passed to the Tez Application Master. tez.am.log.level INFO tez.am.log.level true
Number of Recovery Runs Specifies the total number of time the Application Master will run in case recovery is triggered. tez.am.max.app.attempts 2 tez.am.max.app.attempts true
Maximum Task Attempts The maximum number of allowed task attempt failures on a node before it gets marked as blacklisted. tez.am.maxtaskfailures.per.node 10 tez.am.maxtaskfailures.per.node true
Tez Application Master Memory The amount of memory to be used by the Application Master. Used only if the value is not specified explicitly by the DAG definition. tez.am.resource.memory.mb 2 GiB tez.am.resource.memory.mb true
History URL Template Template to generate the History URL for a particular Tez Application. Template replaces __APPLICATION_ID__ with the actual applicationId and __HISTORY_URL_BASE__ with the value from the tez.tez-ui.history-url.base config property tez.am.tez-ui.history-url.template __HISTORY_URL_BASE__?viewPath=%2F%23%2Ftez-app%2F__APPLICATION_ID__ tez.am.tez-ui.history-url.template true
Tez Application Master View ACLs Application Master view ACLs. This allows the specified users/groups to view the status of the Application Master and all DAGs that run within this Appliation Master. Value format: Comma separated list of users, followed by whitespace, followed by a comma separated list of groups. tez.am.view-acls * tez.am.view-acls false
Tez Additional Classpath Specify additional classpath information to be used for Tez AM and all containers. tez.cluster.additional.classpath.prefix tez.cluster.additional.classpath.prefix false
Maximum Number of Counters The number of allowed counters for the executing DAG. tez.counters.max 10000 tez.counters.max true
Maximum Counter Groups The number of allowed counter groups for the executing DAG. tez.counters.max.groups 3000 tez.counters.max.groups true
Whether to generate debug artifacts Generate debug artifacts such as a text representation of the submitted DAG plan. tez.generate.debug.artifacts false tez.generate.debug.artifacts true
Grouped Split Maximum Size Upper bound on the size (in bytes) of a grouped split, to avoid generating excessively large split. tez.grouping.max-size 1 GiB tez.grouping.max-size true
Grouped Split Minimum Size Lower bound on the size (in bytes) of a grouped split, to avoid generating too many splits. tez.grouping.min-size 16 MiB tez.grouping.min-size true
Queue Capacity Multiplier The multiplier for available queue capacity when determining number of tasks for a Vertex. 1.7 with 100% queue available implies generating a number of tasks roughly equal to 170% of the available containers on the queue. tez.grouping.split-waves 1.7 tez.grouping.split-waves true
Tez history events directory Directory where proto logger writes the history events, should generally be sys.db database directory. tez.history.logging.proto-base-dir /warehouse/tablespace/managed/hive/sys.db tez.history.logging.proto-base-dir true
DAGs per Group DAGs per group. tez.history.logging.timeline-cache-plugin.old-num-dags-per-group 5 tez.history.logging.timeline-cache-plugin.old-num-dags-per-group true
Enable Intermediate Data Compression Whether intermediate data should be compressed or not. tez.runtime.compress true tez.runtime.compress true
Codec for Compressing Intermediate Data The codec to be used if compressing intermediate data. Only applicable if tez.runtime.compress is enabled. tez.runtime.compress.codec org.apache.hadoop.io.compress.SnappyCodec tez.runtime.compress.codec true
Publish Configuration Information Whether to publish configuration information to History logger. tez.runtime.convert.user-payload.to.history-text false tez.runtime.convert.user-payload.to.history-text true
Sort Buffer Size The size of the sort buffer when output needs to be sorted. tez.runtime.io.sort.mb 272 MiB tez.runtime.io.sort.mb true
Enable Accessing the Local Files Directly If the shuffle input is on the local host bypass the http fetch and access the files directly. tez.runtime.optimize.local.fetch true tez.runtime.optimize.local.fetch true
Pipeline Sorter Sort Threads Tez runtime pipelined sorter sort threads. tez.runtime.pipelined.sorter.sort.threads 2 tez.runtime.pipelined.sorter.sort.threads true
Fraction of Memory to Retain Shuffled Data Fraction (0-1) of the available memory which can be used to retain shuffled data. tez.runtime.shuffle.fetch.buffer.percent 0.6 tez.runtime.shuffle.fetch.buffer.percent true
Maximum Percent of Shuffle Segment This property determines the maximum size of a shuffle segment which can be fetched to memory. Fraction (0-1) of shuffle memory (after applying tez.runtime.shuffle.fetch.buffer.percent). tez.runtime.shuffle.memory.limit.percent 0.25 tez.runtime.shuffle.memory.limit.percent true
Buffer Size for Unordered Output The size of the buffer when output does not require to be sorted. tez.runtime.unordered.output.buffer.size-mb 100 MiB tez.runtime.unordered.output.buffer.size-mb true
Timeout for Application Master for a Task Time (in seconds) for which the Tez Application Master should wait for a DAG to be submitted before shutting down. tez.session.am.dag.submit.timeout.secs 5 minute(s) tez.session.am.dag.submit.timeout.secs true
Timeout for Application Master to Come up Time (in seconds) to wait for Application Master to come up when trying to submit a DAG from the client. tez.session.client.timeout.secs -1 second(s) tez.session.client.timeout.secs true
ScatterGather Connection Maximum Fraction of Tasks In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled. Number of tasks ready for scheduling on the current vertex scales linearly between min-fraction and max-fraction. tez.shuffle-vertex-manager.max-src-fraction 0.4 tez.shuffle-vertex-manager.max-src-fraction true
ScatterGather Connection Minimum Fraction of Tasks In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are schedule. tez.shuffle-vertex-manager.min-src-fraction 0.2 tez.shuffle-vertex-manager.min-src-fraction true
TEZ Staging directory The staging dir used while submitting DAGs. tez.staging-dir /tmp/$user.name/staging tez.staging-dir true
Heartbeat Interval Time interval at which task counters are sent to the Application Master. tez.task.am.heartbeat.counter.interval-ms.max 4 second(s) tez.task.am.heartbeat.counter.interval-ms.max true
Generate Counters on a Per-Edge Basis Whether to generate counters on a per-edge basis for a Tez DAG. Helpful for in-depth analysis. tez.task.generate.counters.per.io true tez.task.generate.counters.per.io true
Maximum Time Between Tasks The maximum amount of time, in seconds, to wait before a task asks an Application Master for another task. tez.task.get-task.sleep.interval-ms.max 200 millisecond(s) tez.task.get-task.sleep.interval-ms.max true
Tez Task Default Command Line Options Cluster default Java options for tasks. These will be prepended to the properties specified via tez.task.launch.cmd-opts. tez.task.launch.cluster-default.cmd-opts -server -Djava.net.preferIPv4Stack=true tez.task.launch.cluster-default.cmd-opts true
Tez Task Command Line Options Java options for tasks. The Xmx value is derived based on tez.task.resource.memory.mb and is 80% of this value by default. Used only if the value is not specified explicitly by the DAG definition. tez.task.launch.cmd-opts -XX:+PrintGCDetails -verbose:gc -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp tez.task.launch.cmd-opts true
Tez Task Environment Settings Additional execution environment entries for tez. This is not an additive property. You must preserve the original value if you want to have access to native libraries. Used only if the value is not specified explicitly by the DAG definition. tez.task.launch.env LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native tez.task.launch.env true
Maximum Number of Events in a Heartbeat Maximum number of events to fetch from the Application Master by the tasks in a single heartbeat. tez.task.max-events-per-heartbeat 500 tez.task.max-events-per-heartbeat true
Tez Task Memory The amount of memory to be used by launched tasks. Used only if the value is not specified explicitly by the DAG definition. tez.task.resource.memory.mb 1536 MiB tez.task.resource.memory.mb true
Tez UI URL Base The base of the Tez UI URL. tez.tez-ui.history-url.base tez.tez-ui.history-url.base false
Use Hadoop Libs This being true implies that the deployment is relying on hadoop jars being available on the cluster on all nodes. tez.use.cluster.hadoop-libs false tez.use.cluster.hadoop-libs true
Enable Yarn Timeline-Service Timeline service version we're currently using. yarn.timeline-service.enabled false yarn.timeline-service.enabled true
YARN Service Name of the YARN service that this Tez service instance depends on yarn_service true
Display Name Description Related Name Default Value API Name Required
Suppress Configuration Validator: CDH Version Validator Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator. false role_config_suppression_cdh_version_validator true
Suppress Configuration Validator: Deploy Directory Whether to suppress configuration warnings produced by the Deploy Directory configuration validator. false role_config_suppression_client_config_root_dir true
Suppress Configuration Validator: Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml Whether to suppress configuration warnings produced by the Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml configuration validator. false role_config_suppression_tez-conf/tez-site.xml_client_config_safety_valve true
Suppress Configuration Validator: Gateway Count Validator Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator. false service_config_suppression_gateway_count_validator true
Suppress Parameter Validation: System Group Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter. false service_config_suppression_process_groupname true
Suppress Parameter Validation: System User Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter. false service_config_suppression_process_username true
Suppress Parameter Validation: Service Triggers Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter. false service_config_suppression_service_triggers true
Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter. false service_config_suppression_smon_derived_configs_safety_valve true
Suppress Parameter Validation: Tez Application Master Default Command Line Options Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Application Master Default Command Line Options parameter. false service_config_suppression_tez.am.launch.cluster-default.cmd-opts true
Suppress Parameter Validation: Tez Application Master Command Line Options Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Application Master Command Line Options parameter. false service_config_suppression_tez.am.launch.cmd-opts true
Suppress Parameter Validation: Tez Application Master Environment Settings Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Application Master Environment Settings parameter. false service_config_suppression_tez.am.launch.env true
Suppress Parameter Validation: Number of Recovery Runs Whether to suppress configuration warnings produced by the built-in parameter validation for the Number of Recovery Runs parameter. false service_config_suppression_tez.am.max.app.attempts true
Suppress Parameter Validation: History URL Template Whether to suppress configuration warnings produced by the built-in parameter validation for the History URL Template parameter. false service_config_suppression_tez.am.tez-ui.history-url.template true
Suppress Parameter Validation: Tez Application Master View ACLs Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Application Master View ACLs parameter. false service_config_suppression_tez.am.view-acls true
Suppress Parameter Validation: Tez Additional Classpath Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Additional Classpath parameter. false service_config_suppression_tez.cluster.additional.classpath.prefix true
Suppress Parameter Validation: Tez history events directory Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez history events directory parameter. false service_config_suppression_tez.history.logging.proto-base-dir true
Suppress Parameter Validation: DAGs per Group Whether to suppress configuration warnings produced by the built-in parameter validation for the DAGs per Group parameter. false service_config_suppression_tez.history.logging.timeline-cache-plugin.old-num-dags-per-group true
Suppress Parameter Validation: Codec for Compressing Intermediate Data Whether to suppress configuration warnings produced by the built-in parameter validation for the Codec for Compressing Intermediate Data parameter. false service_config_suppression_tez.runtime.compress.codec true
Suppress Parameter Validation: TEZ Staging directory Whether to suppress configuration warnings produced by the built-in parameter validation for the TEZ Staging directory parameter. false service_config_suppression_tez.staging-dir true
Suppress Parameter Validation: Tez Task Default Command Line Options Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Task Default Command Line Options parameter. false service_config_suppression_tez.task.launch.cluster-default.cmd-opts true
Suppress Parameter Validation: Tez Task Command Line Options Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Task Command Line Options parameter. false service_config_suppression_tez.task.launch.cmd-opts true
Suppress Parameter Validation: Tez Task Environment Settings Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Task Environment Settings parameter. false service_config_suppression_tez.task.launch.env true
Suppress Parameter Validation: Tez UI URL Base Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez UI URL Base parameter. false service_config_suppression_tez.tez-ui.history-url.base true
Suppress Parameter Validation: Tez Service Environment Advanced Configuration Snippet (Safety Valve) Whether to suppress configuration warnings produced by the built-in parameter validation for the Tez Service Environment Advanced Configuration Snippet (Safety Valve) parameter. false service_config_suppression_tez_service_env_safety_valve true

We want your opinion

How can we improve this page?

What kind of feedback do you have?