Configure and Start Apache Tez
Note | |
---|---|
The |
Important | |
---|---|
If you have a secure server, you will need Kerberos credentials for hdfs user access. |
To upgrade Apache Tez:
Copy your previously backed-up copy of tez-site.xml into the
/etc/tez/conf
directory.Check to see if the HDP-2.3.4 Tez tarball libraries are already in the
/hdp/apps/<hdp 2.3>/tez/
version> directory - if so, then skip this step. If not, put the HDP-2.3.4 Tez tarball libraries in the/hdp/apps
directory in HDFS, so that submitted Tez applications to this cluster will have access to these shared Tez tarball libraries. Execute this step on any host that has the Tez client installed.su - hdfs hdfs dfs -mkdir -p /hdp/apps/<hdp_version>/tez/ hdfs dfs -put /usr/hdp/<hdp_version>/tez/lib/tez.tar.gz /hdp/apps/<hdp_version>/tez/ hdfs dfs -chown -R hdfs:hadoop /hdp hdfs dfs -chmod -R 555 /hdp/apps/<hdp_version>/tez hdfs dfs -chmod -R 444 /hdp/apps/<hdp_version>/tez/tez.tar.gz
Where <hdp_version> is the current HDP version, for example 2.3.4-$BUILD.
Edit the tez.lib.uris property in the tez-site.xml file to point to /hdp/apps/<hdp_version>/tez/tez.tar.gz
... <property> <name>tez.lib.uris</name> <value>/hdp/apps/<hdp_version>/tez/tez.tar.gz</value> </property> ...
Where
<hdp_version>
is the current HDP version, for example2.3.4-$BUILD
.Optional Earlier releases of Tez did not have access control. In the current version of Tez, the default behavior restricts the ability to view the Tez history to only the owner of the job. To retain unrestricted access for non-secure clusters, set
tez.am.view-acls
set to "*".Update the following tez-site.xml properties to their new names.
Old Property Name New Property Name tez.am.java.opts
tez.am.launch.cmd-opts
tez.am.env
tez.am.launch.env
tez.am.shuffle-vertex-manager.min-src-fraction
tez.shuffle-vertex-manager.min-src-fraction
tez.am.shuffle-vertex-manager.max-src-fraction
tez.shuffle-vertex-manager.max-src-fraction
tez.am.shuffle-vertex-manager.enable.auto-parallel
tez.shuffle-vertex-manager.enable.auto-parallel
tez.am.shuffle-vertex-manager.desired-task-input-size
tez.shuffle-vertex-manager.desired-task-input-size
tez.am.shuffle-vertex-manager.min-task-parallelism
tez.shuffle-vertex-manager.min-task-parallelism
tez.am.grouping.split-count
tez.grouping.split-count
tez.am.grouping.by-length
tez.grouping.by-length
tez.am.grouping.by-count
tez.grouping.by-count
tez.am.grouping.max-size
tez.grouping.max-size
tez.am.grouping.min-size
tez.grouping.min-size
tez.am.grouping.rack-split-reduction
tez.grouping.rack-split-reduction
tez.am.am.complete.cancel.delegation.tokens
tez.cancel.delegation.tokens.on.completion
tez.am.max.task.attempts
tez.am.task.max.failed.attempts
tez.generate.dag.viz
tez.generate.debug.artifacts
tez.runtime.intermediate-output.key.comparator.class
tez.runtime.key.comparator.class
tez.runtime.intermediate-output.key.class
tez.runtime.key.class
tez.runtime.intermediate-output.value.class
tez.runtime.value.class
tez.runtime.intermediate-output.should-compress
tez.runtime.compress
tez.runtime.intermediate-output.compress.codec
tez.runtime.compress.codec
tez.runtime.intermediate-input.key.secondary.comparator.class
tez.runtime.key.secondary.comparator.class
tez.runtime.broadcast.data-via-events.enabled
tez.runtime.transfer.data-via-events.enabled
tez.runtime.broadcast.data-via-events.max-size
tez.runtime.transfer.data-via-events.max-size
tez.runtime.shuffle.input.buffer.percent
tez.runtime.shuffle.fetch.buffer.percent
tez.runtime.task.input.buffer.percent
tez.runtime.task.input.post-merge.buffer.percent
tez.runtime.job.counters.max
tez.am.counters.max.keys
tez.runtime.job.counters.group.name.max
tez.am.counters.group-name.max.keys
tez.runtime.job.counters.counter.name.max
tez.am.counters.name.max.keys
tez.runtime.job.counters.groups.max
tez.am.counters.groups.max.keys
tez.task.merge.progress.records
tez.runtime.merge.progress.records
tez.runtime.metrics.session.id
tez.runtime.framework.metrics.session.id
tez.task.scale.memory.additional.reservation. fraction.per-io
tez.task.scale.memory.additional-reservation.fraction.per-io
tez.task.scale.memory.additional.reservation. fraction.max
tez.task.scale.memory.additional-reservation.fraction.max
tez.task.initial.memory.scale.ratios
tez.task.scale.memory.ratios
tez.resource.calculator.process-tree.class
tez.task.resource.calculator.process-tree.class
For more information on setting Tez configuration parameters in HDP-2.3.4, see Installing and Configuring Apache Tez in the Non-Ambari Cluster Installation Guide.
Change the value of the tez.tez-ui.history-url.base property to the url for the upgraded Tez View. For more details, see Using the Tez View in the HDP Ambari Views Guide.