Upgrading HDP Manually
Also available as:
PDF
loading table of contents...

Configure and Start Apache Tez

To upgrade Tez:

  1. Copy your previously backed-up copy of tez-site.xml into the /etc/tez/conf directory.

  2. Upload the Apache Tez tarball to HDFS.

    su $HDFS_USER
    hdfs dfs -mkdir -p /hdp/apps/<hdp_version>/tez/
    hdfs dfs -put /usr/hdp/<hdp_version>/tez/lib/tez.tar.gz /hdp/apps/<hdp_version>/tez/
    hdfs dfs -chown -R hdfs:hadoop /hdp
    hdfs dfs -chmod -R 555 /hdp/apps/<hdp_version>/tez
    hdfs dfs -chmod -R 444 /hdp/apps/<hdp_version>/tez/tez.tar.gz

    Where <hdp_version> is the current HDP version, for example 2.2.9.0-$BUILD.

  3. Set the $TEZ_CONF_DIR environment variable to the location of the tez-site.xml file.

    export TEZ_CONF_DIR=/etc/tez/conf
  4. Set the $TEZ_JARS environment variable to the location of the Tez .jar files and their dependencies.

    export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
  5. Edit the tez.lib.uris property in the tez-site.xml file to point to /hdp/apps/<hdp_version>/tez/tez.tar.gz

    ...
    <property>
     <name>tez.lib.uris</name>
     <value>/hdp/apps/<hdp_version>/tez/tez.tar.gz</value>
    </property>
    ...

    Where <hdp_version> is the current HDP version, for example 2.2.9.0-$BUILD.

  6. Remove the following properties from the tez-site.xml configuration file:

    tez.runtime.intermediate-input.key.comparator.class
    tez.runtime.intermediate-input.key.class
    tez.runtime.intermediate-input.value.class
    tez.runtime.intermediate-input.is-compressed
    tez.runtime.intermediate-input.compress.codec
    tez.profile.container.list
    tez.profile.jvm.opts
    tez.am.slowstart-dag-scheduler.min-resource-fraction
  7. Update the following tez-site.xml properties to their new names.

    Old Property NameNew Property Name

    tez.am.java.opts

    tez.am.launch.cmd-opts

    tez.am.env

    tez.am.launch.env

    tez.am.shuffle-vertex-manager.min-src-fraction

    tez.shuffle-vertex-manager.min-src-fraction

    tez.am.shuffle-vertex-manager.max-src-fraction

    tez.shuffle-vertex-manager.max-src-fraction

    tez.am.shuffle-vertex-manager.enable.auto-parallel

    tez.shuffle-vertex-manager.enable.auto-parallel

    tez.am.shuffle-vertex-manager.desired-task-input-size

    tez.shuffle-vertex-manager.desired-task-input-size

    tez.am.shuffle-vertex-manager.min-task-parallelism

    tez.shuffle-vertex-manager.min-task-parallelism

    tez.am.grouping.split-count

    tez.grouping.split-count

    tez.am.grouping.by-length

    tez.grouping.by-length

    tez.am.grouping.by-count

    tez.grouping.by-count

    tez.am.grouping.max-size

    tez.grouping.max-size

    tez.am.grouping.min-size

    tez.grouping.min-size

    tez.am.grouping.rack-split-reduction

    tez.grouping.rack-split-reduction

    tez.am.am.complete.cancel.delegation.tokens

    tez.cancel.delegation.tokens.on.completion

    tez.am.max.task.attempts

    tez.am.task.max.failed.attempts

    tez.generate.dag.viz

    tez.generate.debug.artifacts

    tez.runtime.intermediate-output.key.comparator.class

    tez.runtime.key.comparator.class

    tez.runtime.intermediate-output.key.class

    tez.runtime.key.class

    tez.runtime.intermediate-output.value.class

    tez.runtime.value.class

    tez.runtime.intermediate-output.should-compress

    tez.runtime.compress

    tez.runtime.intermediate-output.compress.codec

    tez.runtime.compress.codec

    tez.runtime.intermediate-input.key.secondary.comparator.class

    tez.runtime.key.secondary.comparator.class

    tez.runtime.broadcast.data-via-events.enabled

    tez.runtime.transfer.data-via-events.enabled

    tez.runtime.broadcast.data-via-events.max-size

    tez.runtime.transfer.data-via-events.max-size

    tez.runtime.shuffle.input.buffer.percent

    tez.runtime.shuffle.fetch.buffer.percent

    tez.runtime.task.input.buffer.percent

    tez.runtime.task.input.post-merge.buffer.percent

    tez.runtime.job.counters.max

    tez.am.counters.max.keys

    tez.runtime.job.counters.group.name.max

    tez.am.counters.group-name.max.keys

    tez.runtime.job.counters.counter.name.max

    tez.am.counters.name.max.keys

    tez.runtime.job.counters.groups.max

    tez.am.counters.groups.max.keys

    tez.task.merge.progress.records

    tez.runtime.merge.progress.records

    tez.runtime.metrics.session.id

    tez.runtime.framework.metrics.session.id

    tez.task.scale.memory.additional.reservation. fraction.per-io

    tez.task.scale.memory.additional-reservation.fraction.per-io

    tez.task.scale.memory.additional.reservation. fraction.max

    tez.task.scale.memory.additional-reservation.fraction.max

    tez.task.initial.memory.scale.ratios

    tez.task.scale.memory.ratios

    tez.resource.calculator.process-tree.class

    tez.task.resource.calculator.process-tree.class

    For more information on setting Tez configuration parameters in HDP-2.2, see "Installing and Configuring Tez" in the HDP Manual Installation Guide.