Command Line Upgrade
Also available as:
PDF
loading table of contents...

Configure Hadoop

RHEL/CentOS/Oracle Linux

  1. Use the HDP Utility script to calculate memory configuration settings. You must update the memory/CPU settings in yarn-site.xml and mapred-site.xml

  2. Paths have changed in HDP 2.5. make sure you remove old path specifications from hadoop-env.sh, such as:

    export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-amd64-64

    If you leave these paths in your hadoop-env.sh file, the lzo compression code does not load, as this is not where lzo is installed.

SLES

  1. Use the HDP Utility script to calculate memory configuration settings. You must update the memory/CPU settings in yarn-site.xml and mapred-site.xml

  2. Paths have changed in HDP 2.5. make sure you remove old path specifications from hadoop-env.sh, such as:

    export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-amd64-64

    If you leave these paths in your hadoop-env.sh file, the lzo compression code does not load, as this is not where lzo is installed.

Ubuntu/Debian

  1. Use the HDP Utility script to calculate memory configuration settings. You must update the memory/CPU settings in yarn-site.xml and mapred-site.xml

  2. Paths have changed in HDP 2.5. make sure you remove lines from hadoop-env.sh such as:

    export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-amd64-64

    If you leave these paths in your hadoop-env.sh file, the lzo compression code does not load as this is not where lzo is installed.