Several client libraries--such as MapReduce, Tez, Pig, Hive, and Sqoop libraries--are used by multiple components, shared via HDFS. Upload the new client libraries into HDFS. These libraries are versioned; they will not be used until they’re enabled via the new clients for their respective components.
Note: you do not need to move the Oozie sharelib tarball into HDFS.
To move client tarballs into HDFS, run the following commands from any one of the
cluster client nodes. Note that $HDFS_USER is defined as the HDFS superuser, such as
hdfs.
su $HDFS_USER # Load Tez client libraries into HDFS hdfs dfs -mkdir -p /hdp/apps/2.2.4.2-2/tez/ hdfs dfs -put /usr/hdp/2.2.4.2-2/tez/lib/tez.tar.gz /hdp/apps/2.2.4.2-2/tez/ hdfs dfs -chown -R hdfs:hadoop /hdp hdfs dfs -chmod -R 555 /hdp/apps/2.2.4.2-2/tez hdfs dfs -chmod -R 444 /hdp/apps/2.2.4.2-2/tez/tez.tar.gz # Load MapReduce client libraries into HDFS hdfs dfs -mkdir -p /hdp/apps/2.2.4.2-2/mapreduce/ hdfs dfs -put /usr/hdp/2.2.4.2-2/hadoop/mapreduce.tar.gz /hdp/apps/2.2.4.2-2/mapreduce/ hdfs dfs -put /usr/hdp/2.2.4.2-2/hadoop-mapreduce/hadoop-streaming.jar /hdp/apps/2.2.4.2-2/mapreduce/ hdfs dfs -chown -R hdfs:hadoop /hdp hdfs dfs -chmod -R 555 /hdp/apps/2.2.4.2-2/mapreduce hdfs dfs -chmod -R 444 /hdp/apps/2.2.4.2-2/mapreduce/mapreduce.tar.gz # Load Hive client libraries into HDFS hdfs dfs -mkdir -p /hdp/apps/2.2.4.2-2/hive/ hdfs dfs -put /usr/hdp/2.2.4.2-2/hive/hive.tar.gz /hdp/apps/2.2.4.2-2/hive/hive.tar.gz hdfs dfs -chown -R hdfs:hadoop /hdp hdfs dfs -chmod -R 555 /hdp/apps/2.2.4.2-2/hive hdfs dfs -chmod -R 444 /hdp/apps/2.2.4.2-2/hive/hive.tar.gz # Load Pig client libraries into HDFS hdfs dfs -mkdir -p /hdp/apps/2.2.4.2-2/pig/ hdfs dfs -put /usr/hdp/2.2.4.2-2/pig/pig.tar.gz /hdp/apps/2.2.4.2-2/pig/ hdfs dfs -chown -R hdfs:hadoop /hdp hdfs dfs -chmod -R 555 /hdp/apps/2.2.4.2-2/pig hdfs dfs -chmod -R 444 /hdp/apps/2.2.4.2-2/pig/pig.tar.gz # Load Sqoop client libraries into HDFS hdfs dfs -mkdir -p /hdp/apps/2.2.4.2-2/sqoop/ hdfs dfs -put /usr/hdp/2.2.4.2-2/sqoop/sqoop.tar.gz /hdp/apps/2.2.4.2-2/sqoop/ hdfs dfs -chown -R hdfs:hadoop /hdp hdfs dfs -chmod -R 555 /hdp/apps/2.2.4.2-2/sqoop hdfs dfs -chmod -R 444 /hdp/apps/2.2.4.2-2/sqoop/sqoop.tar.gz # Log out exit
Check that the files were copied:
su $HDFS_USER hdfs dfs -ls /hdp/apps/2.2.4.2-2 exit
This should return results similar to the following:
Found 5 items
dr-xr-xr-x - hdfs hadoop 0 2015-03-03 17:05 /hdp/apps/2.2.4.2-2/hive dr-xr-xr-x - hdfs hadoop 0 2015-03-03 17:04 /hdp/apps/2.2.4.2-2/mapreduce dr-xr-xr-x - hdfs hadoop 0 2015-03-03 17:05 /hdp/apps/2.2.4.2-2/pig dr-xr-xr-x - hdfs hadoop 0 2015-03-03 17:13 /hdp/apps/2.2.4.2-2/sqoop dr-xr-xr-x - hdfs hadoop 0 2015-03-03 17:02 /hdp/apps/2.2.4.2-2/tez
