Chapter 24. Installing Apache Slider
Prerequisites
You must have at least core Hadoop on your system. See Configure the Remote Repositories for more information.
Verify the HDP repositories are available:
yum list slider
The output should list at least one Slider package similar to the following:
slider.noarch <version>
If yum responds with "Error: No matching package to list" as shown below, yum cannot locate a matching RPM. This can happen if the repository hosting the HDP RPMs is unavailable, or has been disabled. Follow the instructions at Configure the Remote Repositories to configure either a public or private repository before proceeding.
Error: No matching package to list.
Installation
Run the following command to install Slider.
For RHEL/CentOS/Oracle Linux:
yum install slider_2*
For SLES:
zypper install slider_2*
For Ubuntu:
apt-get install slider_2*
As the root user, edit the following properties in the
/etc/hadoop/conf/yarn-site.xml
file.<property> <name>hadoop.registry.zk.quorum</name> <value>$ZOOKEEPERQUORUM-SERVERS</value> <description>List of hostname:port pairs defining the zookeeper quorum binding for the registry </description> </property> <property> <name>hadoop.registry.rm.enabled</name> <value>true</value> <description> Is the registry enabled: does the RM start it up, create the user and system paths, and purge service records when containers, application attempts and applications complete? </description> </property>
Set hadoop.registry.rm.enabled to true and set hadoop.registry.zk.quorum to the address and port number of your ZooKeeper Quorum server (usually assigned to port 2181). For example:
<property> <name>hadoop.registry.zk.quorum</name> <value>node-1.example.com:2181</value> <description>List of hostname:port pairs defining the zookeeper quorum binding for the registry </description> </property> <property> <name>hadoop.registry.rm.enabled</name> <value>true</value> <description>Is the registry enabled: does the RM start it up, create the user and system paths, and purge service records when containers, application attempts and applications complete? </description> </property>
As the root user, specify the JAVA_HOME and HADOOP_CONF_DIR settings in the
/etc/slider/conf/slider-env.sh
file. For example:# this is the shell script to start Slider deploying an application # Usage: slider <action> <commands> # The env variable SLIDER_JVM_OPTS can be used to override # the default JVM opts export JAVA_HOME=/usr/hadoop-jdk1.6.0_31 export HADOOP_CONF_DIR=/etc/hadoop/conf
Use the following command to switch to the slider bin directory:
cd /usr/hdp/current/slider-client/bin
Use the Slider version command to verify that Slider has installed properly:
./slider version
Ensure that there are no errors, and that your results say “Compiled against Hadoop <current_hadoop_version>”.
[root@node-1 bin]# ./slider version 2014-10-27 14:42:45,340 [main] INFO client.SliderClient - Slider Core-0.51.0.2.4.3.0 Built against commit# d766e78d77 on Java 1.6.0_31 by jenkins 2014-10-27 14:42:45,351 [main] INFO client.SliderClient - Compiled against Hadoop 2.4.3.0-$BUILD 2014-10-27 14:42:45,375 [main] INFO client.SliderClient - Hadoop runtime version (no branch) with source checksum 517963c273a1f4f8f5bfc15d92aa013 and build date 2014-10-27T03:27Z 2014-10-27 14:42:45,383 [main] INFO util.ExitUtil - Exiting with status 0 [root@node-1 bin]#
If HDP is installed in a cluster without using Ambari, invoke the following command (as user hdfs) after the slider client is installed:
slider dependency --upload
The above command uploads the Slider dependency tarball in an HDP specific path:
/hdp/apps/<hdp_version>/slider/slider.tar.gz
All subsequent Slider application creation jobs use this dependency tarball. The benefit is twofold. The application submission time reduces drastically from approximately 10-20 seconds to approximately 1-2 seconds, depending on the hardware. It also eliminates all potential jar conflict issues when 3rd party client applications use the slider-core jar to submit Slider application creation requests.