Chapter 26. Installing Apache Slider
Prerequisites
You must have at least core Hadoop on your system. See Configure the Remote Repositories for more information.
Verify the HDP repositories are available:
yum list slider
The output should list at least one Slider package similar to the following:
slider.noarch <version>
If yum responds with "Error: No matching package to list" as shown below, yum cannot locate a matching RPM. This can happen if the repository hosting the HDP RPMs is unavailable, or has been disabled. Follow the instructions at Configure the Remote Repositories to configure private repository before proceeding.
Error: No matching package to list.
Installation
Run the following command to install Slider.
For RHEL/CentOS/Oracle Linux:
yum install slider_2*
For SLES:
zypper install slider_2*
For Ubuntu:
apt-get install slider_2*
As the root user, edit the following properties in the
/etc/hadoop/conf/yarn-site.xml
file.<property> <name>hadoop.registry.zk.quorum</name> <value>$ZOOKEEPERQUORUM-SERVERS</value> <description>List of hostname:port pairs defining the zookeeper quorum binding for the registry </description> </property> <property> <name>hadoop.registry.rm.enabled</name> <value>true</value> <description> Is the registry enabled: does the RM start it up, create the user and system paths, and purge service records when containers, application attempts and applications complete? </description> </property>
Set hadoop.registry.rm.enabled to true and set hadoop.registry.zk.quorum to the address and port number of your ZooKeeper Quorum server (usually assigned to port 2181). For example:
<property> <name>hadoop.registry.zk.quorum</name> <value>node-1.example.com:2181</value> <description>List of hostname:port pairs defining the zookeeper quorum binding for the registry </description> </property> <property> <name>hadoop.registry.rm.enabled</name> <value>true</value> <description>Is the registry enabled: does the RM start it up, create the user and system paths, and purge service records when containers, application attempts and applications complete? </description> </property>
As the root user, specify the JAVA_HOME and HADOOP_CONF_DIR settings in the
/etc/slider/conf/slider-env.sh
file. For example:# this is the shell script to start Slider deploying an application # Usage: slider <action> <commands> # The env variable SLIDER_JVM_OPTS can be used to override # the default JVM opts export JAVA_HOME=/usr/hadoop-jdk1.6.0_31 export HADOOP_CONF_DIR=/etc/hadoop/conf
Use the following command to switch to the slider bin directory:
cd /usr/hdp/current/slider-client/bin
Use the Slider version command to verify that Slider has installed properly:
./slider version
Ensure that there are no errors, and that your results say “Compiled against Hadoop <current_hadoop_version>”.
[root@node-1 bin]# ./slider version 2014-10-27 14:42:45,340 [main] INFO client.SliderClient - Slider Core-0.51.0.2.6.1.0 Built against commit# d766e78d77 on Java 1.6.0_31 by jenkins 2014-10-27 14:42:45,351 [main] INFO client.SliderClient - Compiled against Hadoop 2.6.1.0-2800 2014-10-27 14:42:45,375 [main] INFO client.SliderClient - Hadoop runtime version (no branch) with source checksum 517963c273a1f4f8f5bfc15d92aa013 and build date 2014-10-27T03:27Z 2014-10-27 14:42:45,383 [main] INFO util.ExitUtil - Exiting with status 0 [root@node-1 bin]#
If HDP is installed in a cluster without using Ambari, invoke the following command (as user hdfs) after the slider client is installed:
slider dependency --upload
The above command uploads the Slider dependency tarball in an HDP specific path:
/hdp/apps/<hdp_version>/slider/slider.tar.gz
All subsequent Slider application creation jobs use this dependency tarball. The benefit is twofold. The application submission time reduces drastically from approximately 10-20 seconds to approximately 1-2 seconds, depending on the hardware. It also eliminates all potential jar conflict issues when 3rd party client applications use the slider-core jar to submit Slider application creation requests.