Non-Ambari Cluster Installation Guide
Also available as:
PDF
loading table of contents...

Chapter 24. Installing Apache Slider

Prerequisites

  1. You must have at least core Hadoop on your system. See Configure the Remote Repositories for more information.

  2. Verify the HDP repositories are available:

    yum list slider

    The output should list at least one Slider package similar to the following:

    slider.noarch <version>

    If yum responds with "Error: No matching package to list" as shown below, yum cannot locate a matching RPM. This can happen if the repository hosting the HDP RPMs is unavailable, or has been disabled. Follow the instructions at Configure the Remote Repositories to configure either a public or private repository before proceeding.

    Error: No matching package to list.

Installation

  1. Run the following command to install Slider.

    • For RHEL/CentOS/Oracle Linux:

      yum install slider_2*

    • For SLES:

      zypper install slider_2*

    • For Ubuntu:

      apt-get install slider_2*

  2. As the root user, edit the following properties in the /etc/hadoop/conf/yarn-site.xml file.

    <property>
         <name>hadoop.registry.zk.quorum</name>
         <value>$ZOOKEEPERQUORUM-SERVERS</value>
         <description>List of hostname:port pairs defining the zookeeper quorum binding for the registry
         </description>
    </property>
     
    <property>
         <name>hadoop.registry.rm.enabled</name>
         <value>true</value>
         <description> Is the registry enabled: does the RM start it up, create the user
            and system paths, and purge service records when containers, application attempts
            and applications complete?
         </description>
    </property>

    Set hadoop.registry.rm.enabled to true and set hadoop.registry.zk.quorum to the address and port number of your ZooKeeper Quorum server (usually assigned to port 2181). For example:

    <property>
         <name>hadoop.registry.zk.quorum</name>
         <value>node-1.example.com:2181</value>
         <description>List of hostname:port pairs defining the zookeeper quorum binding for the registry
         </description>
    </property>
     
    <property>
         <name>hadoop.registry.rm.enabled</name>
         <value>true</value>
         <description>Is the registry enabled: does the RM start it up, create the user
            and system paths, and purge service records when containers, application attempts
            and applications complete?
         </description>
    </property>
  3. As the root user, specify the JAVA_HOME and HADOOP_CONF_DIR settings in the /etc/slider/conf/slider-env.sh file. For example:

    # this is the shell script to start Slider deploying an application
    # Usage: slider <action> <commands>
    
    # The env variable SLIDER_JVM_OPTS can be used to override
    # the default JVM opts
    
    export JAVA_HOME=/usr/hadoop-jdk1.6.0_31
    export HADOOP_CONF_DIR=/etc/hadoop/conf
  4. Use the following command to switch to the slider bin directory:

    cd /usr/hdp/current/slider-client/bin

  5. Use the Slider version command to verify that Slider has installed properly:

    ./slider version

  6. Ensure that there are no errors, and that your results say “Compiled against Hadoop <current_hadoop_version>”.

    [root@node-1 bin]# ./slider version
    2014-10-27 14:42:45,340 [main] INFO client.SliderClient - Slider Core-0.51.0.2.3.4.0 Built against commit# d766e78d77 on Java 1.6.0_31 by jenkins
    2014-10-27 14:42:45,351 [main] INFO client.SliderClient - Compiled against Hadoop 2.3.4.0-$BUILD
    2014-10-27 14:42:45,375 [main] INFO client.SliderClient - Hadoop runtime version (no branch) with source checksum 517963c273a1f4f8f5bfc15d92aa013 and build date 2014-10-27T03:27Z
    2014-10-27 14:42:45,383 [main] INFO util.ExitUtil - Exiting with status 0
    [root@node-1 bin]# 
  7. If HDP is installed in a cluster without using Ambari, invoke the following command (as user hdfs) after the slider client is installed:

    slider dependency --upload

    The above command uploads the Slider dependency tarball in an HDP specific path:

    /hdp/apps/<hdp_version>/slider/slider.tar.gz

    All subsequent Slider application creation jobs use this dependency tarball. The benefit is twofold. The application submission time reduces drastically from approximately 10-20 seconds to approximately 1-2 seconds, depending on the hardware. It also eliminates all potential jar conflict issues when 3rd party client applications use the slider-core jar to submit Slider application creation requests.