Hortonworks ManualsHortonworks Data Platform

 

Hortonworks Data Platform

Installing HDP Manually

Copyright © 2012, 2013 Hortonworks, Inc.

Except where otherwise noted, this document is licensed under Creative Commons Attribution ShareAlike 3.0 License

http://creativecommons.org/licenses/by-sa/3.0/legalcode

2013-03-20

Abstract

The Hortonworks Data Platform, powered by Apache Hadoop, is a massively scalable and 100% open source platform for storing, processing and analyzing large volumes of data. It is designed to deal with data from many sources and formats in a very quick, easy and cost-effective manner. The Hortonworks Data Platform consists of the essential set of Apache Hadoop projects including MapReduce, Hadoop Distributed File System (HDFS), HCatalog, Pig, Hive, HBase, Zookeeper and Ambari. Hortonworks is the major contributor of code and patches to many of these projects. These projects have been integrated and tested as part of the Hortonworks Data Platform release process and installation and configuration tools have also been included.

Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of our code back to the Apache Software Foundation. The Hortonworks Data Platform is Apache-licensed and completely open source. We sell only expert technical support, training and partner-enablement services. All of our technology is, and will remain free and open source.

Please visit the Hortonworks Data Platform page for more information on Hortonworks technology. For more information on Hortonworks services, please visit either the Support or Training page. Feel free to Contact Us directly to discuss your specific needs.

Contents

1. Getting Ready to Install
1. Understand the Basics
2. Meet Minimum System Requirements
2.1. Hardware Recommendations
2.2. Operating Systems Requirements
2.3. Software Requirements
2.4. Database Requirements
2.5. JDK Requirements
3. Configure the Remote Repository
4. Decide on Deployment Type
5. Collect Information
6. Prepare the Environment
6.1. Enable NTP on the Cluster
6.2. Check DNS
6.3. Disable SELinux
7. Install the Java Development Kit
8. Create Service Users and Groups
9. Download Companion Files
10. Define Environment Parameters
10.1. Define Users and Groups
10.2. Define Directories
2. Installing HDFS and MapReduce
1. Set Default File and Directory Permissions
2. Install the Hadoop RPMs
3. Install Compression Libraries
3.1. Install Snappy
3.2. Install LZO
4. Create Directories
4.1. Create the NameNode Directories
4.2. Create the SecondaryNameNode Directories
4.3. Create the DataNode and MapReduce Local Directories
4.4. Create the Log and PID Directories
3. Setting Up the Hadoop Configuration
1. Extract the Core Hadoop Configuration Files
2. Modify the Configuration Files
3. Copy the Configuration Files
4. Validating the Core Hadoop Installation
1. Format and Start HDFS
2. Smoke Test HDFS
3. Start MapReduce
4. Smoke Test MapReduce
5. Installing Apache Pig
1. Install the Pig RPMs
2. Set Up Configuration Files
2.1. Extract the Pig Configuration Files
2.2. Copy the Configuration Files
3. Validate the Installation
3.1. Smoke Test Pig
6. Installing Apache Hive and Apache HCatalog
1. Install the Hive and HCatalog RPMs
2. Set Directories and Permissions
2.1. Create Log Directories
3. Set Up the Hive/HCatalog Configuration Files
3.1. Extract the Hive/HCatalog Configuration Files
3.2. Modify the Configuration Files
3.3. Copy the Configuration Files
4. Create Directories on HDFS
5. Download The Database Connector
6. Validate the Installation
7. Installing WebHCat
1. Install the WebHCat RPMs
2. Set Directories and Permissions
3. Modify WebHCat Config Files
4. Set Up the WebHCat Configuration Files
4.1. Copy the Configuration Files
5. Set Up the HDFS User and Prepare WebHCat Directories On HDFS
6. Validate the Installation
8. Installing Apache Oozie
1. Install the Oozie RPMs
1.1. Optional: Add Database Connector JAR Files
2. Set Directories and Permissions
2.1. Create Log Directories
3. Set Up the Oozie Configuration Files
3.1. Extract the Oozie Configuration Files
3.2. Modify the Configuration Files
3.3. Copy the Configuration Files
4. Initialize Database Schema
5. Install the Oozie Sharelib in Hadoop HDFS
6. Validate the Installation
9. Installing HBase and ZooKeeper
1. Install the HBase and ZooKeeper RPMs
2. Set Directories and Permissions
2.1. Create Log and PID Directories
3. Set Up the Configuration Files
3.1. Extract the HBase/ZooKeeper Configuration Files
3.2. Modify the Configuration Files
3.3. Copy the Configuration Files
4. Validate the Installation
4.1. Start HBase and ZooKeeper
4.2. Smoke Test HBase and ZooKeeper
10. Installing Apache Sqoop
1. Install the Sqoop RPMs
2. Optional: Download Database Connector
3. Set Up the Sqoop Configuration
3.1. Extract the Sqoop Configuration Files
3.2. Copy the Configuration Files
4. Validate the Installation
11. Installing Ganglia
1. Install the Ganglia RPMs
2. Install the Configuration Files
2.1. Extract the Ganglia Configuration Files
2.2. Copy the Configuration Files
2.3. Set Up Ganglia Hosts
2.4. Set Up Configurations
2.5. Set Up Hadoop Metrics
3. Validate the Installation
3.1. Start the Ganglia Server
3.2. Start Ganglia Monitoring on All Hosts
3.3. Confirm that Ganglia is Running
12. Installing Nagios
1. Install the Nagio RPMs
2. Install the Configuration Files
2.1. Extract the Nagios Configuration Files
2.2. Copy the Configuration Files
2.3. Set the Nagios Admin Password
2.4. Set the Nagios Admin Email Contact Address
2.5. Register the Hadoop Configuration Files
2.6. Set Hosts
2.7. Set Host Groups
2.8. Set Services
2.9. Set Status
3. Validate the Installation
3.1. Start Nagios
3.2. Confirm Nagios is Running
3.3. Test Nagios Services
3.4. Test Nagios Access
3.5. Test Nagios Alerts
13. Setting Up Security for Manual Installs
1. Preparing Kerberos
1.1. Kerberos Overview
1.2. Installing and Configuring the KDC
1.3. Creating the Database and Setting Up the First Administrator
1.4. Creating Service Principals and Keytab Files for HDP
1.5. Providing the jce-6 Security Jars
2. Configuring HDP
2.1. Configuration Overview
2.2. Creating Mappings Between Principals and OS Service Usernames
2.3. Adding Security Information to Configuration Files

List of Tables

1.1. Typical Service Users and Groups
1.2. Define Users and Groups for Systems
1.3. Define Directories for Core Hadoop
1.4. Define Directories for Ecosystem Components
3.1. core-site.xml
3.2. hdfs-site.xml
3.3. mapred-site.xml
3.4. taskcontroller.cfg
6.1. hive-site.xml
7.1. webhcat-env.xml
8.1. oozie-site.xml
8.2. oozie-env.sh
9.1. zoo.cfg
9.2. hbase-site.xml
12.1. Host Group Parameters
12.2. Core and Monitoring Hosts
12.3. Ecosystem Hosts
13.1. Service Principal Names
13.2. Service Keytab File Names
13.3. core-site.xml
13.4. hdfs-site.xml
13.5. mapred-site.xml
13.6. hbase-site.xml
13.7. hive-site.xml
13.8. oozie-site.xml
13.9. webhcat-site.xml
Legal notices
loading table of contents...