Non-Ambari Cluster Installation Guide
Copyright © 2012-2015 Hortonworks, Inc.
 | Except where otherwise noted, this document is licensed under Creative Commons Attribution ShareAlike 4.0 License |
2015-07-21
Abstract
The Hortonworks Data Platform, powered by Apache Hadoop, is a massively scalable and 100% open source platform for storing, processing and analyzing large volumes of data. It is designed to deal with data from many sources and formats in a very quick, easy and cost-effective manner. The Hortonworks Data Platform consists of the essential set of Apache Software Foundation projects that focus on the storage and processing of Big Data, along with operations, security, and governance for the resulting system. This includes Apache Hadoop -- which includes MapReduce, Hadoop Distributed File System (HDFS), and Yet Another Resource Negotiator (YARN) -- along with Ambari, Falcon, Flume, HBase, Hive, Kafka, Knox, Oozie, Phoenix, Pig, Ranger, Slider, Spark, Sqoop, Storm, Tez, and ZooKeeper. Hortonworks is the major contributor of code and patches to many of these projects. These projects have been integrated and tested as part of the Hortonworks Data Platform release process and installation and configuration tools have also been included.
Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of our code back to the Apache Software Foundation. The Hortonworks Data Platform is Apache-licensed and completely open source. We sell only expert technical support, training and partner-enablement services. All of our technology is, and will remain, free and open source.
Please visit the Hortonworks Data Platform page for more information on Hortonworks technology. For more information on Hortonworks services, please visit either the Support or Training page. Feel free to contact us directly to discuss your specific needs.
Contents
- 1. Getting Ready to Install
- Meet Minimum System Requirements
- Virtualization and Cloud Platforms
- Configure the Remote Repositories
- Decide on Deployment Type
- Collect Information
- Prepare the Environment
- Download Companion Files
- Define Environment Parameters
- [Optional] Create System Users and Groups
- Determine HDP Memory Configuration Settings
- Configuring NameNode Heap Size
- Allocate Adequate Log Space for HDP
- Download the HDP Maven Artifacts
- 2. Installing Apache ZooKeeper
- 3. Installing HDFS, YARN, and MapReduce
- 4. Setting Up the Hadoop Configuration
- 5. Validating the Core Hadoop Installation
- 6. Installing Apache HBase
- 7. Installing Apache Phoenix
- 8. Installing and Configuring Apache Tez
- 9. Installing Apache Hive and Apache HCatalog
- Installing the Hive-HCatalog Package
- Setting Directories and Permissions
- Setting Up the Hive/HCatalog Configuration Files
- Setting Up the Database for the Hive Metastore
- Setting up RDBMS for use with Hive Metastore
- Creating Directories on HDFS
- Enabling Tez for Hive Queries
- Disabling Tez for Hive Queries
- Configuring Tez with the Capacity Scheduler
- Validating Hive-on-Tez Installation
- 10. Installing Apache Pig
- 11. Installing Apache WebHCat
- 12. Installing Apache Oozie
- 13. Installing Apache Ranger
- 14. Installing Hue
- 15. Installing Apache Sqoop
- 16. Installing Apache Mahout
- 17. Installing and Configuring Apache Flume
- 18. Installing and Configuring Apache Storm
- 19. Installing and Configuring Apache Spark
- 20. Installing and Configuring Apache Kafka
- 21. Installing Apache Accumulo
- 22. Installing Apache Falcon
- 23. Installing Apache Knox
- 24. Installing Apache Slider
- 25. Installing and Configuring Apache Atlas
- 26. Setting Up Security for Manual Installs
- 27. Uninstalling HDP
List of Tables
- 1.1. Define Directories for Core Hadoop
- 1.2. Define Directories for Ecosystem Components
- 1.3. Define Users and Groups for Systems
- 1.4. Typical System Users and Groups
- 1.5. hdp-configuration-utils.py Options
- 1.6. Reserved Memory Recommendations
- 1.7. Recommended Values
- 1.8. YARN and MapReduce Configuration Setting Value Calculations
- 1.9. Example Value Calculations
- 1.10. Example Value Calculations
- 1.11. NameNode Heap Size Settings
- 8.1. Tez Configuration Parameters
- 9.1. Hive Configuration Parameters
- 11.1. Hadoop core-site.xml File Properties
- 13.1. install.properties Entries
- 13.2. Properties to Update in the install.properties File
- 13.3. Properties to Edit in the install.properties File
- 13.4. Properties to Edit in the install.properties File
- 13.5. Properties to Edit in the install.properties File
- 13.6. HBase Properties to Edit in the
install.propertiesfile - 13.7. Hive-Related Properties to Edit in the install.properties File
- 13.8. Knox-Related Properties to Edit in the install.properties File
- 13.9. Storm-Related Properties to Edit in the
install.propertiesfile - 14.1. Hue-Supported Browsers
- 14.2. Hue Dependencies on HDP Components
- 14.3. Variables to Configure HDFS Cluster
- 14.4. Variables to Configure the YARN Cluster
- 14.5. Beeswax Configuration Values
- 17.1. Flume 1.5.2 Dependencies
- 18.1. Required jaas.conf Sections for Cluster Nodes
- 18.2. Supported Authorizers
- 18.3. storm.yaml Configuration File Properties
- 18.4. worker-launcher.cfg File Configuration Properties
- 18.5. multitenant-scheduler.yaml Configuration File Properties
- 19.1. Spark Cluster Prerequisites
- 20.1. Kafka Configuration Properties
- 25.1. Atlas Cluster Prerequisites
- 26.1. Service Principals
- 26.2. Service Keytab File Names
- 26.3. General core-site.xml, Knox, and Hue
- 26.4. core-site.xml Master Node Settings -- Knox Gateway
- 26.5. core-site.xml Master Node Settings -- Hue
- 26.6. hdfs-site.xml File Property Settings
- 26.7. yarn-site.xml Property Settings
- 26.8. mapred-site.xml Property Settings
- 26.9. hbase-site.xml Property Settings -- HBase Server
- 26.10. hive-site.xml Property Settings
- 26.11. oozie-site.xml Property Settings
- 26.12. webhcat-site.xml Property Settings