Command Line Installation
Copyright © 2012-2016 Hortonworks, Inc.
 | Except where otherwise noted, this document is licensed under Creative Commons Attribution ShareAlike 4.0 License |
2016-08-29
Abstract
The Hortonworks Data Platform, powered by Apache Hadoop, is a massively scalable and 100% open source platform for storing, processing and analyzing large volumes of data. It is designed to deal with data from many sources and formats in a very quick, easy and cost-effective manner. The Hortonworks Data Platform consists of the essential set of Apache Software Foundation projects that focus on the storage and processing of Big Data, along with operations, security, and governance for the resulting system. This includes Apache Hadoop -- which includes MapReduce, Hadoop Distributed File System (HDFS), and Yet Another Resource Negotiator (YARN) -- along with Ambari, Falcon, Flume, HBase, Hive, Kafka, Knox, Oozie, Phoenix, Pig, Ranger, Slider, Spark, Sqoop, Storm, Tez, and ZooKeeper. Hortonworks is the major contributor of code and patches to many of these projects. These projects have been integrated and tested as part of the Hortonworks Data Platform release process and installation and configuration tools have also been included.
Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of our code back to the Apache Software Foundation. The Hortonworks Data Platform is Apache-licensed and completely open source. We sell only expert technical support, training and partner-enablement services. All of our technology is, and will remain, free and open source.
Please visit the Hortonworks Data Platform page for more information on Hortonworks technology. For more information on Hortonworks services, please visit either the Support or Training page. Feel free to contact us directly to discuss your specific needs.
Contents
- 1. Preparing to Manually Install HDP
- Meeting Minimum System Requirements
- Virtualization and Cloud Platforms
- Configuring Remote Repositories
- Deciding on a Deployment Type
- Collect Information
- Prepare the Environment
- Download Companion Files
- Define Environment Parameters
- Creating System Users and Groups
- Determining HDP Memory Configuration Settings
- Configuring NameNode Heap Size
- Allocating Adequate Log Space for HDP
- Downloading the HDP Maven Artifacts
- 2. Installing Apache ZooKeeper
- 3. Installing HDFS, YARN, and MapReduce
- 4. Setting Up the Hadoop Configuration
- 5. Validating the Core Hadoop Installation
- 6. Installing Apache HBase
- 7. Installing Apache Phoenix
- 8. Installing and Configuring Apache Tez
- 9. Installing Apache Hive and Apache HCatalog
- Installing the Hive-HCatalog Package
- Setting Directories and Permissions
- Setting Up the Hive/HCatalog Configuration Files
- Setting Up the Database for the Hive Metastore
- Setting up RDBMS for use with Hive Metastore
- Creating Directories on HDFS
- Enabling Tez for Hive Queries
- Disabling Tez for Hive Queries
- Configuring Tez with the Capacity Scheduler
- Validating Hive-on-Tez Installation
- Installing Hive LLAP
- Prerequisites
- Preparing to Install LLAP
- Installing LLAP on an Unsecured Cluster
- Installing LLAP on a Secured Cluster
- Stopping the LLAP Service
- Tuning LLAP for Performance
- 10. Installing Apache Pig
- 11. Installing Apache WebHCat
- 12. Installing Apache Oozie
- 13. Installing Apache Ranger
- 14. Installing Hue
- 15. Installing Apache Sqoop
- 16. Installing Apache Mahout
- 17. Installing and Configuring Apache Flume
- 18. Installing and Configuring Apache Storm
- 19. Installing and Configuring Apache Spark
- 20. Installing and Configuring Apache Kafka
- 21. Installing and Configuring Zeppelin
- 22. Installing Apache Accumulo
- 23. Installing Apache Falcon
- 24. Installing Apache Knox
- 25. Installing Apache Slider
- 26. Installing and Configuring Apache Atlas
- Atlas Prerequisites
- Installing Atlas
- Installing Atlas Metadata Apache Hive Plugin
- Configuring Apache Hive Hook
- Configuring Atlas For Apache Kafka Notifications
- Configuring Apache Storm Hook
- Configuring the Graph Database
- Configuring Apache Falcon Hook
- Configuring Apache Sqoop Hook
- Configuring for Secure Clusters
- Configuring Atlas in a Kerberized Cluster
- Validating Atlas
- 27. Setting Up Kerberos Security for Manual Installs
- 28. Uninstalling HDP
List of Tables
- 1.1. Directories Needed to Install Core Hadoop
- 1.2. Directories Needed to Install Ecosystem Components
- 1.3. Define Users and Groups for Systems
- 1.4. Typical System Users and Groups
- 1.5. hdp-configuration-utils.py Options
- 1.6. Reserved Memory Recommendations
- 1.7. Recommended Container Size Values
- 1.8. YARN and MapReduce Configuration Values
- 1.9. Example Value Calculations Without HBase
- 1.10. Example Value Calculations with HBase
- 1.11. Recommended NameNode Heap Size Settings
- 8.1. Tez Configuration Parameters
- 9.1. Hive Configuration Parameters
- 9.2.
- 9.3.
- 9.4. LLAP Properties to Set in hive-site.xml
- 9.5. HiveServer2 Properties to Set in hive-site.xml to Enable Concurrent Queries with LLAP
- 9.6. Properties to Set in hive-site.xml for Secured Clusters
- 9.7. Properties to Set in ssl-server.xml for LLAP on Secured Clusters
- 9.8. LLAP Package Parameters
- 11.1. Hadoop core-site.xml File Properties
- 13.1. install.properties Entries
- 13.2. Properties to Update in the install.properties File
- 13.3. Properties to Edit in the install.properties File
- 13.4. Properties to Edit in the install.properties File
- 13.5. Properties to Edit in the install.properties File
- 13.6. HBase Properties to Edit in the
install.propertiesfile - 13.7. Hive-Related Properties to Edit in the install.properties File
- 13.8. Knox-Related Properties to Edit in the install.properties File
- 13.9. Storm-Related Properties to Edit in the
install.propertiesfile - 13.10. install.properties File Properties
- 13.11. Run-time Directory Structure
- 17.1. Flume 1.5.2 Dependencies
- 18.1. Required jaas.conf Sections for Cluster Nodes
- 18.2. Supported Authorizers
- 18.3. storm.yaml Configuration File Properties
- 18.4. worker-launcher.cfg File Configuration Properties
- 18.5. multitenant-scheduler.yaml Configuration File Properties
- 19.1. Prerequisites for running Spark 1.6
- 20.1. Kafka Configuration Properties
- 21.1. Installation Prerequisites
- 26.1. Atlas Cluster Prerequisites