Chapter 19. Installing and Configuring Apache Spark
This section describes how to install and configure Apache Spark for HDP:
For more information about Spark on HDP (including how to install Spark using Ambari), see the Spark Guide.
- 1. Preparing to Manually Install HDP
- Meeting Minimum System Requirements
- Virtualization and Cloud Platforms
- Configuring Remote Repositories
- Deciding on a Deployment Type
- Collect Information
- Prepare the Environment
- Download Companion Files
- Define Environment Parameters
- Creating System Users and Groups
- Determining HDP Memory Configuration Settings
- Configuring NameNode Heap Size
- Allocating Adequate Log Space for HDP
- Downloading the HDP Maven Artifacts
- 2. Installing Apache ZooKeeper
- 3. Installing HDFS, YARN, and MapReduce
- 4. Setting Up the Hadoop Configuration
- 5. Validating the Core Hadoop Installation
- 6. Installing Apache HBase
- 7. Installing Apache Phoenix
- 8. Installing and Configuring Apache Tez
- 9. Installing Apache Hive and Apache HCatalog
- Installing the Hive-HCatalog Package
- Setting Directories and Permissions
- Setting Up the Hive/HCatalog Configuration Files
- Setting Up the Database for the Hive Metastore
- Setting up RDBMS for use with Hive Metastore
- Creating Directories on HDFS
- Enabling Tez for Hive Queries
- Disabling Tez for Hive Queries
- Configuring Tez with the Capacity Scheduler
- Validating Hive-on-Tez Installation
- Installing Hive LLAP
- Prerequisites
- Preparing to Install LLAP
- Installing LLAP on an Unsecured Cluster
- Installing LLAP on a Secured Cluster
- Stopping the LLAP Service
- Tuning LLAP for Performance
- 10. Installing Apache Pig
- 11. Installing Apache WebHCat
- 12. Installing Apache Oozie
- 13. Installing Apache Ranger
- 14. Installing Hue
- 15. Installing Apache Sqoop
- 16. Installing Apache Mahout
- 17. Installing and Configuring Apache Flume
- 18. Installing and Configuring Apache Storm
- 19. Installing and Configuring Apache Spark
- 20. Installing and Configuring Apache Kafka
- 21. Installing and Configuring Zeppelin
- 22. Installing Apache Accumulo
- 23. Installing Apache Falcon
- 24. Installing Apache Knox
- 25. Installing Apache Slider
- 26. Installing and Configuring Apache Atlas
- Atlas Prerequisites
- Installing Atlas
- Installing Atlas Metadata Apache Hive Plugin
- Configuring Apache Hive Hook
- Configuring Atlas For Apache Kafka Notifications
- Configuring Apache Storm Hook
- Configuring the Graph Database
- Configuring Apache Falcon Hook
- Configuring Apache Sqoop Hook
- Configuring for Secure Clusters
- Configuring Atlas in a Kerberized Cluster
- Validating Atlas
- 27. Setting Up Kerberos Security for Manual Installs
- 28. Uninstalling HDP