Meeting Minimum System Requirements

Hardware Recommendations

Although there is no single hardware requirement for installing HDP, there are some basic guidelines. A complete installation of HDP 2.6.0 consumes about 6.5 GB of disk space. For more information about HDP hardware recommendations, see the HDP Cluster Planning Guide.

Operating System Requirements

Refer to the Support Matrix for information regarding supported opeating systems.

Software Requirements

You must install the following software on each of your hosts:

apt-get (for Ubuntu and Debian)
chkconfig (Ubuntu and Debian)
curl
php_curl (for SLES)
reposync (might not be installed by default on all SLES hosts)
rpm (for RHEL, CentOS, or SLES)
scp
tar
unzip
wget
yum (for RHEL or CentOS)
zypper (for SLES)

In addition, if you are creating local mirror repositories as part of the installation process and you are using RHEL, CentOS, or SLES, you need the following utilities on the mirror repo server:

createrepo
reposync
yum-utils

See Deploying HDP in Production Data Centers with Firewalls.

JDK Requirements

Your system must have the correct Java Development Kit (JDK) installed on all cluster nodes.

Refer to the Support Matrix for information regarding supported JDKs.

	Important
	Before enabling Kerberos in the cluster, you must deploy the Java Cryptography Extension (JCE) security policy files on all hosts in the cluster. See Installing the JCE for more information.

The following sections describe how to install and configure the JDK.

Manually Installing Oracle JDK 1.7 or 1.8

Use the following instructions to manually install JDK 1.7 or JDK 1.8:

If you do not have a /usr/java directory, create one:
mkdir /usr/java
Download the Oracle 64-bit JDK (jdk-7u67-linux-x64.tar.gz or jdk-8u51-linux-x64.tar.gz) from the Oracle download site.
Open a web browser and navigate to http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html or http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html.
Copy the downloaded jdk.tar.gz file to the /usr/java directory.
Navigate to the /usr/java directory and extract the jdk.tar.gz file:
```
cd /usr/java && tar zxvf jdk-7u67-linux-x64.tar.gz
              or
cd /usr/java  tar zxvf jdk-8u51-linux-x64.tar.gz
```
The JDK files are extracted into a /usr/java/jdk1.7.0_67 directory or a /usr/java/jdk1.8.0_51 directory.
Create a symbolic link (symlink) to the JDK:
ln -s /usr/java/jdk1.7.0_67 /usr/java/default
or
ln -s /usr/java/jdk1.8.0_51 /usr/java/default
Set the JAVA_HOME and PATH environment variables:
```
export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH
```

Verify that Java is installed in your environment:

java -version

You should see output similar to the following:

java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.67-b01, mixed mode)

Manually Installing OpenJDK 1.7

OpenJDK7 on HDP 2.6.0 does not work if you are using SLES as your OS. Use the following instructions to manually install OpenJDK 1.7 on a Linux OS other than SLES:

Check your currently installed version from a terminal window:
java -version
If the JDK version is earlier than 7, uninstall it, as in the following example using Centos:
rpm -qa | grep java
yum remove {java-1.*}
If you followed Step 2, verify that Java is uninstalled:
which java
(Optional) Download OpenJDK 1.7 RPMs from the appropriate command line:
RedHat, CentOS, or Oracle Linux:
yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel
Ubuntu or Debian:
apt-get install openjdk-7-jdk
(Optional) Create symbolic links (symlinks) to the JDK:
mkdir /usr/java
ln -s /usr/hdp/current/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64 /usr/java/default
ln -s /usr/java/default/bin/java /usr/bin/java
(Optional) Set your environment to define JAVA_HOME to put the Java Virtual Machine and the Java compiler on your path:
```
export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH
```
(Optional) Verify that Java is installed in your environment by running the following command from the command-line console:
java -version
You should see output similar to the following:
```
openjdk version "1.7.0"
OpenJDK Runtime Environment (build 1.7.0)
OpenJDK Client VM (build 20.6-b01, mixed mode)
```

Manually Installing the JCE

Unless you are using OpenJDK with unlimited-strength JCE, you must manually install the Java Cryptography Extension (JCE) security policy files on all hosts in the cluster:

Obtain the JCE policy file appropriate for the JDK version in your cluster:
- Oracle JDK 1.8
  http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
- Oracle JDK 1.7
  http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html
Save the policy file archive in a temporary location.
On each host in the cluster, add the unlimited security policy JCE jars to $JAVA_HOME/jre/lib/security/.
For example, run the following command to extract the policy jars into the JDK installed on your host:
```
unzip -o -j -q jce_policy-8.zip -d /usr/jdk64/jdk1.8.0_60/jre/lib/security/
```

Metastore Database Requirements

If you are installing Apache projects Hive and HCatalog, Oozie, Hue, or Ranger, you must install a database to store metadata information in the metastore. You can either use an existing database instance or install a new instance manually.

Refer to the Support Matrix for information regarding supported metastore databases.

The following sections describe how to install and configure the metastore database.

Metastore Database Prerequisites

The database administrator must create the following users and specify the following values:

For Apache Hive: hive_dbname, hive_dbuser, and hive_dbpasswd.
For Apache Oozie: oozie_dbname, oozie_dbuser, and oozie_dbpasswd.
Note
By default, Hive uses the Derby database for the metastore. However, Derby is not supported for production systems.
For Hue: Hue user name and Hue user password
For Apache Ranger: RANGER_ADMIN_DB_NAME

Installing and Configuring PostgreSQL

The following instructions explain how to install PostgreSQL as the metastore database. See your third-party documentation for instructions on how to install other supported databases.

	Important
	Prior to using PostgreSQL as your Hive metastore, consult with the offiical PostgreSQL documentation and ensure you are using a JDBC 4+ driver that corresponds to your implementation of PostgreSQL.

Installing PostgreSQL on RHEL, CentOS, and Oracle Linux

Use the following instructions to install a new instance of PostgreSQL:

Using a terminal window, connect to the host machine where you plan to deploy a PostgreSQL instance:
yum install postgresql-server
Start the instance:
/etc/init.d/postgresql start
For some newer versions of PostgreSQL, you might need to execute the command /etc/init.d/postgresql initdb.
Reconfigure PostgreSQL server:
- Edit the /var/lib/pgsql/data/postgresql.conf file.
  Change the value of #listen_addresses = 'localhost' to listen_addresses = '*'.
- Edit the /var/lib/pgsql/data/postgresql.conf file.
  Remove comments from the "port = " line and specify the port number (default 5432).
- Edit the /var/lib/pgsql/data/pg_hba.conf file by adding
  the following:
  host all all 0.0.0.0/0 trust
- If you are using PostgreSQL v9.1 or later, add the following to the /var/lib/pgsql/data/postgresql.conf file:
  standard_conforming_strings = off

Create users for PostgreSQL server by logging

in as the root user and entering the following syntax:

echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres
echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres
echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres

The previous syntax should have the following values:

$postgres is the postgres user.
$user is the user you want to create.
$dbname is the name of your PostgreSQL database.

	Note
	For access to the Hive metastore, you must create hive_dbuser after Hive has been installed, and for access to the Oozie metastore, you must create oozie_dbuser after Oozie has been installed.

On the Hive metastore host, install the connector:
yum install postgresql-jdbc*
Confirm that the .jar file is in the Java share directory:
ls -l /usr/share/java/postgresql-jdbc.jar

Installing PostgreSQL on SUSE Linux Enterprise Server (SLES)

To install a new instance of PostgreSQL:

Connect to the host machine where you plan to deploy the PostgreSQL instance.
At a terminal window, enter:
zypper install postgresql-server
Start the instance.
/etc/init.d/postgresql start
Note
For some newer versions of PostgreSQL, you might need to execute the command:
/etc/init.d/postgresql initdb
Reconfigure the PostgreSQL server:
- Edit the /var/lib/pgsql/data/postgresql.conf file.
  Change the value of #listen_addresses = 'localhost' to listen_addresses = '*'
- Edit the /var/lib/pgsql/data/postgresql.conf file.
  Change the port setting #port = 5432 to port = 5432
- Edit the /var/lib/pgsql/data/pg_hba.conf file.
  Add the following:
  host all all 0.0.0.0/0 trust
- Optional: If you are using PostgreSQL v9.1 or later, add the following to the /var/lib/pgsql/data/postgresql.conf file:
  standard_conforming_strings = off

Create users for PostgreSQL server.

Log in as the root and enter:

echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres
echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres
echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres

Where:

$postgres is the postgres user
$user is the user you want to create
$dbname is the name of your PostgreSQL database

	Note
	For access to the Hive metastore, create hive_dbuser after Hive has been installed, and for access to the Oozie metastore, create oozie_dbuser after Oozie has been installed.

On the Hive Metastore host, install the connector.
zypper install -y postgresql-jdbc
Copy the connector .jar file to the Java share directory.
cp /usr/share/pgsql/postgresql-*.jdbc3.jar /usr/share/java/postgresql-jdbc.jar
Confirm that the .jar is in the Java share directory.
ls /usr/share/java/postgresql-jdbc.jar
Change the access mode of the .jar file to 644.
chmod 644 /usr/share/java/postgresql-jdbc.jar

Installing PostgreSQL on Ubuntu and Debian

To install a new instance of PostgreSQL:

Connect to the host machine where you plan to deploy PostgreSQL instance.
At a terminal window, enter:
apt-get install postgresql-server
Start the instance.
Note
For some newer versions of PostgreSQL, you might need to execute the command:
/etc/init.d/postgresql initdb
Reconfigure PostgreSQL server:
- Edit the /var/lib/pgsql/data/postgresql.conf file.
  Change the value of #listen_addresses = 'localhost' to listen_addresses = '*'
- Edit the /var/lib/pgsql/data/postgresql.conf file.
  Change the port setting from #port = 5432 to port = 5432
- Edit the /var/lib/pgsql/data/pg_hba.conf
  Add the following:
  host all all 0.0.0.0/0 trust
- Optional: If you are using PostgreSQL v9.1 or later, add the following to the /var/lib/pgsql/data/postgresql.conf file:
  standard_conforming_strings = off
Create users for PostgreSQL server.
Log in as the root and enter:
```
echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres
echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres
echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres 
```
Where:
$postgres is the postgres user, $user is the user you want to create, and $dbname is the name of your PostgreSQL database.
Note
For access to the Hive metastore, create hive_dbuser after Hive has been installed, and for access to the Oozie metastore, create oozie_dbuser after Oozie has been installed.
On the Hive Metastore host, install the connector.
apt-get install -y libpostgresql-jdbc-java
Copy the connector .jar file to the Java share directory.
cp /usr/share/java/postgresql-*jdbc3.jar /usr/share/java/postgresql-jdbc.jar
Confirm that the .jar is in the Java share directory.
ls /usr/share/java/postgresql-jdbc.jar
Change the access mode of the .jar file to 644.
chmod 644 /usr/share/java/postgresql-jdbc.jar

Installing and Configuing MariaDB

This section describes how to install MariaDB as the metastore database. For instructions on how to install other supported databases, see your third-party documentation.

For additional information regarding MariaDB, see MariaDB.

Installing MariaDB on RHEL and CentOS

	Important
	If you are installing on CentOS or RHEL, it is highly recommended that you install from a repository using yum.

Follow these steps to install a new instance of MariaDB on RHEL and CentOS:

There are YUM repositories for several YUM-based Linux distributions. Use the Maria DB Downloads page to generate the YUM repopository.
Move the MariaDB repo file to the directory /etc/yum.repos.d/.
It is suggested that you name your file MariaDB.repo.
The following is an example MariaDB.repo file for CentOS 7:
```
[mariadb]
name=MariaDB
baseurl=http://yum.mariadb.org/10.1/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
```
In this example the gpgkey line automatically fetches the GPG key that is used to sign the repositories. gpgkey enables yum and rpm to verify the integrity of the packages that it downloads.The id of MariaDB's signing key is 0xcbcb082a1bb943db. The short form of the id is 0x1BB943DB and the full key fingerprint is:1993 69E5 404B D5FC 7D2F E43B CBCB 082A 1BB9 43DB.
If you want to fix the version to an older version, follow the instructions on Adding the MariaDB YUM Repository.
If you do not have the MariaDB GPG signing key installed, YUM prompts you to install it after downloading the packages. If you are prompted to do so, install the MariaDB GPG signing key.
Use the following command to install MariaDB:
```
sudo yum install MariaDB-server MariaDB-client
```
If you already have the MariaDB-Galera-server package installed, you might need to remove it prior to installing MariaDB-server. If you need to remove MariaDB-Galera-server, use the following command:
```
sudo yum remove MariaDB-Galera-server
```
No databases are removed when the MariaDB-Galera-server rpm package is removed, though with any upgrade, it is best to have backups.
Install TokuDB with YUM by following the directions at Enabling TokuDB.
Use one of the following commands to start MariaDB:
- If your system is using systemctl:
```
sudo systemctl start mariadb
```
- If your system is not using systemctl:
- ```
sudo /etc/init.d/mysql start
```

Installing and Configuring MySQL

This section describes how to install MySQL as the metastore database. For instructions on how to install other supported databases, see your third-party documentation.

	Important
	When you use MySQL as your Hive metastore, you must use `mysql-connector-java-5.1.35.zip` or later JDBC driver.

Installing MySQL on RHEL and CentOS

To install a new instance of MySQL:

Connect to the host machine you plan to use for Hive and HCatalog.
Install MySQL server.
From a terminal window, enter:
yum install mysql-server (for CentOS6)
yum install mysql-community-release For CentOS7, install MySQL server from the HDP-Utils repository.
Start the instance.
/etc/init.d/mysqld start
Set the root user password using the following command format:
mysqladmin -u root password $mysqlpassword
For example, use the following command to set the password to "root":
mysqladmin -u root password root
Remove unnecessary information from log and STDOUT:
mysqladmin -u root 2>&1 >/dev/null
Log in to MySQL as the root user:
mysql -u root -proot
In this syntax, "root" is the root user password.

Log in as the root user, create the “dbuser,” and grant dbuser adequate privileges:

[root@c6402 /]# mysql -u root -proot

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE USER 'dbuser'@'localhost' IDENTIFIED BY 'dbuser';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'localhost';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'dbuser'@'%' IDENTIFIED BY 'dbuser';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'%' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql>

Use the exit command to exit MySQL.
You should now be able to reconnect to the database as "dbuser" by using the following command:
mysql -u dbuser -pdbuser
After testing the dbuser login, use the exit command to exit MySQL.
Install the MySQL connector .jar file:
yum install mysql-connector-java*

SUSE Linux Enterprise Server (SLES)

To install a new instance of MySQL:

Connect to the host machine you plan to use for Hive and HCatalog.
Install MySQL server.
From a terminal window, enter:
zypper install mysql-server
Start the instance:
/etc/init.d/mysqld start
Set the root user password by using the following command format:
mysqladmin -u root password $mysqlpassword
For example, to set the password to "root", use the following command:
mysqladmin -u root password root
Remove unnecessary information from log and STDOUT:
mysqladmin -u root 2>&1 >/dev/null
Log in to MySQL as the root user:
mysql -u root -proot

Log in as the root user, create dbuser, and grant dbuser adequate privileges:

[root@c6402 /]# mysql -u root -proot

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE USER 'dbuser'@'localhost' IDENTIFIED BY 'dbuser';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'localhost';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'dbuser'@'%' IDENTIFIED BY 'dbuser';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'%' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql>

Use the exit command to exit MySQL.
You should now be able to reconnect to the database as dbuser by using the following command:
mysql -u dbuser -pdbuser
After testing the dbuser login, use the exit command to exit MySQL.
Install the MySQL connector .jar file:
zypper install mysql-connector-java*

Ubuntu/Debian

To install a new instance of MySQL:

Connect to the host machine you plan to use for Hive and HCatalog.
Install MySQL server.
From a terminal window, enter:
apt-get install mysql-server
Start the instance.
/etc/init.d/mysql start
Set the root user password using the following command format:
mysqladmin -u root password $mysqlpassword
For example, to set the password to "root":
mysqladmin -u root password root
Remove unnecessary information from log and STDOUT.
mysqladmin -u root 2>&1 >/dev/null
Log in to MySQL as the root user:
mysql -u root -proot

Log in as the root user, create the dbuser, and grant it adequate privileges.

This user provides access to the Hive metastore. Use the following series of commands (shown here with the returned responses) to create dbuser with password dbuser.

[root@c6402 /]# mysql -u root -proot

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE USER 'dbuser'@'localhost' IDENTIFIED BY 'dbuser';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'localhost';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'dbuser'@'%' IDENTIFIED BY 'dbuser';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'%' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql>

Use the exit command to exit MySQL.
You should now be able to reconnect to the database as dbuser, using the following command:
mysql -u dbuser -pdbuser
After testing the dbuser login, use the exit command to exit MySQL.
Install the MySQL connector JAR file.
apt-get install mysql-connector-java*

Configuring Oracle as the Metastore Database

You can select Oracle as the metastore database. For instructions on how to install the databases, see your third-party documentation. To configure Oracle as the Hive Metastore, install HDP and Hive, and then follow the instructions in "Set up Oracle DB for use with Hive Metastore" in this guide.

	Note
	By default, Hive uses the Derby database for the metastore. However, Derby is not supported for production systems.