The local mirror setup for Option II is shown in the following illustration:
To configure local repositories for HDP deployment, your system must meet the following minimum requirements:
Your mirror server host must run on one of the following operating systems:
64 bit RHEL, CentOS, or Oracle Linux
64 bit SUSE Linux Enterprise Server (SLES) 11 SP1
This server and the cluster nodes must all run the same OS.
Note To support repository mirroring for heterogeneous clusters requires a more complex procedure than the one documented here.
The mirror server host must have several GB of storage available.
The firewall must allow all cluster nodes (the servers on which you want to install HDP) to access this server.
Make sure that the mirror server has the
yum
orzypper
installed.If your mirror host uses RHEL, CentOS, or Oracle Linux, add the
yum-utils
andcreaterepo
packages to the mirror server.yum install yum-utils createrepo
If your mirror host uses SLES, install the following required packages:
zypper -n --no-gpg-checks install apache2 apache2-prefork apache2-utils python-curl python-gobject2 python-gpgme python-urlgrabber python-iniparse wget curl yum-metadata-parser yum yum-updatesd yum-utils createrepo
Temporarily reconfigure your firewall to allow Internet access from your mirror server host.
Download the appropriate HDP client configuration file and save it in the appropriate directory on the mirror server host.
Table 4.4. Deploying HDP - Option II Cluster OS HDP Repository Tarballs RHEL/CentOS/Oracle Linux 5.x
HDP Repository:
wget http://public-repo-1.hortonworks.com/HDP-1.2.0-1/centos5/1.x/updates/1.2.1/hdp.repo -O /etc/yum.repos.d/hdp.repo
Ambari Repository (Optional):
wget http://public-repo-1.hortonworks.com/ambari/centos5/1.x/updates/1.2.2.5/ambari.repo -O /etc/yum.repos.d/ambari.repo
RHEL/CentOS/Oracle Linux 6.x
HDP Repository:
wget http://public-repo-1.hortonworks.com/HDP-1.2.0-1/centos6/1.x/updates/1.2.1/hdp.repo -O /etc/yum.repos.d/hdp.repo
Ambari Repository (Optional):
wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.2.2.5/ambari.repo -O /etc/yum.repos.d/ambari.repo
SLES 11
HDP Repository:
wget http://public-repo-1.hortonworks.com/HDP-1.2.0-1/suse11/1.x/updates/1.2.1/hdp.repo -O /etc/zypp/repos.d/hdp.repo
Ambari Repository (Optional):
wget http://public-repo-1.hortonworks.com/ambari/suse11/1.x/updates/1.2.2.5/ambari.repo -O /etc/zypp/repos.d/ambari.repo
Note If you are using Ambari to perform the HDP installation, you will need to setup the Ambari repository using the information provided above.
Create an HTTP server.
On the mirror server, install an HTTP server (such as Apache
httpd
) using the instructions provided here.Activate this web server.
Make sure that the firewall settings (if any) allow inbound HTTP access from your cluster nodes to your mirror server.
Note If you are using EC2, make sure that SELinux is disabled.
If your mirror server uses SLES, modify the
default-server.conf
file to enable the docs root folder listing.sed -e "s/Options None/Options Indexes MultiViews/ig" /etc/apache2/default-server.conf > /tmp/tempfile.tmp mv /tmp/tempfile.tmp /etc/apache2/default-server.conf
On your mirror server, create a directory for your web server.
For example, from a shell window, type:
For RHEL/CentOS/Oracle Linux:
mkdir –p
/var/www/html/hdp/
For SLES:
mkdir –p
/srv/www/htdocs/rpms/
If you are using a symlink, enable the
followsymlinks
on your web server.
Copy the contents of entire HDP repository from the remote yum server to your local mirror server.
Continuing the previous example, from a shell window, type:
Navigate to the directory created on your web server previously:
For RHEL/CentOS/Oracle Linux:
cd /var/www/html/hdp/
For SLES:
cd /srv/www/htdocs/rpms/
Copy the contents of entire HDP repository from the remote server to your local mirror server.
HDP Repository:
reposync -r HDP-1.x reposync -r Updates-HDP-1.x
Ambari Repository (Optional):
reposync –r ambari-1.x reposync –r Updates-ambari-1.2.2.5
Generate appropriate metadata. This step defines each directory as a repository.
From a shell window, type:
For RHEL/CentOS:
HDP Repository:
createrepo /var/www/html/hdp/HDP-1.x createrepo /var/www/html/hdp/Updates-HDP-1.2.1
Ambari Repository (Optional):
createrepo /var/www/html/hdp/ambari-1.x createrepo /var/www/html/hdp/Updates-ambari-1.2.2.5
For SLES:
HDP Repository:
createrepo /srv/www/htdocs/rpms/hdp/HDP-1.x createrepo /var/www/html/hdp/Updates-HDP-1.2.1
Ambari Repository (Optional):
createrepo /srv/www/htdocs/rpms/hdp/ambari-1.x createrepo/srv/www/htdocs/rpms/hdp/Updates-ambari-1.2.2.5
You should see a new
repodata
directory under the HDP and Updates-HDP-1.x directories.If using Ambari, you should also see the
repodata
directory under theambari-1.x
andUpdates-ambari-1.x
directories.
Verify the configuration.
Browse to the following URLs:
HDP Repository:
http://$yourwebserver/HDP-1.x/$os/1.x/updates/1.2.1
Ambari Repository (Optional):
http://$yourwebserver/ambari-1.x/$os/1.x/updates/1.2.2.4
where,
$os
can be centos5, centos6, or suse11.You should now see a directory listing for all the HDP components.
At this point, it is okay to disable external Internet access for the mirror server, so that the mirror server is once again entirely within your datacenter firewall.
Configure the
yum
orzypper
clients on all the nodes in your cluster.Edit the repo file(s), changing the value of the baseurl property to the local mirror URL.
Edit the
/etc/yum.repos.d/hdp.repo
file changing the baseurl property as shown below:[HDP-1.x] name=Hortonworks Data Platform Version - HDP-1.x baseurl=http://$yourwebserver/HDP/$os/$yourrelease/GA gpgcheck=1 gpgkey=//public-repo-1.hortonworks.com/HDP-1.2.0-1/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.15] name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.15 baseurl=http://$yourwebserver/HDP-UTILS-1.1.0.15/repos/$os gpgcheck=1 gpgkey=//public-repo-1.hortonworks.com/HDP-1.2.0-1/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [Updates-HDP-1.2.1] name=HDP-1.2.1 - Updates baseurl=http://$yourwebserver/HDP/$os/1.x/updates/1.2.1 gpgcheck=1 gpgkey=//public-repo-1.hortonworks.com/HDP-1.2.0-1/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
Edit the
/etc/yum.repos.d/ambari.repo
file changing the baseurl property as shown below:[ambari-1.x] name=Ambari 1.x baseurl=http://$yourwebserver/ambari/$os/1.x/GA/ambari.repo gpgcheck=1 gpgkey=//public-repo-1.hortonworks.com/ambari/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.15] name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.15 baseurl=http://$yourwebserver/HDP-UTILS-1.1.0.15/repos/$os gpgcheck=0 gpgkey=//public-repo-1.hortonworks.com/ambari/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [Updates-ambari-1.2.2.5] name=ambari-1.2.2.5 - Updates baseurl=http://$yourwebserver/ambari/$os/1.x/updates/1.2.2.5 gpgcheck=1 gpgkey=//public-repo-1.hortonworks.com/ambari/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
where,
$os
can be centos5, centos6, or suse11.Copy the yum/zypper client configuration file to all nodes in your cluster.
For RHEL and CentOS: Use scp or pdsh to copy the client yum configuration file to
/etc/yum.repos.d/
directory on every node in the cluster.For SLES: On every node, invoke the following command:
HDP Repository:
zypper addrepo -r http://$yourwebserver/HDP/suse11/1.x/updates/1.2.1/hdp.repo
Ambari Repository (Optional):
zypper addrepo -r http://$yourwebserver/ambari/suse11/1.x/updates/1.2.2.4/ambari.repo
If you are using Ambari, verify the configuration by deploying Ambari server on one of the cluster nodes.
yum install ambari-server or zypper install ambari-server
If your cluster runs RHEL, CentOS, or Oracle Linux, and if you have multiple repositories configured in your environment, deploy the following plugin on all the nodes in your cluster.
Install the plugin.
For RHEL/CentOS/Oracle Linux v5.x
yum install yum-priorities
For RHEL/CentOS/Oracle Linux v6.x
yum install yum-plugin-priorities
Edit the
/etc/yum/pluginconf.d/priorities.conf
file to add the following:[main] enabled=1 gpgcheck=0