Option II - Mirror server has temporary or continuous access to the Internet

Complete the following instructions to set up a mirror server that has temporary access to the Internet:

Check Your Prerequisites.

Select a local mirror server host with the following characteristics:

The server OS is Debian 7, CentOS (6,7), RHEL (6,7), Oracle Linux(6,7), SLES (11,12), or Ubuntu (12,14,16), and has several GB of storage available.

The local mirror server and the cluster nodes must have the same OS. If they are not running CentOS or RHEL, the mirror server must not be a member of the Hadoop cluster.

	Note
	To support repository mirroring for heterogeneous clusters requires a more complex procedure than the one documented here.

To support repository mirroring for heterogeneous clusters requires a more complex procedure than the one documented here.

The firewall allows all cluster nodes (the servers on which you want to install HDP) to access this server.
Ensure that the mirror server hasyum installed.
Add the yum-utils and createrepo packages on the mirror server.
```
yum install yum-utils createrepo
```

Install the Repos.

Temporarily reconfigure your firewall to allow Internet access from your mirror server host.

Execute the following command to download the appropriate Hortonworks yum client configuration file and save it in /etc/yum.repos.d/ directory on the mirror server host.

Table 3.5. Yum Client Configuration Commands

Cluster OS	HDP Repository Tarballs
Debian 7	`wget http://public-repo-1.hortonworks.com/HDP/debian7/2.x/updates/2.6.4.0/hdp.list -O /etc/apt/sources.list.d/hdp.list`
RHEL/CentOS/Oracle LINUX 6	`wget http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.4.0/hdp.repo -O /etc/yum.repos.d/hdp.repo`
RHEL/CentOS/Oracle LINUX 7	`wget http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/hdp.repo -O /etc/zypp/repos.d/hdp.repo`
SLES 11 SP3/SP4	`wget http://public-repo-1.hortonworks.com/HDP/suse11sp3/2.x/updates/2.6.4.0/hdp.repo -O /etc/zypp/repos.d/hdp.repo`
SLES 12	`wget http://public-repo-1.hortonworks.com/HDP/sles12/2.x/updates/2.6.4.0/hdp.repo -O /etc/zypp/repos.d/hdp.repo`
Ubuntu 12	`wget http://public-repo-1.hortonworks.com/HDP/ubuntu12/2.x/updates/2.6.4.0/hdp.list -O /etc/apt/sources.list.d/hdp.list`
Ubuntu 14	`wget http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.6.4.0/hdp.list -O /etc/apt/sources.list.d/hdp.list`
Ubuntu 16	`wget http://public-repo-1.hortonworks.com/HDP/ubuntu16/2.x/updates/2.6.4.0/hdp.list -O /etc/apt/sources.list.d/hdp.list`

Create an HTTP server.
- On the mirror server, install an HTTP server (such as Apache httpd using the instructions provided
- Activate this web server.
- Ensure that the firewall settings (if any) allow inbound HTTP access from your cluster nodes to your mirror server.
  Note
  If you are using EC2, make sure that SELinux is disabled.
- Optional - If your mirror server uses SLES, modify the default-server.conf file to enable the docs root folder listing.
```
sed -e s/Options None/Options Indexes MultiViews/ig /etc/apache2/default-server.conf /tmp/tempfile.tmp
mv /tmp/tempfile.tmp /etc/apache2/default-server.conf
```
On your mirror server, create a directory for your web server.
- For example, from a shell window, type:
  - For RHEL/CentOS/Oracle:
    mkdir –p /var/www/html/hdp/
  - For SLES:
    mkdir –p /srv/www/htdocs/rpms
  - For Ubuntu and Debian:
    mkdir –p /var/www/html/hdp/
- If you are using a symlink, enable the followsymlinks on your web server.
- Copy the contents of entire HDP repository for your desired OS from the remote yum server to your local mirror server.
  - Continuing the previous example, from a shell window, type:
    For RHEL/CentOS/Oracle/Ubuntu:
    cd/var/www/html/hdp
    For SLES:
    cd /srv/www/htdocs/rpms
    Then for all hosts, type:
    HDP Repository
    reposync -r HDP reposync -r HDP-2.6.4.0 reposync -r HDP-UTILS-1.1.0.21
    You should see both an HDP-2.6.4.0 directory and an HDP-UTILS-1.1.0.21 directory, each with several subdirectories.
- Generate appropriate metadata.
  This step defines each directory as a yum repository. From a shell window, type:
  - For RHEL/CentOS/Oracle:
    HDP Repository:
    createrepo /var/www/html/hdp/HDP-2.6.4.0 createrepo /var/www/html/hdp/HDP-UTILS-1.1.0.21
  - For SLES:
    HDP Repository:
    createrepo /srv/www/htdocs/rpms/hdp/HDP
  You should see a new folder called repodata inside both HDP directories.
- Verify the configuration.
  - The configuration is successful, if you can access the above directory through your web browser.
    To test this out, browse to the following location:
    HDP:http://$yourwebserver/hdp/HDP-2.6.4.0/
  - You should now see directory listing for all the HDP components.
- At this point, you can disable external Internet access for the mirror server, so that the mirror server is again entirely within your data center firewall.
- Depending on your cluster OS, configure the yum clients on all the nodes in your cluster
  - Edit the repo files, changing the value of the baseurl property to the local mirror URL.
    Edit the /etc/yum.repos.d/hdp.repo file, changing the value of the baseurl property to point to your local repositories based on your cluster OS.
    [HDP-2.x] name=Hortonworks Data Platform Version - HDP-2.x baseurl=http:// $yourwebserver /HDP/ $os /2.x/GA gpgcheck=1 gpgkey=http://public-repo-1.hortonworks.com/HDP/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.21] name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.21 baseurl=http:// $yourwebserver /HDP-UTILS-1.1.0.21/repos/ $os gpgcheck=1 gpgkey=http://public-repo-1.hortonworks.com/HDP/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-2.6.4.0] name=Hortonworks Data Platform HDP-2.6.4.0 baseurl=http:// $yourwebserver /HDP/ $os /2.x/updates/2.6.4.0 gpgcheck=1 gpgkey=http://public-repo-1.hortonworks.com/HDP/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
    where
    $yourwebserver is the FQDN of your local mirror server.
    $os can be centos6, centos7, suse11sp3, sles12, ubuntu12, ubuntu14, or ubuntu16. Use the following options table for $os parameter:
    
    Table 3.6. $OS Parameter Values
    Operating System Value
    Debian 7 debian7
    RHEL 6 centos6
    RHEL 7 centos7
    SLES 11 SP3/SP4 suse11sp3
    SLES 12 sles12
    Ubuntu 12 ubuntu12
    Ubuntu 14 ubuntu14
    Ubuntu 16 ubuntu16
  - Copy the yum/zypper client configuration file to all nodes in your cluster.
    RHEL/CentOS/Oracle Linux:
    Use scp or pdsh to copy the client yum configuration file to /etc/yum.repos.d/ directory on every node in the cluster.
    For SLES:
    On every node, invoke the following command:
    HDP Repository:
    zypper addrepo -r http://$yourwebserver/hdp/HDP/suse11/2.x/updates/2.6.4.0/hdp.repo
    For Ubuntu:
    On every node, invoke the following command:
    HDP Repository:
    sudo add-apt-repository deb http://$yourwebserver/hdp/HDP/ubuntu12/2.x/hdp.list
    Optional - Ambari Repository:
    sudo add-apt-repository deb http://$yourwebserver/hdp/ambari/ubuntu12/1.x/updates/1.7.0/ambari.list
    If using Ambari, verify the configuration by deploying Ambari server on one of the cluster nodes.
    yum install ambari-server
- If your cluster runs CentOS, Oracle, or RHEL and if you have multiple repositories configured in your environment, deploy the following plugin on all the nodes in your cluster.
  - Install the plugin.
    For RHEL and CentOs v5.x
    yum install yum-priorities
    For RHEL and CentOs v6.x
    yum install yum-plugin-priorities
  - Edit the /etc/yum/pluginconf.d/priorities.conf file to add the following:
    [main] enabled=1 gpgcheck=0

	Note
	If you are using EC2, make sure that SELinux is disabled.

Operating System	Value
Debian 7	debian7
RHEL 6	centos6
RHEL 7	centos7
SLES 11 SP3/SP4	suse11sp3
SLES 12	sles12
Ubuntu 12	ubuntu12
Ubuntu 14	ubuntu14
Ubuntu 16	ubuntu16

​Option II - Mirror server has temporary or continuous access to the Internet

Option II - Mirror server has temporary or continuous access to the Internet