Configure Ranger Admin HA with a Load Balancer with TLS/SSL

How to enable Ranger Admin high availability with an external load balancer in an environment with TLS/SSL.

Configure an external load balancer to use with Ranger.
In Cloudera Manager, select Ranger, then select Actions > Add Role Instances.
On the Add Role Instances page, click Select hosts.
On the selected hosts page, the primary Ranger Admin host is selected by default. Select your configured backup Ranger host (ranger-host2-fqdn). A Ranger Admin (RA) icon appears in the Added Roles column for the selected backup host. Click OK to continue.
The Add Role Instances page is redisplayed with the new backup host. Click Continue.
Review the settings on the Review Changes page, then click Continue.
Update the Ranger Load Balancer Address property (ranger.externalurl) with the load balancer host URL and port, then click Save Changes.
If Kerberos is configured on your cluster, use SSH to connect to the KDC server host. Use the kadmin.local command to access the Kerberos CLI, then check the list of principals for each domain where Ranger Admin and the load-balancer are installed.
note
This step assumes you are using an MIT KDC (and kadmin.local). This step will be different if you are using AD or IPA.
```
kadmin.local
kadmin.local: list_principals
```
For example, if Ranger Admin is installed on <host1> and <host2>, and the load-balancer is installed on <host3>, the list returned should include the following entries:
```
HTTP/ <host3>@EXAMPLE.COM
HTTP/ <host2>@EXAMPLE.COM
HTTP/ <host1>@EXAMPLE.COM
```
If the HTTP principal for any of these hosts is not listed, use the following command to add the principal:
```
kadmin.local: addprinc -randkey HTTP/<host3>@EXAMPLE.COM
```
note

This step will need to be performed each time the Spnego keytab is regenerated.
If Kerberos is configured on your cluster, complete the following steps to create a composite keytab.

note
These steps assume you are using an MIT KDC (and kadmin.local). These steps will be different if you are using AD or IPA.
1. SSH into the Ranger Admin host, then create a keytabs directory.
```
mkdir /etc/security/keytabs/
```
2. Copy the ranger.keytab from the current running process.
```
cp /var/run/cloudera-scm-agent/process/<current-ranger-process>/ranger.keytab /etc/security/keytabs/ranger.ha.keytab
```
3. Run the following command to invoke kadmin.local.
```
kadmin.local
```
4. Run the following command to add the SPNEGO principal entry on the load balancer node.
```
ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/load-balancer-host@EXAMPLE.COM
```
  note
  
  As shown above, the domain portion of the URL must be in capital letters. You can use list_principals * to view a list of all of the principals.
5. Run the following command to add the SPNEGO principal entry on the node where the first Ranger Admin is installed.
```
ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/ranger-admin-host1@EXAMPLE.COM
```
6. Run the following command to add the SPNEGO principal entry on the node where the second Ranger Admin is installed.
```
ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/ranger-admin-host2@EXAMPLE.COM
```
7. Run the following command to exit kadmin.local.
```
exit
```
8. Run the following command to verify that the /etc/security/keytabs/ranger.ha.keytab file has entries for all of the required SPNEGO principals.
```
klist -kt /etc/security/keytabs/ranger.ha.keytab
```
9. On the backup (ranger-admin-host2) Ranger Admin node, run the following command to create a keytabs folder.
```
mkdir /etc/security/keytabs/
```
10. Copy the ranger.ha.keytab file from the primary Ranger Admin node (ranger-admin-host1) to the backup (ranger-admin-host2) Ranger Admin node.
```
scp /etc/security/keytabs/ranger.ha.keytab root@ranger-host2-fqdn:/etc/security/keytabs/ranger.ha.keytab
```
11. Run the following commands on all of the Ranger Admin nodes.
```
chmod 440 /etc/security/keytabs/ranger.ha.keytab
chown ranger:hadoop /etc/security/keytabs/ranger.ha.keytab
```

Update the following ranger-admin-site.xml configuration settings using the Safety Valve.

ranger.ha.spnego.kerberos.keytab=/etc/security/keytabs/ranger.ha.keytab
ranger.spnego.kerberos.principal=*

Restart all cluster services that require a restart, then click Finish.
Use a browser to check the load-balancer host URL (with port). You should see the Ranger Admin page.

HTTPD Load Balancer Configuration Example

The following steps are an example of how to configure an HTTPD load balancer.

Use SSH to connect to the cluster node where you will set up the load balancer. In this procedure, we use the IP address load-balancer-host-fqdn.
Use the following command to switch to the /usr/local directory:
```
cd /usr/local
```

Download the latest httpd file and its dependencies (apr and apr-util) from https://httpd.apache.org/download.cgi. For example:

wget https://archive.apache.org/dist/httpd/httpd-2.4.43.tar.gz
wget https://archive.apache.org/dist/apr/apr-1.7.0.tar.gz 
wget https://archive.apache.org/dist/apr/apr-util-1.6.1.tar.gz

Extract the contents of these files:

tar -xvf httpd-2.4.43.tar.gz
tar -xvf apr-1.7.0.tar.gz 
tar -xvf apr-util-1.6.1.tar.gz

Run the following commands to move apr and apr-util to the srclib directory under httpd:

mv apr-1.7.0/ apr
mv apr httpd-2.4.43/srclib/ 
mv apr-util-1.6.1/ apr-util
mv apr-util httpd-2.4.43/srclib/

Install the required packages:

yum groupinstall "Development Tools"
yum install openssl-devel
yum install pcre-devel

Install PCRE (Perl-Compatible Regular Expressions Library):
```
yum install pcre pcre-devel
```
note

Here we are using yum install, but you can also download the latest bits from http://www.pcre.org/
Install gcc (ANSI-C Compiler and Build System):
```
yum install gcc
```
Install expat:
```
yum install expat-devel
```

Run the following commands to configure the source tree:

cd /usr/local/httpd-2.4.43
./configure --enable-so --enable-ssl --with-mpm=prefork --with-included-apr

Run the following command to make the build:
```
make
```
Run the install:
```
make install
```
Run the following commands to start the Apache server:
```
cd /usr/local/apache2/bin
./apachectl start
```
To confirm that httpd is running, run the following command:
```
curl localhost
```
This should return:
```
<html><body><h1>It works!</h1></body></html>
```

Run the following commands to create a backup conf file.

cd /usr/local/apache2/conf
cp httpd.conf ~/httpd.conf.backup

Use the following steps to generate the self-signed certificate:

Switch to the directory that will contain the self-signed certificate:
```
cd ~/
```
Generate the private key:
```
openssl genrsa -out server.key 2048
```

Generate the CSR:

openssl req -new -key server.key -out server.csr

Generate the self-signed key:

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Generate the keystore in PEM format:

openssl pkcs12 -export -passout pass:ranger -in server.crt -inkey server.key -out lbkeystore.p12 -name httpd.lb.server.alias

Use the keytool to convert the PEM format keystore to JKS format:

keytool -importkeystore -deststorepass ranger -destkeypass ranger -destkeystore httpd_lb_keystore.jks -srckeystore lbkeystore.p12 -srcstoretype PKCS12 -srcstorepass ranger -alias httpd.lb.server.alias

Create a truststore of the load-balancer self-signed keystore:

keytool -export -keystore httpd_lb_keystore.jks -alias httpd.lb.server.alias -file httpd-lb-trust.cer

Copy the generated key and certificate into the /usr/local/apache2/conf/ directory.
```
cp server.crt /usr/local/apache2/conf/
cp server.key /usr/local/apache2/conf/
```
Edit the httpd.conf file:
```
vi /usr/local/apache2/conf/httpd.conf
```
Make the following updates:
- If you are not running the load-balancer on the default port 80, change the default listening port in line Listen 80 to match the port setting.
- Un-comment the following module entries (remove the # symbol at the beginning of each line):
```
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
LoadModule ssl_module modules/mod_ssl.so
```
  note
  
  If LoadModule ssl_module modules/mod_ssl.so is not available in the httpd.conf file, check to make sure you performed all of the previous installation steps. The load balancer will not work properly without the SSL module.
- Update the ServerAdmin email address, or comment out that line.
```
#ServerAdmin you@example.com
```
- At the end of the httpd.conf file, add the following line to read the custom configuration file:
```
Include /usr/local/apache2/conf/ranger-lb-ssl.conf
```

Create a custom conf file:

vi /usr/local/apache2/conf/ranger-lb-ssl.conf

Make the following updates:

Add the following lines, then change the <VirtualHost *:8443> port to match the default port you previously set in the httpd.conf file.

<VirtualHost *:8443>

        SSLEngine On
        SSLProxyEngine On
        SSLCertificateFile /usr/local/apache2/conf/server.crt
        SSLCertificateKeyFile /usr/local/apache2/conf/server.key

        #SSLCACertificateFile /usr/local/apache2/conf/ranger_lb_crt.pem
        #SSLProxyCACertificateFile /usr/local/apache2/conf/ranger_lb_crt.pem
        SSLVerifyClient optional
        SSLOptions +ExportCertData
        SSLProxyVerify none
        SSLProxyCheckPeerCN off
        SSLProxyCheckPeerName off
        SSLProxyCheckPeerExpire off
        ProxyRequests off
        ProxyPreserveHost off

        Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

        <Proxy balancer://rangercluster>
                BalancerMember http://ranger-host1-fqdn:6080 loadfactor=1 route=1
                BalancerMember http://ranger-host2-fqdn:6080 loadfactor=1 route=2

                Order Deny,Allow
                Deny from none
                Allow from all

                ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off
        </Proxy>

        # balancer-manager
        # This tool is built into the mod_proxy_balancer
        # module and will allow you to do some simple
        # modifications to the balanced group via a gui
        # web interface.
        <Location /balancer-manager>
                SetHandler balancer-manager
                Order deny,allow
                Allow from all
        </Location>

RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
RequestHeader set "X-Forwarded-SSL" expr=%{HTTPS}

       ProxyPass /balancer-manager !
       ProxyPass / balancer://rangercluster/
       ProxyPassReverse / balancer://rangercluster/

</VirtualHost>

Run the following commands to restart the httpd server:
```
cd /usr/local/apache2/bin
./apachectl restart
```
To confirm that httpd is running, run the following command:
```
curl localhost
```
This should return:
```
<html><body><h1>It works!</h1></body></html>
```
In Cloudera Manager, select Ranger, then select Actions > Add Role Instances.
On the Add Role Instances page, click Select hosts.
On the selected hosts page, the primary Ranger Admin host is selected by default. Select your configured backup Ranger host (ranger-host2-fqdn). A Ranger Admin (RA) icon appears in the Added Roles column for the selected backup host. Click OK to continue.
The Add Role Instances page is redisplayed with the new backup host. Click Continue.
Review the settings on the Review Changes page, then click Continue.
Update the Ranger Load Balancer Address property (ranger.externalurl) with the load balancer host URL and port, then click Save Changes.
Run the following command to enable Usersync to communicate with Ranger via the load-balancer. This command copies the previously generated truststore file from the /tmp directory imports the certificate into the Usersync truststore.
```
keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/ranger/usersync/conf/mytruststore.jks -storepass changeit
```
Restart Ranger Usersync.
Run the following command to enable the HDFS plug-in to communicate with Ranger via the load-balancer. This command copies the previously generated truststore file from the /tmp directory imports the certificate into the HDFS truststore.
```
keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/hadoop/conf/ranger-plugin-truststore.jks -storepass changeit
```
Restart HDFS.
In the Ranger Admin UI, select Audit > Plugins. You should see an entry for your repo name with HTTP Response Code 200.
Use SSH to connect to the KDC server host. Use the kadmin.local command to access the Kerberos CLI, then check the list of principals for each domain where Ranger Admin and the load-balancer are installed.
```
kadmin.local
kadmin.local: list_principals
```
For example, if Ranger Admin is installed on <host1> and <host2>, and the load-balancer is installed on <host3>, the list returned should include the following entries:
```
HTTP/ <host3>@EXAMPLE.COM
HTTP/ <host2>@EXAMPLE.COM
HTTP/ <host1>@EXAMPLE.COM
```
If the HTTP principal for any of these hosts is not listed, use the following command to add the principal:
```
kadmin.local: addprinc -randkey HTTP/<host3>@EXAMPLE.COM
```
note

This step will need to be performed each time the Spnego keytab is regenerated.
If Kerberos is configured on your cluster, complete the following steps to create a composite keytab.
1. SSH into the Ranger Admin host, then create a keytabs directory.
```
mkdir /etc/security/keytabs/
```
2. Copy the ranger.keytab from the current running process.
```
cp /var/run/cloudera-scm-agent/process/<current-ranger-process>/ranger.keytab /etc/security/keytabs/ranger.ha.keytab
```
3. Run the following command to invoke kadmin.local.
```
kadmin.local
```
4. Run the following command to add the SPNEGO principal entry on the load balancer node.
```
ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/load-balancer-host@EXAMPLE.COM
```
  note
  
  As shown above, the domain portion of the URL must be in capital letters. You can use list_principals * to view a list of all of the principals.
5. Run the following command to add the SPNEGO principal entry on the node where the first Ranger Admin is installed.
```
ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/ranger-admin-host1@EXAMPLE.COM
```
6. Run the following command to add the SPNEGO principal entry on the node where the second Ranger Admin is installed.
```
ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/ranger-admin-host2@EXAMPLE.COM
```
7. Run the following command to exit kadmin.local.
```
exit
```
8. Run the following command to verify that the /etc/security/keytabs/ranger.ha.keytab file has entries for all of the required SPNEGO principals.
```
klist -kt /etc/security/keytabs/ranger.ha.keytab
```
9. On the backup (ranger-admin-host2) Ranger Admin node, run the following command to create a keytabs folder.
```
mkdir /etc/security/keytabs/
```
10. Copy the ranger.ha.keytab file from the primary Ranger Admin node (ranger-admin-host1) to the backup (ranger-admin-host2) Ranger Admin node.
```
scp /etc/security/keytabs/ranger.ha.keytab root@ranger-host2-fqdn:/etc/security/keytabs/ranger.ha.keytab
```
11. Run the following commands on all of the Ranger Admin nodes.
```
chmod 440 /etc/security/keytabs/ranger.ha.keytab
chown ranger:hadoop /etc/security/keytabs/ranger.ha.keytab
```

Update the following ranger-admin-site.xml configuration settings using the Safety Valve.

ranger.ha.spnego.kerberos.keytab=/etc/security/keytabs/ranger.ha.keytab
ranger.spnego.kerberos.principal=*

Restart all cluster services that require a restart, then click Finish.
Use a browser to check the load-balancer host URL (with port) as specified in the BalanceMember entries in the ranger-cluster.conf file. You should see the Ranger Admin page.