Configuring Ranger Admin HA With SSL
Set up a load-balancer with SSL and enable Ranger Admin HA on an Ambari-managed cluster.
- Use SSH to connect to the cluster node where you will set up the load-balancer. In this procedure, we use the IP address 172.22.71.37.
- Use the following command to switch to the
/usr/local
directory:cd /usr/local
- Download the
httpd
file and its dependencies (apr
andapr-util
):wget https://archive.apache.org/dist/httpd/httpd-2.4.16.tar.gz wget https://archive.apache.org/dist/apr/apr-1.5.2.tar.gz wget https://archive.apache.org/dist/apr/apr-util-1.5.4.tar.gz
- Extract the contents of these files:
tar -xvf httpd-2.4.16.tar.gz tar -xvf apr-1.5.2.tar.gz tar -xvf apr-util-1.5.4.tar.gz
- Run the following commands to move
apr
andapr-util
to thesrclib
directory underhttpd
:mv apr-1.5.2/ apr mv apr httpd-2.4.16/srclib/ mv apr-util-1.5.4/ apr-util mv apr-util httpd-2.4.16/srclib/
- Install the required packages:
yum groupinstall "Development Tools" yum install openssl-devel yum install pcre-devel
- Run the following commands to configure the source tree:
cd /usr/local/httpd-2.4.16 ./configure --enable-so --enable-ssl --with-mpm=prefork --with-included-apr
- Run the following command to make the build:
make
- Run the install:
make install
- Run the following commands to confirm the preceding configuration steps:
cd /usr/local/apache2/bin ./apachectl start curl localhost
This should return:
<html><body><h1>It works!</h1></body></html>
- Run the following commands to create a backup
conf
file.cd /usr/local/apache2/conf cp httpd.conf ~/httpd.conf.backup
- Edit the
httpd.conf
file:vi /usr/local/apache2/conf/httpd.conf
Make the following updates:
-
If you are not running the load-balancer on the default port 80, change the default listening port in line
Listen 80
to match the port setting. -
Un-comment the following module entries (remove the
#
symbol at the beginning of each line):LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule slotmem_shm_module modules/mod_slotmem_shm.so LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so LoadModule ssl_module modules/mod_ssl.so
NoteIf
LoadModule ssl_module modules/mod_ssl.so
is not available in thehttpd.conf
file, check to make sure that you performed all of the previous installation steps. The load balancer will not work properly without the SSL module. -
Update the ServerAdmin email address, or comment out that line.
#ServerAdmin you@example.com
-
- Run the following command to restart the
httpd
server:/usr/local/apache2/bin/apachectl restart
You should now be able to use Curl or a browser to access the load-balancer server IP address (with the port configured in the
httpd.conf
file) using the HTTPS protocol. - Enable Ranger Admin HA using Ambari. On the Ambari dashboard of the
first Ranger host, select Services > Ranger, then
select Service Actions > Enable Ranger Admin HA to
launch the Enable Ranger Admin HA Wizard.
- On the Get Started page, enter the load-balancer URL and port number (in this example, 172.22.71.37:88), then click Next.
- On the Select Hosts page, confirm the host assignments, then click Next.
- Check the settings on the Review page, then click Next.
- Click Complete on the Install, Start, and Test page to complete the installation.
- When the installation is complete, the Ranger Admin instances are listed on the
Ranger Summary page. Select Actions > Restart All
Required to restart all services that require a restart.
Note
For Oracle, clear the
Setup DB and DB user
check box under "Advanced ranger-env" to avoid DB setup. - To test the load-balancer and Ranger HA configuration, select Ranger > Service Actions > Stop on one of the Ranger hosts.
- Use a browser to check the load-balancer host URL (with port). You should see the Ranger Admin page.
- Use the following steps to generate the self-signed certificate:
- Switch to the directory that will contain the self-signed certificate:
cd /tmp
- Generate the private key:
openssl genrsa -out server.key 2048
- Generate the CSR:
openssl req -new -key server.key -out server.csr
- Generate the self-signed key:
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
- Generate the keystore in PEM format:
openssl pkcs12 -export -passout pass:ranger -in server.crt -inkey server.key -out lbkeystore.p12 -name httpd.lb.server.alias
- Use the
keytool
to convert the PEM format keystore to JKS format:keytool -importkeystore -deststorepass ranger -destkeypass ranger -destkeystore httpd_lb_keystore.jks -srckeystore lbkeystore.p12 -srcstoretype PKCS12 -srcstorepass ranger -alias httpd.lb.server.alias
- Create a truststore of the load-balancer self-signed keystore:
keytool -export -keystore httpd_lb_keystore.jks -alias httpd.lb.server.alias -file httpd-lb-trust.cer
- Switch to the directory that will contain the self-signed certificate:
- Copy the generated key and certificate into the
/usr/local/apache2/conf/
directory.cp server.crt /usr/local/apache2/conf/ cp server.key /usr/local/apache2/conf/
- Add the following entry at the end of the
/usr/local/apache2/conf/httpd.conf
file to read the custom configuration file:Include /usr/local/apache2/conf/ranger-lb-ssl.conf
- Create a custom
conf
file for the load-balancer SSL configuration:vi /usr/local/apache2/conf/ranger-lb-ssl.conf
Make the following updates:
Add the following lines, then change the
<VirtualHost *:8443>
port to match the default port you set previously in thehttpd.conf
file.<VirtualHost *:8443> SSLEngine On SSLProxyEngine On SSLCertificateFile /usr/local/apache2/conf/server.crt SSLCertificateKeyFile /usr/local/apache2/conf/server.key #SSLCACertificateFile /usr/local/apache2/conf/ranger_lb_crt.pem #SSLProxyCACertificateFile /usr/local/apache2/conf/ranger_lb_crt.pem SSLVerifyClient optional SSLOptions +ExportCertData SSLProxyVerify none SSLProxyCheckPeerCN off SSLProxyCheckPeerName off SSLProxyCheckPeerExpire off ProxyRequests off ProxyPreserveHost off Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED <Proxy balancer://rangercluster> BalancerMember http://172.22.71.39:6080 loadfactor=1 route=1 BalancerMember http://172.22.71.38:6080 loadfactor=1 route=2 Order Deny,Allow Deny from none Allow from all ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off </Proxy> # balancer-manager # This tool is built into the mod_proxy_balancer # module and will allow you to do some simple # modifications to the balanced group via a gui # web interface. <Location /balancer-manager> SetHandler balancer-manager Order deny,allow Allow from all </Location> ProxyPass /balancer-manager ! ProxyPass / balancer://rangercluster/ ProxyPassReverse / balancer://rangercluster/ </VirtualHost>
NoteThe URLs listed in the
BalancerMember
entries are the IP addresses of the Ranger Admin hosts. In this example, the Ranger Admin host adresses are:http://172.22.71.38:6080 http://172.22.71.39:6080
- Run the following command to restart the
httpd
server:/usr/local/apache2/bin/apachectl restart
If you use a browser to check the load-balancer host (with port), you should see the Ranger Admin page.
- Run the following command to enable Usersync to communicate with Ranger via the
load-balancer. This command copies the previously generated truststore file from the
/tmp
directory imports the certificate into the Usersync truststore.keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/ranger/usersync/conf/mytruststore.jks -storepass changeit
- Restart Ranger Usersync.
- Run the following command to enable the HDFS plug-in to communicate with Ranger
via the load-balancer. This command copies the previously generated truststore file
from the
/tmp
directory imports the certificate into the HDFS truststore.keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/hadoop/conf/ranger-plugin-truststore.jks -storepass changeit
- Restart HDFS.
- In the Ranger Admin UI, select Audit > Plugins. You should see an entry for your repo name with HTTP Response Code 200.
- Use SSH to connect to the KDC server host. Use the
kadmin.local
command to access the Kerberos CLI, then check the list of principals for each domain where Ranger Admin and the load-balancer are installed.kadmin.local kadmin.local: list_principals
For example, if Ranger Admin is installed on <host1> and <host2>, and the load-balancer is installed on <host3>, the list returned should include the following entries:
HTTP/ <host3>@EXAMPLE.COM HTTP/ <host2>@EXAMPLE.COM HTTP/ <host1>@EXAMPLE.COM
If the HTTP principal for any of these hosts is not listed, use the following command to add the principal:
kadmin.local: addprinc -randkey HTTP/<host3>@EXAMPLE.COM
NoteThis step will need to be performed each time the Spnego keytab is regenerated.
- Use the following
kadmin.local
commands to add the HTTP Principal of each of the Ranger Admin and load-balancer nodes to the Spnego keytab file:kadmin.local: ktadd -norandkey -kt /etc/security/keytabs/spnego.service.keytab HTTP/ <host3>@EXAMPLE.COM kadmin.local: ktadd -norandkey -kt /etc/security/keytabs/spnego.service.keytab HTTP/ <host2>@EXAMPLE.COM kadmin.local: ktadd -norandkey -kt /etc/security/keytabs/spnego.service.keytab HTTP/ <host1>@EXAMPLE.COM
Use the
exit
command to exitktadmin.local
. - Run the following command to check the Spnego keytab file:
klist -kt /etc/security/keytabs/spnego.service.keytab
The output should include the principals of all of the nodes on which Ranger Admin and the load-balancer are installed. For example:
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 07/22/16 06:27:31 HTTP/ <host3>@EXAMPLE.COM 1 07/22/16 06:27:31 HTTP/ <host3>@EXAMPLE.COM 1 07/22/16 06:27:31 HTTP/ <host3>@EXAMPLE.COM 1 07/22/16 06:27:31 HTTP/ <host3>@EXAMPLE.COM 1 07/22/16 06:27:31 HTTP/ <host3>@EXAMPLE.COM 1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM 1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM 1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM 1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM 1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM 1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM 1 07/22/16 08:37:35 HTTP/ <host1>@EXAMPLE.COM 1 07/22/16 08:37:36 HTTP/ <host1>@EXAMPLE.COM 1 07/22/16 08:37:36 HTTP/ <host1>@EXAMPLE.COM 1 07/22/16 08:37:36 HTTP/ <host1>@EXAMPLE.COM 1 07/22/16 08:37:36 HTTP/ <host1>@EXAMPLE.COM 1 07/22/16 08:37:36 HTTP/ <host1>@EXAMPLE.COM
- Use
scp
to copy the Spnego keytab file to every node in the cluster on which Ranger Admin and the load-balancer are installed. Verify that the/etc/security/keytabs/spnego.service.keytab
file is present on all Ranger Admin and load-balancer hosts. -
Configure the Ranger HA Keytab file.
When setting up Ranger in HA where Kerberos is enabled, the Ambari-managed SPNEGO keytab file is altered. Since Ambari expects the file to contain certain data, the file is now overwritten with the Ambari-cached data during some Kerberos operations. This breaks Ranger in HA.
In HDP 3.0+, you must duplicate the SPNEGO keytab file for Ranger HA, make various changes, and add the custom property "ranger.ha.spnego.kerberos.keytab". This enables Ambari to use the relevant SPNEGO properties differently depending on whether the Ambari server or the Ambari agent is acting.
- Login to the load balancer node.
-
Enter:
cp /etc/security/keytabs/spnego.service.keytab /etc/security/keytabs/ranger.ha.keytab
. -
Run
kadmin.local
. -
Add the SPNEGO principal entry of the node where the first ranger_admin is
installed:
ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/as-amb-21-1.openstacklocal@EXAMPLE.COM
. -
Add the SPNEGO principal entry of the node where the second ranger_admin is
installed:
ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/as-amb-21-1.openstacklocal@EXAMPLE.COM
. -
Verify /etc/security/keytabs/ranger.ha.keytab contains
an entry of all the required SPNEGO principals:
klist -kt /etc/security/keytabs/ranger.ha.keytab
. -
Copy the Ranger keytab file to other nodes where ranger_admin is installed:
scp /etc/security/keytabs/ranger.ha.keytab
. -
Update permission:
chmod 440 /etc/security/keytabs/ranger.ha.keytab
. -
Update ownership:
chown root:hadoop /etc/security/keytabs/ranger.ha.keytab
. -
Add config from Ambari > Ranger > Configs > Advanced > Custom ranger-admin-site:
ranger.ha.spnego.kerberos.keytab=/etc/security/keytabs/ranger.ha.keytab
. - Restart the Ranger service.