Configure Ranger Admin HA with SSL

Set up the load-balancer and enable Ranger Admin.

  1. Use SSH to connect to the cluster node where you will set up the load balancer. In this procedure, we use the IP address load-balancer-host-fqdn.
  2. Use the following command to switch to the /usr/local directory:
    cd /usr/local
  3. Download the latest httpd file and its dependencies (apr and apr-util) from https://httpd.apache.org/download.cgi. For example:
    wget https://archive.apache.org/dist/httpd/httpd-2.4.43.tar.gz
    wget https://archive.apache.org/dist/apr/apr-1.7.0.tar.gz 
    wget https://archive.apache.org/dist/apr/apr-util-1.6.1.tar.gz
  4. Extract the contents of these files:
    tar -xvf httpd-2.4.43.tar.gz
    tar -xvf apr-1.7.0.tar.gz 
    tar -xvf apr-util-1.6.1.tar.gz
  5. Run the following commands to move apr and apr-util to the srclib directory under httpd:
    mv apr-1.7.0/ apr
    mv apr httpd-2.4.43/srclib/ 
    mv apr-util-1.6.1/ apr-util
    mv apr-util httpd-2.4.43/srclib/
  6. Install the required packages:
    yum groupinstall "Development Tools"
    yum install openssl-devel
    yum install pcre-devel
  7. Install PCRE (Perl-Compatible Regular Expressions Library):
    yum install pcre pcre-devel
  8. Install gcc (ANSI-C Compiler and Build System):
    yum install gcc
  9. Install expat:
    yum install expat-devel
  10. Run the following commands to configure the source tree:
    cd /usr/local/httpd-2.4.43
    ./configure --enable-so --enable-ssl --with-mpm=prefork --with-included-apr
  11. Run the following command to make the build:
    make
  12. Run the install:
    make install
  13. Run the following commands to start the Apache server:
    cd /usr/local/apache2/bin
    ./apachectl start
  14. To confirm that httpd is running, run the following command:
    curl localhost

    This should return:

    <html><body><h1>It works!</h1></body></html>
  15. Run the following commands to create a backup conf file.
    cd /usr/local/apache2/conf
    cp httpd.conf ~/httpd.conf.backup
  16. Use the following steps to generate the self-signed certificate:
    1. Switch to the directory that will contain the self-signed certificate:
      cd ~/
    2. Generate the private key:
      openssl genrsa -out server.key 2048
    3. Generate the CSR:
      openssl req -new -key server.key -out server.csr
    4. Generate the self-signed key:
      openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
    5. Generate the keystore in PEM format:
      openssl pkcs12 -export -passout pass:ranger -in server.crt -inkey server.key -out lbkeystore.p12 -name httpd.lb.server.alias
    6. Use the keytool to convert the PEM format keystore to JKS format:
      keytool -importkeystore -deststorepass ranger -destkeypass ranger -destkeystore httpd_lb_keystore.jks -srckeystore lbkeystore.p12 -srcstoretype PKCS12 -srcstorepass ranger -alias httpd.lb.server.alias
    7. Create a truststore of the load-balancer self-signed keystore:
      keytool -export -keystore httpd_lb_keystore.jks -alias httpd.lb.server.alias -file httpd-lb-trust.cer
  17. Copy the generated key and certificate into the /usr/local/apache2/conf/ directory.
    cp server.crt /usr/local/apache2/conf/
    cp server.key /usr/local/apache2/conf/
  18. Edit the httpd.conf file:
    vi /usr/local/apache2/conf/httpd.conf

    Make the following updates:

    • If you are not running the load-balancer on the default port 80, change the default listening port in line Listen 80 to match the port setting.

    • Un-comment the following module entries (remove the # symbol at the beginning of each line):

      LoadModule proxy_module modules/mod_proxy.so
      LoadModule proxy_http_module modules/mod_proxy_http.so
      LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
      LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
      LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
      LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
      LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
      LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
      LoadModule ssl_module modules/mod_ssl.so
    • Update the ServerAdmin email address, or comment out that line.

      #ServerAdmin you@example.com
    • At the end of the httpd.conf file, add the following line to read the custom configuration file:

      Include /usr/local/apache2/conf/ranger-lb-ssl.conf
  19. Create a custom conf file:
    vi /usr/local/apache2/conf/ranger-lb-ssl.conf

    Make the following updates:

    • Add the following lines, then change the <VirtualHost *:8443> port to match the default port you previously set in the httpd.conf file.

      <VirtualHost *:8443>
      
              SSLEngine On
              SSLProxyEngine On
              SSLCertificateFile /usr/local/apache2/conf/server.crt
              SSLCertificateKeyFile /usr/local/apache2/conf/server.key
      
              #SSLCACertificateFile /usr/local/apache2/conf/ranger_lb_crt.pem
              #SSLProxyCACertificateFile /usr/local/apache2/conf/ranger_lb_crt.pem
              SSLVerifyClient optional
              SSLOptions +ExportCertData
              SSLProxyVerify none
              SSLProxyCheckPeerCN off
              SSLProxyCheckPeerName off
              SSLProxyCheckPeerExpire off
              ProxyRequests off
              ProxyPreserveHost off
      
              Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
      
              <Proxy balancer://rangercluster>
                      BalancerMember http://ranger-host1-fqdn:6080 loadfactor=1 route=1
                      BalancerMember http://ranger-host2-fqdn:6080 loadfactor=1 route=2
      
                      Order Deny,Allow
                      Deny from none
                      Allow from all
      
                      ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off
              </Proxy>
      
              # balancer-manager
              # This tool is built into the mod_proxy_balancer
              # module and will allow you to do some simple
              # modifications to the balanced group via a gui
              # web interface.
              <Location /balancer-manager>
                      SetHandler balancer-manager
                      Order deny,allow
                      Allow from all
              </Location>
      
      RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
      RequestHeader set "X-Forwarded-SSL" expr=%{HTTPS}
      
             ProxyPass /balancer-manager !
             ProxyPass / balancer://rangercluster/
             ProxyPassReverse / balancer://rangercluster/
      
      </VirtualHost>
  20. Run the following commands to restart the httpd server:
    cd /usr/local/apache2/bin
    ./apachectl restart

    To confirm that httpd is running, run the following command:

    curl localhost

    This should return:

    <html><body><h1>It works!</h1></body></html>
  21. In Cloudera Manager, select Ranger, then select Actions > Add Role Instances.
  22. On the Add Role Instances page, click Select hosts.
  23. On the selected hosts page, the primary Ranger Admin host is selected by default. Select the backup Ranger host you specified previously (ranger-host2-fqdn) . A Ranger Admin (RA) icon appears in the Added Roles column for the selected backup host. Click OK to continue.
  24. The Add Role Instances page is redisplayed with the new backup host. Click Continue.
  25. Review the settings on the Review Changes page, then click Continue.
  26. Update the Ranger Load Balancer Address property (ranger.externalurl) with the load balancer host URL and port (load-balancer-host-fqdn:80), then click Save Changes.
  27. Run the following command to enable Usersync to communicate with Ranger via the load-balancer. This command copies the previously generated truststore file from the /tmp directory imports the certificate into the Usersync truststore.
    keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/ranger/usersync/conf/mytruststore.jks -storepass changeit
  28. Restart Ranger Usersync.
  29. Run the following command to enable the HDFS plug-in to communicate with Ranger via the load-balancer. This command copies the previously generated truststore file from the /tmp directory imports the certificate into the HDFS truststore.
    keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/hadoop/conf/ranger-plugin-truststore.jks -storepass changeit
  30. Restart HDFS.
  31. In the Ranger Admin UI, select Audit > Plugins. You should see an entry for your repo name with HTTP Response Code 200.
  32. Use SSH to connect to the KDC server host. Use the kadmin.local command to access the Kerberos CLI, then check the list of principals for each domain where Ranger Admin and the load-balancer are installed.
    kadmin.local
    kadmin.local: list_principals

    For example, if Ranger Admin is installed on <host1> and <host2>, and the load-balancer is installed on <host3>, the list returned should include the following entries:

    HTTP/ <host3>@EXAMPLE.COM
    HTTP/ <host2>@EXAMPLE.COM
    HTTP/ <host1>@EXAMPLE.COM

    If the HTTP principal for any of these hosts is not listed, use the following command to add the principal:

    kadmin.local: addprinc -randkey HTTP/<host3>@EXAMPLE.COM
  33. If Kerberos is configured on your cluster, complete the following steps to create a composite keytab.
    1. SSH into the Ranger Admin host, then create a keytabs directory.
      mkdir /etc/security/keytabs/
    2. Copy the ranger.keytab from the current running process.
      cp /var/run/cloudera-scm-agent/process/<current-ranger-process>/ranger.keytab /etc/security/keytabs/ranger.ha.keytab
    3. Run the following command to invoke kadmin.local.
      kadmin.local
    4. Run the following command to add the SPNEGO principal entry on the load balancer node.
      ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/load-balancer-host@EXAMPLE.COM
    5. Run the following command to add the SPNEGO principal entry on the node where the first Ranger Admin is installed.
      ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/ranger-admin-host1@EXAMPLE.COM
    6. Run the following command to add the SPNEGO principal entry on the node where the second Ranger Admin is installed.
      ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/ranger-admin-host2@EXAMPLE.COM
    7. Run the following command to exit kadmin.local.
      exit
    8. Run the following command to verify that the /etc/security/keytabs/ranger.ha.keytab file has entries for all of the required SPNEGO principals.
      klist -kt /etc/security/keytabs/ranger.ha.keytab
    9. On the backup (ranger-admin-host2) Ranger Admin node, run the following command to create a keytabs folder.
      mkdir /etc/security/keytabs/
    10. Copy the ranger.ha.keytab file from the primary Ranger Admin node (ranger-admin-host1) to the backup (ranger-admin-host2) Ranger Admin node.
      scp /etc/security/keytabs/ranger.ha.keytab root@ranger-host2-fqdn:/etc/security/keytabs/ranger.ha.keytab
    11. Run the following commands on all of the Ranger Admin nodes.
      chmod 440 /etc/security/keytabs/ranger.ha.keytab
      chown ranger:hadoop /etc/security/keytabs/ranger.ha.keytab
  34. Update the following ranger-admin-site.xml configuration setting using the Safety Valve.
    ranger.ha.spnego.kerberos.keytab=/etc/security/keytabs/ranger.ha.keytab
  35. Restart all cluster services that require a restart, then click Finish.
  36. Use a browser to check the load-balancer host URL (with port) as specified in the BalanceMember entries in the ranger-cluster.conf file. You should see the Ranger Admin page.