Using a custom Kerberos configuration path

How to use a custom Kerberos configuration path ( a path other than the default file path, /etc/krb5.conf) for an existing cluster.

From CDP 7.1.9 ( Cloudera Manager 7.11.3 ) onwards, you are able to set a custom path for the Kerberos Configuration file, krb5.conf. The valid paths are limited to /etc/hadoop/* or /etc/krb5.conf when Manage krb5.conf through Cloudera Manager setting is enabled in Cloudera Manager; otherwise, any path is acceptable.

The steps below show how to change the default kerberos path(/etc/krb5.conf) to a custom path(e.g. /etc/hadoop/[***custom_path***]/krb5.conf)

Begin with a running cluster with Kerberos path set to /etc/krb5.conf. Then, make sure the original krb5.conf under /etc/krb5.conf has been removed from all the hosts on the cluster before following the steps.

Define the Kerberos file path by following Option1 - if you want Cloudera Manager to manage the krb configuration, or else Option 2.
Option 1: Configure Manage krb5.conf by Cloudera Manager and set the new location of krb5.conf to /etc/hadoop/[***custom_path***]/krb5.conf.
1. Go to Cloudera Manager > Administration > Settings.
2. Enter /etc/hadoop/[***custom_path***]/krb5.conf for krb5.conf file path.
  note
  Ensure all the required Kerberos fields are populated before enabling Manage krb5.conf by Cloudera Manager.
  
  Ensure Active Directory Set Encryption Types is selected if Kerberos Encryption Type is set to a stronger encryption type like AES128-cts or AES256-cts.
3. Select the Manage krb5.conf by Cloudera Manager checkbox.
4. Click on Save Changes.
5. Restart Cloudera Manager Server.
6. Stop the cluster.
7. Stop Cloudera Management services.
8. Run the Kerberos Staleness Wizard to ensure the new krb5.conf file is deployed on all hosts. Go to Home > Configuration > Configuration Issues.Click on Cluster has stale Kerberos client configuration and complete the wizard for Deploy Kerberos Client Configuration.
Option 2: Where Manage krb5.conf by Cloudera Manager is unselected before changing the krb5 file path.
1. Go to Cloudera Manager > Administration > Settings.
2. Enter /etc/hadoop/[***custom_path***]/krb5.conf for krb5.conf file path.
3. Click on Save Changes.
4. Restart Cloudera Manager Server.
5. Stop the cluster.
6. Stop Cloudera Management services.

If using MIT KDC, fix the KDC services to point to the new file location. Skip this step if using AD.

# vim /etc/sysconfig/krb5kdc
                    
# cat /etc/sysconfig/krb5kdc
                    
  KRB5KDC_ARGS=
                    
  KRB5_CONFIG=/etc/hadoop/[***custom_path***]/krb5.conf

# vim /etc/sysconfig/kadmin
                        
# cat /etc/sysconfig/kadmin
                        
  KADMIND_ARGS=
                        
  KRB5_CONFIG=/etc/hadoop/[***custom_path***]/krb5.conf

# service krb5kdc restart && service kadmin restart
Redirecting to /bin/systemctl restart krb5kdc.service
                        
Redirecting to /bin/systemctl restart kadmin.service

Add the following to /etc/default/cloudera-scm-server.

CMF_JAVA_OPTS="${CMF_JAVA_OPTS} -Djava.security.krb5.conf=/etc/hadoop/[***custom_path***]/krb5.conf"

Restart the Cloudera Manager Server by running the following command on the Cloudera Manager Server host:
```
sudo systemctl restart cloudera-scm-server
```
Once the CM server has started , go to Cloudera Manager > Administration > Security > Kerberos Credentials, and click Setup KDC for this Cloudera Manager wizard to re-add the KDC.
Go to Cloudera Manager > Security > Kerberos Credentials, and click on Regenerate selected to regenerate credentials for all the roles. If 'Regenerate credentials' fails, then “sleep 30” should be added between the account deletion ( ldapdelete) and account modification with AES encryption (ldapmodify).
The AD Kerberos server is sometimes unable to absorb the ldapdelete and quickly reflect the changes when ldapmodify is called. Hence, adding a sleep gets the AD Kerberos server some time to sync within itself.

SSH to the Cloudera Manager server host and execute the below commands to edit the required script.
```
# vim /opt/cloudera/cm/bin/gen_credentials_ad.sh
# grep -A 5 -B 5 "sleep 30" /opt/cloudera/cm/bin/gen_credentials_ad.sh
          if [ $? -ne 0 ]; then
              echo "Deletion of the Active Directory account $PRINC failed."
              exit 1
          fi
        done <<< "$LOCAL_CNS" # read LOCAL_CNS and split at newlines
        sleep 30
        fi
                        
 # Add account in AD
 # servicePrincipalName is obtained from $PRINC by removing "@REALM" from the end.
 # password needs to be specified in unicode using "iconv"
                    
```
note

Cloudera does not support a scenario where an Active Directory creates account principals as non-leaf objects. This might cause Kerberos credentials regeneration to stop deleting the objects recursively.

As a workaround, add -r argument to the ldapdelete call in the /opt/cloudera/cm/bin/gen_credentials_ad.sh script. This does a recursive delete. But if the datanode specified is not a leaf, its children and all their children are deleted down the tree without any verification. So, use the -r argument with your own discretion.
To enable the HTTPFS role in HDFS, go to Cloudera Manager > HDFS > Configuration > HttpFS Environment Advanced Configuration Snippet (Safety Valve). Click on 'Add', and specify Key as HADOOP_OPTS, and Value as -Djava.security.krb5.conf=/etc/hadoop/[***custom_path***]/krb5.conf
Click on Save Changes.
Click the Stale Configuration icon to launch the Stale Configuration wizard and restart the necessary services.
Restart Cloudera Management Services. On the Home > Status tab, click the options menu to the right of Cloudera Management Service and select Start.
Start the cluster. On the Home > Status tab, click to the right of the cluster name and select Start.

Configure custom KRB5_CONFIG for Impala:

Before using impala-shell to connect to the Impala server with a valid Kerberos ticket for the first time, ensure the custom KRB5_CONFIG is accessible to impala-shell. To achieve this, export the custom KRB5_CONFIG to a file like .bashrc:

# .bashrc

# User specific aliases and functions
  alias rm='rm -i'
  alias cp='cp -i'
  alias mv='mv -i'

# Source global definitions
  if [ -f /etc/bashrc ]; then
     . /etc/bashrc
  fi

export KRB5_CONFIG=/etc/hadoop/[***custom_path***]/krb5.conf

# cd /run/cloudera-scm-agent/process/
# ls -lrt | grep -i impala
                        drwxr-x--x  7 impala	impala	620 Oct 25 07:18 133-ImpalaGracefulShutDownCmdWork
                        drwxr-x--x  7 impala	impala	640 Oct 25 07:18 118-impala-IMPALAD
                        drwxr-x--x  7 impala	impala	640 Oct 25 18:17 263-impala-IMPALAD
                        drwxr-x--x  7 impala	impala	620 Oct 25 18:17 394-impala-IMPALAD
# cd 394-impala-IMPALAD/
# kinit -kt impala.keytab impala/quasar-gvpuyw-1.quasar-gvpuyw.root.hwx.site@QE-INFRA-AD.CLOUDERA.COM
# impala-shell -i quasar-gvpuyw-5.quasar-gvpuyw.root.hwx.site:25003 -d default -k --ssl --ca_cert=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem
                        Starting Impala Shell using Kerberos authentication
                        Using service name 'impala'
                        SSL is enabled
                        No handlers could be found for logger "thrift.transport.sslcompat"
                        Opened TCP connection to quasar-gvpuyw-5.quasar-gvpuyw.root.hwx.site:25003
                        Connected to quasar-gvpuyw-5.quasar-gvpuyw.root.hwx.site:25003
                        Server version: impalad version 3.4.0-SNAPSHOT RELEASE (build bc0299cf11ff42368fcb39ddaf04f4fe32c1c445)
                        Query: use `default`
                        ***********************************************************************************
                        Welcome to the Impala shell.
                        (Impala Shell v3.4.0-SNAPSHOT (bc0299c) built on Sat Jan 28 00:17:17 UTC 2023)
                        
                        Press TAB twice to see a list of available commands.
                        ***********************************************************************************
                        Query: use `default`
                        [quasar-gvpuyw-5.quasar-gvpuyw.root.hwx.site:25003] default> select 1;
                        Query: select 1
                        Query submitted at: 2023-10-25 18:25:10 (Coordinator: https://quasar-gvpuyw-2.quasar-gvpuyw.root.hwx.site:25000)
                        Query progress can be monitored at: https://quasar-gvpuyw-2.quasar-gvpuyw.root.hwx.site:25000/query_plan?query_id=d44a181c32b9b1e9:0208a9f700000000
                        +---+
                        | 1 |
                        +---+
                        | 1 |
                        +---+
                        Fetched 1 row(s) in 0.29s
 default> Goodbye root