How to use a custom Kerberos configuration path ( a path other than the default file path, /etc/krb5.conf) for an existing cluster.
From CDP 7.1.9 ( Cloudera Manager 7.11.3 ) onwards, you are able to set a custom path
for the Kerberos Configuration file, krb5.conf. The valid paths
are limited to /etc/hadoop/* or /etc/krb5.conf
when Manage krb5.conf through Cloudera Manager setting is
enabled in Cloudera Manager; otherwise, any path is acceptable.
The steps below show how to change the default kerberos
path(/etc/krb5.conf) to a custom path(e.g.
/etc/hadoop/[***custom_path***]/krb5.conf)
Begin with a running cluster with Kerberos path set to
/etc/krb5.conf. Then, make sure the original krb5.conf under
/etc/krb5.conf has been removed from all the hosts on the cluster
before following the steps.
Define the Kerberos file path by following Option1 - if you want Cloudera
Manager to manage the krb configuration, or else Option 2.
Option 1: Configure Manage krb5.conf by Cloudera
Manager and set the new location of krb5.conf to
/etc/hadoop/[***custom_path***]/krb5.conf.
Go to Cloudera Manager > Administration > Settings.
Enter /etc/hadoop/[***custom_path***]/krb5.conf for
krb5.conf file path.
Select the Manage krb5.conf by Cloudera Manager
checkbox.
Click on Save Changes.
Restart Cloudera Manager Server.
Stop the cluster.
Stop Cloudera Management services.
Run the Kerberos Staleness Wizard to ensure the new krb5.conf file is
deployed on all hosts. Go to Home > Configuration > Configuration Issues.Click on Cluster has stale Kerberos client
configuration and complete the wizard for
Deploy Kerberos Client Configuration.
Option 2: Where Manage krb5.conf by Cloudera
Manager is unselected before changing the krb5 file path.
Go to Cloudera Manager > Administration > Settings.
Enter /etc/hadoop/[***custom_path***]/krb5.conf for
krb5.conf file path.
Click on Save Changes.
Restart Cloudera Manager Server.
Stop the cluster.
Stop Cloudera Management services.
If using MIT KDC, fix the KDC services to point to the new file location. Skip
this step if using AD.
# vim /etc/sysconfig/krb5kdc
# cat /etc/sysconfig/krb5kdc
KRB5KDC_ARGS=
KRB5_CONFIG=/etc/hadoop/[***custom_path***]/krb5.conf
# vim /etc/sysconfig/kadmin
# cat /etc/sysconfig/kadmin
KADMIND_ARGS=
KRB5_CONFIG=/etc/hadoop/[***custom_path***]/krb5.conf
# service krb5kdc restart && service kadmin restart
Redirecting to /bin/systemctl restart krb5kdc.service
Redirecting to /bin/systemctl restart kadmin.service
Add the following to /etc/default/cloudera-scm-server.
Restart the Cloudera Manager Server by running the following command on the Cloudera Manager Server host:
sudo systemctl restart cloudera-scm-server
Once the CM server has started , go to Cloudera Manager > Administration > Security > Kerberos Credentials, and click Setup KDC for this Cloudera
Manager wizard to re-add the KDC.
Go to Cloudera Manager > Security > Kerberos Credentials, and click on Regenerate selected
to regenerate credentials for all the roles.
If 'Regenerate credentials' fails, then “sleep 30” should be
added between the account deletion ( ldapdelete) and account modification with
AES encryption (ldapmodify).
The AD Kerberos server is sometimes unable to absorb the ldapdelete and quickly reflect the changes when ldapmodify is called. Hence, adding a sleep gets the AD Kerberos server some time to sync within itself.
SSH to the Cloudera Manager server host and execute the below commands to edit the required script.
# vim /opt/cloudera/cm/bin/gen_credentials_ad.sh
# grep -A 5 -B 5 "sleep 30" /opt/cloudera/cm/bin/gen_credentials_ad.sh
if [ $? -ne 0 ]; then
echo "Deletion of the Active Directory account $PRINC failed."
exit 1
fi
done <<< "$LOCAL_CNS" # read LOCAL_CNS and split at newlines
sleep 30
fi
# Add account in AD
# servicePrincipalName is obtained from $PRINC by removing "@REALM" from the end.
# password needs to be specified in unicode using "iconv"
To enable the HTTPFS role in HDFS, go to Cloudera Manager > HDFS > Configuration > HttpFS Environment Advanced Configuration Snippet (Safety
Valve). Click on 'Add', and specify Key as
HADOOP_OPTS, and Value as
-Djava.security.krb5.conf=/etc/hadoop/[***custom_path***]/krb5.conf
Click on Save Changes.
Click the Stale Configuration icon to launch the Stale Configuration wizard and restart the necessary services.
Restart Cloudera Management Services. On the
Home > Status
tab, click the options menu to the right of Cloudera Management
Service and select Start.
Start the cluster. On the
Home > Status
tab, click to the right of the cluster name and
select Start.
Configure custom KRB5_CONFIG for Impala:
Before using impala-shell to connect to the Impala server with a valid Kerberos ticket for the first time, ensure the custom KRB5_CONFIG is accessible to impala-shell. To achieve this, export the custom KRB5_CONFIG to a file like .bashrc:
# .bashrc
# User specific aliases and functions
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# cd /run/cloudera-scm-agent/process/
# ls -lrt | grep -i impala
drwxr-x--x 7 impala impala 620 Oct 25 07:18 133-ImpalaGracefulShutDownCmdWork
drwxr-x--x 7 impala impala 640 Oct 25 07:18 118-impala-IMPALAD
drwxr-x--x 7 impala impala 640 Oct 25 18:17 263-impala-IMPALAD
drwxr-x--x 7 impala impala 620 Oct 25 18:17 394-impala-IMPALAD
# cd 394-impala-IMPALAD/
# kinit -kt impala.keytab impala/quasar-gvpuyw-1.quasar-gvpuyw.root.hwx.site@QE-INFRA-AD.CLOUDERA.COM
# impala-shell -i quasar-gvpuyw-5.quasar-gvpuyw.root.hwx.site:25003 -d default -k --ssl --ca_cert=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem
Starting Impala Shell using Kerberos authentication
Using service name 'impala'
SSL is enabled
No handlers could be found for logger "thrift.transport.sslcompat"
Opened TCP connection to quasar-gvpuyw-5.quasar-gvpuyw.root.hwx.site:25003
Connected to quasar-gvpuyw-5.quasar-gvpuyw.root.hwx.site:25003
Server version: impalad version 3.4.0-SNAPSHOT RELEASE (build bc0299cf11ff42368fcb39ddaf04f4fe32c1c445)
Query: use `default`
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v3.4.0-SNAPSHOT (bc0299c) built on Sat Jan 28 00:17:17 UTC 2023)
Press TAB twice to see a list of available commands.
***********************************************************************************
Query: use `default`
[quasar-gvpuyw-5.quasar-gvpuyw.root.hwx.site:25003] default> select 1;
Query: select 1
Query submitted at: 2023-10-25 18:25:10 (Coordinator: https://quasar-gvpuyw-2.quasar-gvpuyw.root.hwx.site:25000)
Query progress can be monitored at: https://quasar-gvpuyw-2.quasar-gvpuyw.root.hwx.site:25000/query_plan?query_id=d44a181c32b9b1e9:0208a9f700000000
+---+
| 1 |
+---+
| 1 |
+---+
Fetched 1 row(s) in 0.29s
default> Goodbye root
to the right of the cluster name and
select Start.
