Virtual Warehouse Fails to Start

This topic provides steps to troubleshoot issues where the Virtual Warehouse fails to start due to outdated Kerberos settings, resulting in HiveServer2 POD launch failures.

Condition

The Virtual Warehouse (VW) fails to start, and the HiveServer2 POD fails to launch. The following trace is present in the HiveServer2 logs:

+ DOWNLOAD_PATH=/aux-jars/
+ '[' true == true ']'
+ SERVICE_KEYTAB=/etc/security/keytabs/hive.service.keytab
+ SERVICE_PRINCIPAL=hive/dwx-env-feddatalakedev-env.cdp.local@US-POCLAB.DELLPOC.COM
+ kinit -V -k -t /etc/security/keytabs/hive.service.keytab hive/dwx-env-feddatalakedev-env.cdp.local@US-POCLAB.DELLPOC.COM
Using default cache: /tmp/krb5cc_1000
Using principal: hive/dwx-env-feddatalakedev-env.cdp.local@US-POCLAB.DELLPOC.COM
Using keytab: /etc/security/keytabs/hive.service.keytab
kinit: Client's credentials have been revoked while getting initial credentials
+ klist
klist: No credentials cache found (filename: /tmp/krb5cc_1000)
+ [[ -z '' ]]
+ echo 'CDW_HIVE_AUX_JARS_PATH is not defined. Skipping jars download from path..'
+ exit
CDW_HIVE_AUX_JARS_PATH is not defined. Skipping jars download from path..

Cause

The issue occurs when Kerberos settings are changed but not updated in Cloudera Data Warehouse, leading to outdated keytab files or credentials cache.

Solution

Log in to Cloudera Manager as an administrator.
Stop all services in the base cluster, including Management services.
Go to Cluster Actions and deploy the Kerberos client configuration.

note
If krb5.conf is managed by Cloudera Manager, this step updates it automatically. If krb5.conf is not managed by Cloudera Manager, manually copy the updated /etc/krb5.conf file to all servers in the base cluster.
Go to Administration > Security > Kerberos, select all Kerberos principals, and click Regenerate Keytabs.
Start all stopped services.
Refresh the environment, database catalog, and all Virtual Warehouses by clicking the refresh option for each respective window. For more information, see Refresh Cloudera Data Warehouse.

Alternative Resolution Steps

If the above steps do not resolve the issue, perform the following:

Delete the Virtual Warehouses.
Deactivate the environment.
Reactivate the environment.
Create the Hive Virtual Warehouses again. See Add Virtual Warehouse.