Switching from Azure AD Pod Identity (aad-pod-identity) to Workload Identity in Azure clusters

In Cloudera Data Engineering 1.25.2, Workload Identity replaces the Azure AD Pod Identity (aad-pod-identity) component used for some of the workloads in Cloudera Data Engineering to pull logger credentials.

Workload Identity is more secure, provides faster startup times with better scaling and enables you to use more granular permissions.

For more information, see Migrate Azure Kubernetes Service (AKS) pods from pod-managed identity to Microsoft Entra Workload ID in the Azure documentation.

Due to the limitation of Federated Identity Credential (FIC) Mapping, which allows only 20 FIC mappings per Managed Identity, in Cloudera Data Engineering, you cannot use the logger identity provided at the environment level to a full scale.

For more information, see Use managed identity as a Federated Identity Credential (FIC) on an Entra ID app in the Microsoft documentation.

Prerequisites for using Workload Identity in Azure clusters:

You must create two Managed Identities and assign appropriate roles to them. For more information, see Creating user-assigned Managed Identities.
important
Dedicated Managed Identities are required. You must create and use two unique Managed Identities for each Cloudera Data Engineering service. Do not share these two identities with another Cloudera Data Engineering service or any other data service.
While creation workflows often block the use of shared identities, upgrade workflows may not strictly enforce this rule. Bypassing this requirement will lead to unexpected operational issues later.
You need to assign the account contributor permissions for Azure Environment credentials. For more information, see, Adding the account contributor permissions for Azure environment credentials.

Workload Identity-related prerequisites for the in-place upgrade

In-place upgrade operations require patching the existing Cloudera Data Engineering service to update Managed Identities through the Cloudera Data Engineering UI or the patchCluster API.

For more information, see In-place upgrade with Airflow Operators and Libraries.

Workload Identity-related prerequisites for the Cloudera Data Engineering service-level backup and restore

Restore operations require overriding existing Managed Identities with new ones through CDE CLI options.

For more information, see Using the backup-restore-based upgrade script.