Secret rotation timeout

Condition

Secret rotation times out after approximately 30 minutes without Cloudera-side error with the following conditions:

Cloudera Management Console shows only a generic Salt timeout.
Multiple nodes are affected at once.
Minions are running but unresponsive to the master with stuck jobs accumulating.
Host-side EDR or antivirus process (mdatp, falcon-sensor, sentinel, etc.) is consuming CPU and holding filesystem locks.

This indicates that Salt is blocked at the host OS layer, not a Cloudera-side rotation failure.

Run the following commands on an affected node:

# Active EDR/antivirus?
sudo top -b -n1 | head -30
ps -ef | grep -i -E 'mdatp|crowdstrike|sentinel|carbon|esets|clamd|sophos' | grep -v grep
# Minions up but unresponsive, stuck jobs accumulating?
sudo systemctl status salt-minion
sudo salt-run jobs.active
sudo salt-run jobs.list_jobs
salt --version

Remedy

Wait for the active EDR or antivirus scans to finish. This typically requires 20–60 minutes per node.
Restart Salt minions cluster-wide to drain residual stuck jobs:
salt '*' service.restart salt-minion
Update the orchestrator state for the affected cluster:
- Data Hub: cdp datahub update-orchestrator-state --cluster-crn <DATA HUB CRN>
- FreeIPA: cdp environments update-orchestrator-state for the environment
Re-run the failed rotation :
- Data Hub: cdp datahub rotate-secrets --datahub <datahubCrn> --secret-types <...>
- FreeIPA: cdp environments rotate-freeipa-secrets --environment <environmentCrn> --secret-types <...>
Instruct your security team to apply the EDR exclusions detailed in Manually renewing public certificates for Data Lake and Cloudera Data Hub clusters so they don't recur on the next scan window.