Cloudera Docs

Troubleshooting salt secret rotations

When the salt secret rotation fails on the Cloudera Data Hub clusters, the issue may not be resolved by retrying the operation. A manual workaround is necessary to fix the rotation error.

Condition

Based on the last operation that failed because of the failed salt password rotation on the Cloudera Data Hub cluster, the following error messages can indicate that the rotation failed:
  • Status: 401 Unauthorized
  • Secret rotation rollback finished: [SaltStack orchestrator password]
  • There is already a failed secret rotation for SALT_PASSWORD secret type

Remedy

  1. Access the Gateway nodes of the Cloudera Data Hub cluster, where the secret rotation has failed: 
    $ ssh [***WORKLOAD USER***]@[***NODE IP ADDRESS***]
  2. Delete the saltuser on the nodes using the following command: 
    userdel saltuser
  3. Run the update-orchestrator-state command to resolve the SaltStack connection: 
    cdp datahub update-orchestrator-state --cluster-crn [***DATA HUB CRN***]
  4. Rotate the SaltStack orchestrator password from Cloudera Management Console or SALT_PASSWORD from CDP CLI based on the Rotating Cloudera Data Hub secrets documentation.
  5. Retry any failed operation due to the salt secret error for healthy cluster state.