Cloudera Docs

Recovering from failed upgrades

If a Data Lake upgrade fails and you are unable to manually troubleshoot the problem, you may be able to use the recovery process to return the cluster to its pre-upgrade state.

If FreeIPA is available and the Data Lake cluster is in a recoverable state (meaning that there has been an uncorrected failed upgrade or failed recovery), a recovery option may be available after a failed upgrade. Recovery after a failed upgrade retains the Data Lake CRN, UMS mappings, load balancers, and RDS instance and brings up new instances with the original image and Cloudera Runtime version, but new disks and new databases.
  1. Use the CDP CLI to recover the Data Lake after a failed upgrade:
    cdp datalake recover-datalake
--datalake-name <value>
[--recovery-type <value>]
    Parameter Description
    --datalake-name Name or CRN of the Data Lake that you want to recover after a failed upgrade.
    --recovery-type

    The type of the recovery. The default value is RECOVER_WITHOUT_DATA.

    Currently, the option RECOVER_WITH_DATA is not supported.

    The status of the Data Lake appears as "Datalake recovery in progress. Recovery process takes a while as the nodes are being terminated and new nodes are launched with the original runtime."
  2. Restore the Data Lake from the pre-upgrade backup. For more information, see Restore Data Lake content.
  3. If necessary, run the cdp datalake sync-component-versions-from-cm command from the CDP CLI.

    When an upgrade fails, the versions of Cloudera Manager, Cloudera Runtime, and other components may become out-of-sync with the Cloudera Management Console. Similarly, if you try to fix errors by installing parcels manually, it may not be reflected in the Cloudera Management Console.

    To overcome the mismatch between versions reflected in Cloudera Management Console, run the cdp datalake sync-component-versions-from-cm CDP CLI command. This commands reads the Cloudera Manager, Cloudera Runtime, and other parcel versions (if applicable) from Cloudera Manager and updates the versions in Cloudera Management Console. Using this command forces Cloudera Management Console back in sync so that it shows the actual versions installed in Cloudera Manager. 

    cdp datalake sync-component-versions-from-cm --datalake-name <datalake name>