Cloudera Docs

Troubleshooting DataFlow upgrade errors

Learn how to recognize and address common errors with your Cloudera DataFlow (CDF) service and flow deployment upgrades.

Setting up kubectl to connect to the DataFlow Kubernetes cluster

It is helpful to have access to the DataFlow Kubernetes cluster using command line tools such as kubectl when you are troubleshooting deployment or upgrade failures. To set up kubectl access, follow these steps:

  1. In DataFlow, from the Environments page, select the DataFlow service for which you want to add or remove user access.
  2. Click Manage DataFlow.
  3. From the Actions menu, click Manage Kubernetes API Server User Access.
  4. Add the AWS IAM role that you will authenticate as to the list of authorized users for DataFlow by entering the ARN in the Add User dialog.
  5. Use the Download Kubeconfig action to retrieve the kubeconfig file for connecting to your cluster.
  6. Set up your kubectl to use the downloaded kubeconfig file:
    export KUBECONFIG=[***PATH/TO/DOWNLOADED/KUBECONFIG/FILE***]
  7. Run kubectl get ns and validate that your output looks similar to:

With kubectl being set up correctly, you are able to access NiFi and other DataFlow logs directly through the CLI.

Understanding upgrade failures

There are various reasons that may prevent an upgrade from happening. This document covers the following scenarios:

  • The upgrade fails upon initiation.
  • The upgrade starts but fails, and rollback is possible.
  • The upgrade starts but fails and rollback is not possible.

Upgrade fails on initiation

Symptom: When the upgrade fails to start, the status of the environment does not change to Upgrading and you receive an error message stating the cause of the failure.

Possible causes: The service is in one of the following states:

  • The service is in one of the following states:

    • Failed to Update

    • Failed to Update NiFi version

    • Any transitional / in-progress state

  • Any of the deployments is in one of the following states:

    • New

    • Failed to Update

    • Failed to Update NiFi version

    • Any transitional / in progress state

How to fix it: Fix the error indicated in the error message, then retry the upgrade. If the upgrade still fails, contact Cloudera Support.

Upgrade fails and rollback is possible

Symptom: Upgrade rollback process starts. If the rollback is successful, the Status of the service returns to Good Health, CDF and NiFi versions return to their pre-upgrade values. If the rollback fails, the Status of both the CDF service and running deployments becomes Failed to Upgrade. The order in which this transition happens depends on where the rollback actually failed.

How to fix it: In the event of an upgrade failure, DataFlow determines if rollback is possible and automatically initiates it. No user intervention is required.

If the rollback is successful:

  1. After the rollback ends, from the Environments page, select the DataFlow service you were trying to upgrade and click the Alerts tab. Check Event History to see if there is anything obvious in there or something that narrows down the possible cause of the failure.

  2. From the Dashboard page select running deployments that failed to upgrade and check Alerts>Event History to see if there is anything obvious in there or something that narrows down the possible cause of the failure.

  3. Check the K8s cluster using CLI and try to fix errors based on the information you obtained from Event History in the previous steps.

  4. Retry the upgrade.

  5. If the upgrade still fails contact Cloudera Support.

If the rollback fails:

The status of both the service and the running deployments changes to Failed to Update. The order in which they transition depends on where the rollback actually failed.

Upgrade fails and rollback is not possible

Symptom: The status of the service, and shortly after the deployments, changes to Failed to Upgrade. The status icon on the service changes to Bad Health.

How to fix it:

  1. If K8s upgrade failed, check the Event History if there is anything obvious in there.

  2. Check on the cloud provider side whether there is anything obvious as a reason for the failure. For example, check activity logs and node pools on Azure, or EKS cluster on AWS.

  3. If the K8s upgrade succeeded, but the upgrade failed somewhere later, then the previous point in the "Upgrade fails and rollback is possible" can follow here (check the events and then the K8s cluster).

  4. If you managed to pinpoint and fix the possible cause of the failure, retry the upgrade.

  5. If you did not find the cause of the failure, or the upgrade still fails upon retry, Contact Cloudera Support.

Retrying an upgrade

  1. On the DataFlow Dashboard select Environments from the left navigation pane.
  2. Select the service where you want to retry the upgrade and click Manage DataFlow.
  3. Click Actions > Retry Upgrade DataFlow.