Known issues and limitations

When you encounter unexpected behavior or plan deployments, you can use this reference to identify known issues, their impact, and available workarounds in Cloudera Data Flow. Reviewing these items helps you avoid problems and resolve issues more quickly when they occur.

Known issues

CFM-6903: Flow deployments migrated using the Migration Tool may fail with Parameter Context update errors when sensitive parameters are involved

The Migration Tool does not correctly preserve user-defined sensitive processor properties during migration. Sensitive parameters are incorrectly treated as non-sensitive, causing conflicts in inherited parameter contexts. Subsequent flow deployment fails with an HTTP 409 error related to sensitive parameter validation.

Open the NiFi 2.x flow in Flow Designer, delete the user-defined properties that cause conflicts, and re-add them with the exact same names and values but with the 'Sensitive' flag explicitly set to 'true'.

For more information on adding parameters to drafts, see Adding a parameter to your draft.
For more information on migrating flows, see Cloudera Data Flow NiFi 2 migrations.

ENGESC-33336: Parameter loses its file‑type control during deployment

When an asset file is uploaded to a property within Flow Designer and subsequently published to the Catalog, Cloudera Data Flow does not recognize the property as an Asset Type during deployment. Consequently, it does not accept a file as input by the user.

You may resolve this issue by downloading the relevant flow definition and manually updating the JSON structure. Specifically, adding a resourceDefinition field with the appropriate property descriptors enables asset file uploads upon re-uploading the definition to the Catalog.

Build the flow and publish it to the Catalog, if it is not already built.
For more information, see Publishing a draft as a new flow definition.
Download the flow definition from the Catalog.
For more information, see Downloading the latest version of a flow definition.
Add the following resourceDefinition field to the downloaded flow definition, under the desired property definition:
```
"resourceDefinition": {
    "cardinality": "MULTIPLE",
    "resourceTypes": ["TEXT", "FILE"]
}
```
Save the changes to the flow definition file.
Upload the edited flow definition to the Catalog.
For more information, see Importing flow definitions from Apache NiFi.
Deploy the flow from the Catalog, uploading the asset file to the desired property in the deployment wizard.
For more information, see Deploying a flow using the wizard.

This example shows a “Command Arguments” property definition, but any other property that accepts asset files may be amended:

"Command Arguments": {
    "name": "Command Arguments",
    "displayName": "Command Arguments",
    "identifiesControllerService": false,
    "sensitive": false,
    "dynamic": false,
    "resourceDefinition": {
        "cardinality": "MULTIPLE",
        "resourceTypes": ["TEXT", "FILE"]
    }
}

ENGESC-28680: Flow Missing on Cloudera Data Flow Canvas

In rare cases, a single-node deployment may disappear from Cloudera Data Flow after a deployment restart. The root cause is related to leader election timeout within NiFi cluster deployments when scaling up from a single node: the cluster restarts with a single node and quickly scales to two nodes. The race condition between leader election and the single node restarting leads to an unhealthy deployment state and causes the NiFi deployment to disappear from Cloudera Data Flow. There is currently no fix that allows single-node deployments to avoid this race condition.

One way to prevent this situation from occurring is to configure the NiFi cluster deployment with a minimum node count of two nodes instead of one. This setup improves initial cluster stability and reduces the chance of electing a leader node without any flow, preventing the loss of flow definitions.

An alternative workaround for regularly inactive jobs is suspending a deployment when flows are not running, instead of stopping it completely. This keeps the deployment definition intact while reducing operational overhead.

CDPDFX-7121: Environment unavailable for new deployments while diagnostics collection is running

During deployment creation you cannot select Cloudera Data Flow environments where diagnostic bundle collection is ongoing.

Wait for diagnostic bundle collection to finish before proceeding with flow deployment.

CDPDFX-10009: Pinecone processors require value for obsolete environment parameter

Pinecone no longer requires the environment to connect, but you still need to provide a value for the Pinecone Environment parameter for deployments to succeed as the related property in Pinecone processors is required.

Provide a random string as the parameter value.

Cloudera Data Flow Functions: DEFAULT_PARAM_CONTEXT variable no longer works alone

This issue only occurs in AWS environments and affects configurations where there are no PARAM_CONTEXT_ variables defined, only a DEFAULT_PARAM_CONTEXT.

The DEFAULT_PARAM_CONTEXT configuration variable instructs Cloudera Data Flow Functions which default secret to use when there is no secret that matches the parameter contexts in the flow. This variable is now ignored.

Create an environment variable in Configuration called PARAM_CONTEXT_[***NAME***] where [***NAME***] is the user-defined name of the parameter context. Specify the name of the AWS Secret you want to use as the value of this variable.

IAM Policy Simulator preflight check fails with resource policy validation

With all cross account policies in place, IAM Policy Simulator preflight check still fails with the following error message:

IAM Resource Policy validation failed on AWS. CrossAccount role does not have permissions for these operations : : ssm:GetParameter, ssm:GetParameters, ssm:GetParameterHistory, ssm:GetParametersByPath

This happens because even if a given cross account role is allowed to perform a certain action (granted through IAM Policies), an attached Service Control Policy (SCP) may override that capability if it enforces a Deny on that action. SCP takes precedence over IAM Policies. SCPs are either applied at the root of an organization, or can be applied to individual accounts. A permission can be blocked at any level above the account, either implicitly or explicitly (by including it in a Deny policy statement).

As the IAM Simulator SDK does not have an option to include or exclude an organization’s SCP policy, the preflight check will fail if an SCP policy is denying an action, even though the IAM role has the necessary permissions.

This is a known issue in AWS.

Do not select the Skip Validations option when enabling Cloudera Data Flow to bypass this issue. This bypasses all preflight validation checks. Instead, submit a request to add the LIFTIE_DISABLE_IAM_PREFLIGHT_CHECK entitlement to your account which ensures only the IAM Policy preflight validation check is skipped.

Limitations

NiFi 2.x runtime downgrades

Apache NiFi 2 introduced breaking changes and backwards compatibility cannot be guaranteed, therefore NiFi 2 downgrades are not permitted. This does not include hotfix version downgrades within the same NiFi runtime version (for example, 2.6.0.4.3.4.0-166 to 2.6.0.4.3.4.0-123).

NiFi 2 upgrades

Apache NiFi version 2.3 became generally available (GA) in Cloudera Data Flow release 2.10.0. Previous NiFi 2 versions in Cloudera Data Flow 2.9.0 or lower were provided as Technical Preview and therefore are not supported for upgrade to 2.10.0 or higher. Before performing a service upgrade from release 2.9.0 or lower to release 2.10.0 or higher, you must remove all NiFi 2 flow deployments.

Reassigning resources

Parameter groups that have referencing flow drafts cannot be reassigned to another project.
Flow drafts referencing a parameter group cannot be reassigned to another project.
Project reassignment does not move assets. When reassigning a parameter group that includes a FILE type parameter (asset) reference to another project, that asset needs to be re-uploaded to the new project.
Flow deployments referencing a parameter group cannot be reassigned to another project.

Parameter group duplication

Duplication of parameter groups referencing assets does not duplicate referenced assets.

After duplicating a parameter group, assets have to be re-uploaded manually. This means that if you reassign a parameter group to another project, duplicate a parameter group, or export a parameter group, the asset will not be moved or duplicated.

Duplication can only happen inside a given project. If the newly created group is to be assigned to another project, that step can happen only after the duplication concludes. (Another project cannot be targeted for duplication.)

Diagnostic Bundle collection through the Management Console is available on the US Control Plane only

There is no workaround for this issue.

Data Lineage information is not automatically reported to Atlas in the Data Catalog

Flow deployments created by Cloudera Data Flow do not come with a pre-configured ReportLineageToAtlas Reporting Task.

If you have been assigned the DFFlowAdmin role, you can manually create and configure the ReportLineageToAtlas Reporting Task in the NiFi canvas after a deployment is completed.

PowerUsers are not able to create flow deployments without additional Cloudera Data Flow roles

While the PowerUser role gives you the ability to view flow deployments in the Dashboard, view flow definitions in the Catalog, and allows the user to initiate flow deployments, the Deployment Wizard fails after selecting an environment for which the user does not have the DFFlowAdmin resource role assigned.

Assign the DFFlowAdmin role to the user of the environment to which they want to deploy flow definitions.

Cloudera Data Flow reports "Bad Health" during Data Lake upgrade

Cloudera Data Flow monitors the state of the associated Cloudera on cloud environment to decide which actions Cloudera Data Flow users can take. Cloudera Data Flow detects Data Lake upgrades of the associated Cloudera on cloud environment and puts the Cloudera Data Flow service into Bad Health for the duration of the upgrade blocking new deployments.

To work around this issue, wait for the Data Lake upgrade to complete before creating new flow deployments.

Deployments and Cloudera Data Flow Services are no longer visible in the Cloudera Data Flow Dashboard or Environments page when the associated Cloudera on cloud Environment has been deleted

If the associated Cloudera on cloud Environment is deleted while a Cloudera Data Flow Service is enabled, it will become orphaned. Orphaned resources are no longer visible to users without the PowerUser role.

To work around this issue, open the Environments or Dashboard page with a user who has been assigned the PowerUser role. PowerUsers are able to view orphaned deployments and Cloudera Data Flow services.

Non-transparent proxies are not supported on Azure

There is no workaround for this issue.