Deployments

When you access the NiFi cluster, the ability to view or edit the flow is based on your Cloudera DataFlow authorizations. The DFFlowUser role has read-only privileges. The DFFlowAdmin role has full privileges.

Starting a flow deployment starts all processors of a Cloudera DataFlow deployment.

Stopping a flow results in the following:

• All processors are stopped and no data processing happens within the NiFi flow.
• KPI alerts are stopped. Your KPI alerts are activated again when the flow is restarted.
• Any active KPI alerts are resolved.
• All underlying cloud resources remain allocated for the Cloudera DataFlow deployment.
• You can modify deployment configuration while the flow is stopped.
• Stopped flows are still billable however if auto-scaling is enabled for the flow, a certain amount of cost reduction may occur.

Restrictions

The following restrictions apply to flow version changes:

• Changing inbound connections is not supported.

• Changing custom resource (custom NARs and custom Python resources) configurations is not supported.

• While you can add, change or remove assets when moving to a new version, you cannot introduce assets (text files, binaries, JARs, or similar) if the currently deployed version does not have any.
• Components where state or provenance and other repositories must be kept between flow versions must keep their flow JSON ids. The id changes if you move the component to a different process group or if you delete and then re-add the component to the same process group. NiFi Identifies components by these ids. If you move a component to a different process group between versions, its id changes and NiFi perceives it as a new component. This results in the original component being deleted during flow version change together with its state and a new, identical processor being created in a different process group. In an extreme case, you could change to an identical flow version with just the component ids changed and it would result in the deletion of the entire NiFi flow and the recreation of an identical one, with all history and data lost.

• Remapping Parameter Group and Parameter Context assignments is not supported as the original assignment is not removed. For example, you have Process Group 1 (PG1) with Parameter Context 1 (PC1) and Process Group 2 (PG2) with Parameter Context 2 (PC2) assigned. If you initiate a flow version change where parameter contexts are flipped, resulting in a PG1-PC2 and PG2-PC1 assignment, NiFi will not re-map the PG to PC assignments.

Version change strategies

Depending on the type of your flow, you may select the flow version change strategy most appropriate to you.

Stop & Process Data

This strategy prioritizes data consistency by stopping source processors and waiting until data is processed before stopping all other components. Once all components have stopped, the flow version is changed and components are started.

Use this strategy when your sources are durable and can handle your source processor being stopped. This generally works well when your source processors are pulling data from sources like Kafka or other messaging queues, databases or file systems.

Should the queued data not be processed within the set time, version change will fail and you can retry the operation with a bigger timeout or you can cancel

Only Restart Affected Components

This strategy prioritizes uptime by identifying and stopping only components that have changed while keeping all others running, replacing and then starting affected components.

Use this strategy when you want to prioritize uptime of unchanged components or you have made only minor processor configuration changes.

This works well for deployments with inbound connections and will keep your source processors running if they have not changed compared to the previous version.

Stop & Empty Queues

This strategy forces a version change by stopping all components, emptying all queues, changing flow version, and then starting all components.

Use this strategy only when you want to force a flow version change without keeping any processors running or attempting to process queued data.

All processors will be stopped and all queues will be emptied as part of this strategy.

Suspending a Cloudera DataFlow deployment results in the following:

• The NiFi flow stops processing data and all underlying cloud resources are terminated. Any unprocessed data in the flow is stored in memory and its processing resumes when you resume the deployment.

• Flow persistence is maintained while a deployment is suspended.
• You cannot modify deployment configuration while the deployment is suspended.
• Suspended deployments are not billable, resulting in reduced costs.

Resuming a Cloudera DataFlow deployment reallocates the underlying cloud resources and returns a deployment to the state it was in before being suspended.

• Exported configurations may be edited, and you can also modify them after the importing step during flow deployment.
• One deployment can have only one exported configuration. Performing a new export overwrites the existing one.
• Exported deployment configurations are available for every user who can start a new deployment in a given environment, even if the exported deployment was originally created under a specific Project.

If you terminate a deployment, you delete the associated NiFi resources and your flow no longer remains active. The associated flow definition remains in the catalog and is available to be deployed again in a new deployment.