Recommended migration workflow

Cloudera recommends performing the migration in two sequential stages (Stage 1 and Stage 2) to ensure a structured transition from source to target compatibility.

A complete migration workflow consists of a Stage 1 migration of the incoming flow followed by a Stage 2 migration of the Stage 1 migration output. The final output of Stage 2 is a fully migrated and NiFi 2 compatible flow.

Stage 1: Source compatibility

It processes the incoming NiFi flow (input flow.json) to create a partially updated flow as the output, compatible with NiFi 1.

Running Stage 1 commands for a template, variable, component, or using the aggregate command produces transformations compatible with the original version.
Generates a list of manual changes needed for further adjustments.
Requires iterative execution after applying manual changes until the flow reaches stability (no further modifications are required).

Stage 2: Target compatibility

It takes Stage 1 output as input to produce a flow compatible with NiFi 2.

- Applies transformations beyond the source NiFi version's capabilities to meet NiFi 2 standards.
- Ensures that the final output is a fully migrated and compliant data flow.

Both migration stages consist of multiple steps that can be run individually or as a whole. Cloudera recommends completing all steps within a stage before proceeding to the next stage to ensure consistency and avoid potential issues.

Component migration is present in both stages but involve different transformations and serve different purposes:

In Stage 1, component migration applies changes compatible with the source version.
In Stage 2, component migration applies changes required by the target version.

Stage 1 migration

Template migration
1. Migrate templates for Stage 1.
2. Validate the results.
3. Address any issues and manual change requests.
Variable migration
1. Migrate variables to parameter contexts for Stage 1.
2. Validate the results.
3. Address any issues and manual change requests.
Component migration
1. Migrate components in smaller batches by specifying Process Group IDs for Stage 1.
2. Validate the results for each batch.
3. Address any issues and manual change requests.
4. Repeat iteratively until no further changes are required.
Management-level component migration
1. Perform component migration without specifying a Process Group ID to ensure management-level components such as Controller Services are migrated.
2. Validate the changes.
3. Address any issues and manual change requests.
Final validation
1. When all Stage 1 steps are complete, validate the entire flow in a NiFi 1 instance for final validation.
2. After this point, the flow will no longer be compatible with the source version.

Stage 2 migration: Target version compatibility

Run the migration commands without specifying the Stage argument to process both stages in sequence and address any remaining compatibility issues with NiFi 2.

note

This runs both Stage 1 and Stage 2, but if previous recommendations were followed, Stage 1 has no impact on the flow. While Stage 1 commands should no longer modify the flow, some Stage 2 commands depend on Stage 1 outputs as input (for example for template migration).
Follow the same steps as in Stage 1: Migrate templates → Migrate variables → Migrate components, while validating the results and addressing manual adjustment needs.
Stage 2 migration is not iterative in a sense that the output of Stage 2 cannot be reintroduced into the migration process. To ensure a seamless workflow, fully complete Stage 1 before starting Stage 2, and run Stage 2 only once. The final output will reflect all transformations applied by the Migration Tool. Resolving manual change requests will not trigger further migration events. Any modifications beyond this point are considered business logic adjustments and fall outside the tool’s scope.

For detailed instructions and examples of the end-to-end migration process, see Migrating a data flow.