Cloudera Docs

Using migrate-all with flow definition JSON as input

Learn how to use the migrate-all command to migrate variables and components in a single operation using a flow definition JSON file as input.

This command combines the functionality of the migrate-variables and migrate-components commands for ease and convenience.

While migrate-all simplifies the migration process, for larger flows, it is usually better to run the individual migration commands separately. You may also want to migrate process groups individually to ensure activity logs and manual validation remain manageable.

The migrate-all command can be useful in the following scenarios:

  • The flow you want to migrate is simple and of manageable size.

  • The flow is large and complex, and you need an initial high-level overview of the migration outcome before performing a step-by-step migration. Running migrate-all provides a preview of the final migrated flow. The Activity Log shows you the actions taken during migration and helps assess the required manual intervention by listing manual-change-requests and manual-validation-requests.

Migrating variables and components together using migrate-all with flow definition JSON as input

  1. Offload all flowfiles from NiFi.
  2. Stop all processors and disable all controller services in NiFi.
  3. Stop NiFi.
  4. Copy the flow.json.gz file from NiFi’s conf directory to the Migration Tool’s input folder (/etc/migration-tool-input).
  1. Run the following command to migrate the flow. 
    bin/migration.sh nifi migrate-all \
-i /etc/migration-tool-input/flow.json.gz \
-od /etc/migration-tool-output

    This generates the following output:

    sourceVersion
├── NiFi_Flow_b4175f57-0194-1000-8470-9251a24519b4
│   ├── elastic_template.json
│   └── elastic_template.xml
├── activity_log.json
└── migrated_flow.json.gz

targetVersion
├── NiFi_Flow_b4175f57-0194-1000-8470-9251a24519b4
│   └── elastic_template.json
├── activity_log.json
└── migrated_flow.json.gz
  2. Review the migration output.

    • The sourceVersion directory contains the NiFi 1-compatible version of the flow, including the exported template and its process group counterpart, along with the Activity Log.

    • The targetVersion directory contains the NiFi 2-compatible version of the flow and its corresponding Activity Log.

  3. Address issues in NiFi 1 (if needed).

    If sourceVersion/activity_log.json contains manual-change-requests, follow these steps:

    1. Load sourceVersion/migrated_flow.json.gz into your NiFi 1 instance.
    2. Apply the required manual changes.
    3. Run the migrate-all command again using the manually modified flow.
  4. Validate the migrated flow.
    1. Rename targetVersion/migrated_flow.json.gz to flow.json.gz.
    2. Load flow.json.gz into your NiFi 2 instance to inspect the migrated flow.
    3. Open targetVersion/activity_log.json and review any manual-change-requests and manual-validation-requests.
    4. Apply the required manual modifications to the flow.