Cloudera Docs

Using migrate-all using a directory with flow definition JSON files as input

Learn how to use the migrate-all command to migrate templates, variables, and components in a single operation using a directory with flow definition JSON files as input.

This command combines the functionality of the migrate-variables and migrate-components commands for ease and convenience.

While migrate-all simplifies the migration process, for larger flows, it is usually better to run the individual migration commands separately. You may also want to migrate process groups individually to ensure activity logs and manual validation remain manageable.

The migrate-all command can be useful in the following scenarios:

  • The flow you want to migrate is simple and of manageable size.

  • The flow is large and complex, and you need an initial high-level overview of the migration outcome before performing a step-by-step migration. Running migrate-all provides a preview of the final migrated flow. The Activity Log shows you the actions taken during migration and helps assess the required manual intervention by listing manual-change-requests and manual-validation-requests.

Copy the TCP_Listener_flow_definition.json and Elastic_flow_definition.json to the Migration Tool’s input folder (/etc/migration-tool-input).

  1. Run the following command to migrate the flow. 
    bin/migration.sh nifi migrate-all \
-i /etc/migration-tool-input/ \
-od /etc/migration-tool-output

    This generates the following output:

    sourceVersion
    ├── activity_log.json
    └── migrated_output
        ├── migrated_Elastic.json
        └── migrated_TCP_Listener.json

targetVersion
    ├── activity_log.json
    └── migrated_output
        ├── migrated_Elastic.json
        └── migrated_TCP_Listener.json
  2. Review the migration output.

    • The sourceVersion directory contains the NiFi 1-compatible version of the flow definition and the Activity Log.

    • The targetVersion directory contains the NiFi 2-compatible version of the flow definition and its corresponding Activity Log.

  3. Address issues in NiFi 1 (if needed).

    If sourceVersion/activity_log.json contains manual-change-requests, follow these steps:

    1. Load sourceVersion/migrated_output/<flow_definition.json> into your NiFi 1 instance.

    2. Apply the required manual changes.

    3. Run the migrate-all command again using the manually modified flow.

  4. Validate the migrated flow.

    1. Load targetVersion/migrated_output/<flow_definition.json> into your NiFi 2 instance to inspect the migrated flow.

    2. Open targetVersion/activity_log.json and review any manual-change-requests and manual-validation-requests.

    3. Apply the required manual modifications to the flow.