Cloudera Docs

Using migrate-all

Learn how to use the migrate-all command to migrate templates, variables, and components in a single operation.

This command combines the functionality of the migrate-templates, migrate-variables, and migrate-components commands for ease and convenience. While migrate-all simplifies the migration process, for larger flows, it is usually better to run the three migration commands separately. You may also want to migrate process groups individually to ensure activity logs and manual validation remain manageable.

The migrate-all command can be useful in the following scenarios:

  • The flow you want to migrate is simple and of manageable size.

  • The flow is large and complex, and you need an initial high-level overview of the migration outcome before performing a step-by-step migration. Running migrate-all provides a preview of the final migrated flow. The Activity Log shows you the actions taken during migration and helps assess the required manual intervention by listing manual-change-requests and manual-inspection-requests.

Migrating templates, variables, and components together using migrate-all

  1. Offload all flowfiles from NiFi.
  2. Stop all processors and disable all controller services in NiFi.
  3. Stop NiFi.
  4. Copy the flow.json.gz file from NiFi’s conf directory to the Migration Tool’s input folder (/etc/migration-tool-input).
  5. Unzip the file to obtain flow.json.
  1. Run the following command to migrate the flow. 
    bin/migration.sh nifi migrate-all \
-i /etc/migration-tool-input/flow.json \
-od /etc/migration-tool-output

    This generates the following output:

    v1
├── NiFi_Flow_b4175f57-0194-1000-8470-9251a24519b4
│   ├── elastic_template.json
│   └── elastic_template.xml
├── activity_log.json
└── migrated_flow.json

v2
├── NiFi_Flow_b4175f57-0194-1000-8470-9251a24519b4
│   └── elastic_template.json
├── activity_log.json
└── migrated_flow.json
  2. Review the migration output.

    • The v1 directory contains the NiFi 1-compatible version of the flow, including the exported template and its process group counterpart, along with the Activity Log.

    • The v2 directory contains the NiFi 2-compatible version of the flow and its corresponding Activity Log.

  3. Address issues in NiFi 1 (if needed).

    If v1/activity_log.json contains manual-change-requests, follow these steps:

    1. Load v1/migrated_flow.json into your NiFi 1 instance.
    2. Apply the required manual changes.
    3. Run the migrate-all command again using the manually modified flow.
  4. Validate the migrated flow.
    1. Rename v2/migrated_flow.json to flow.json.
    2. Compress it into flow.json.gz.
    3. Load flow.json.gz into your NiFi 2 instance to inspect the migrated flow.
    4. Open v2/activity_log.json and review any manual-change-requests and manual-inspection-requests.
    5. Apply the required manual modifications to the flow.

Migrating a process group using migrate-all

For complex flows, you can migrate one process group and on one stage at a time. See below for an example workflow.

  1. Migrate the TCP Listener process group using the following command. 
    bin/migration.sh nifi migrate-all \
-i /etc/migration-tool-input/flow.json \
-od /etc/migration-tool-output -pgid b41940d7-0194-1000-42fc-458834630567 \
--stage STAGE_1
  2. Use the output v1/migrated_flow.json (the result of the previous command moved to the input folder) as the input for migrating the Elastic process group. 
    bin/migration.sh nifi migrate-all \
-i /etc/migration-tool-input/flow.json \
-od /etc/migration-tool-output -pgid b42881c7-0194-1000-3cdf-1bd453a0ed0f \
--stage STAGE_1
  3. Use the output v1/migrated_flow.json (the result of the previous command moved to the input folder) as the input for migrating the root process group and completing the Stage 2 migration. 
    bin/migration.sh nifi migrate-all \
-i /etc/migration-tool-input/flow.json \
-od /etc/migration-tool-output

    This produces a NiFi 2-compatible version of the flow.