Cloudera Docs

Using migrate-all with flow.json.gz as input

Learn how to use the migrate-all command to migrate templates, variables, and components in a single operation using a flow.json.gz file as input.

This command combines the functionality of the migrate-templates, migrate-variables, and migrate-components commands for ease and convenience. While migrate-all simplifies the migration process, for larger flows, it is usually better to run the three migration commands separately. You may also want to migrate process groups individually to ensure activity logs and manual validation remain manageable.

The migrate-all command can be useful in the following scenarios:

  • The flow you want to migrate is simple and of manageable size.

  • The flow is large and complex, and you need an initial high-level overview of the migration outcome before performing a step-by-step migration. Running migrate-all provides a preview of the final migrated flow. The Activity Log shows you the actions taken during migration and helps assess the required manual intervention by listing manual-change-requests and manual-validation-requests.

Migrating templates, variables, and components together using migrate-all with flow.json.gz as input

  1. Offload all flowfiles from NiFi.
  2. Stop all processors and disable all controller services in NiFi.
  3. Stop NiFi.
  4. Copy the flow.json.gz file from NiFi’s conf directory to the Migration Tool’s input folder (/etc/migration-tool-input).
  1. Run the following command to migrate the flow. 
    bin/migration.sh nifi migrate-all \
-i /etc/migration-tool-input/flow.json.gz \
-od /etc/migration-tool-output

    This generates the following output:

    sourceVersion
├── NiFi_Flow_b4175f57-0194-1000-8470-9251a24519b4
│   ├── elastic_template.json
│   └── elastic_template.xml
├── activity_log.json
└── migrated_flow.json.gz

targetVersion
├── NiFi_Flow_b4175f57-0194-1000-8470-9251a24519b4
│   └── elastic_template.json
├── activity_log.json
└── migrated_flow.json.gz
  2. Review the migration output.

    • The sourceVersion directory contains the NiFi 1-compatible version of the flow, including the exported template and its process group counterpart, along with the Activity Log.

    • The targetVersion directory contains the NiFi 2-compatible version of the flow and its corresponding Activity Log.

  3. Address issues in NiFi 1 (if needed).

    If sourceVersion/activity_log.json contains manual-change-requests, follow these steps:

    1. Load sourceVersion/migrated_flow.json.gz into your NiFi 1 instance.
    2. Apply the required manual changes.
    3. Run the migrate-all command again using the manually modified flow.
  4. Validate the migrated flow.
    1. Rename targetVersion/migrated_flow.json.gz to flow.json.gz.
    2. Load flow.json.gz into your NiFi 2 instance to inspect the migrated flow.
    3. Open targetVersion/activity_log.json and review any manual-change-requests and manual-validation-requests.
    4. Apply the required manual modifications to the flow.

Migrating a process group using migrate-all with flow.json.gz as input

For complex flows, you can migrate one process group and on one stage at a time. See below for an example workflow.

  1. Migrate the TCP Listener process group using the following command. 
    bin/migration.sh nifi migrate-all \
-i /etc/migration-tool-input/flow.json.gz \
-od /etc/migration-tool-output -pgid b41940d7-0194-1000-42fc-458834630567 \
--sourceCompatibleOutput
  2. Use the output sourceVersion/migrated_flow.json.gz (the result of the previous command moved to the input folder) as the input for migrating the Elastic process group. 
    bin/migration.sh nifi migrate-all \
-i /etc/migration-tool-input/flow.json.gz \
-od /etc/migration-tool-output -pgid b42881c7-0194-1000-3cdf-1bd453a0ed0f \
--sourceCompatibleOutput
  3. Use the output sourceVersion/migrated_flow.json.gz (the result of the previous command moved to the input folder) as the input for migrating the root process group and completing the Stage 2 migration. 
    bin/migration.sh nifi migrate-all \
-i /etc/migration-tool-input/flow.json.gz \
-od /etc/migration-tool-output

    This produces a NiFi 2-compatible version of the flow.