Migrating variables using a directory with template.xml files as input

Learn how to use the Cloudera Flow Management Migration Tool to convert variables to parameters and parameter contexts for template files.

Example flows for migrating variables

The following NiFi flows are used to demonstrate both variable and component migration.

This example uses two template files:

  • TCP_Listener_template.xml

    The file consists of process group TCP Listener (ID: b41940d7-0194-1000-42fc-458834630567).

  • Elastic_flow_template.xml

    This file contains process group Elastic (ID: b42881c7-0194-1000-3cdf-1bd453a0ed0f).

TCP Listener Process Group

This process group contains the following simple flow:

The process group defines a variable called TCP Listener Port, which is referenced by the ListenTCPRecord processor.

Elastic Process Group

This process group also contains a simple flow:

The process group has a variable, Elasticsearch Index, referenced in the Index property of the PutElasticsearchJson processor.

Your goal is to migrate the templates used in Cloudera Flow Management 2.1.7.2000 powered by NiFi 1 to parameters used in Cloudera Flow Management 4.10.0 powered by NiFi 2.

The example guides you through variable migration and maintaining a clear activity log. While this example template is simple, the step-by-step approach shows how this method improves clarity for more complex migrations. In real-world scenarios, you have to define a migration strategy based on your flow’s structure. Individual template migration may not always be necessary.

Copy the TCP_Listener_template.xml and Elastic_template.xml files to the Migration Tool’s input folder (/etc/migration-tool-input).

  1. Run Stage 1 variable migration using the following command.
    bin/migration.sh nifi migrate-variables \
    -i /etc/migration-tool-input \
    -od /etc/migration-tool-output/variables \
    --sourceCompatibleOutput

    This generates a sourceVersion folder that contains the output files of the Stage 1 migration.

    variables
    └── sourceVersion
        ├── activity_log.json
        └── migrated_output
            ├── migrated_Elastic_template.json
            └── migrated_TCP_Listener_template.json
    activity_log.json
    • A new parameter context was created with a new parameter called TCP Listener Port, which replaces the corresponding variable.

    • The TCP Listener Port variable was removed.

    • A new parameter context was created with a new parameter called Elasticsearch Index, which replaces the corresponding variable of the same name.

    • The new parameter is referenced from the PutElasticsearchJson processor.

    • The Elasticsearch Index variable was removed.

    migrated_TCP_Listener_template.json
    • A modified NiFi 1 flow, which is not compatible with NiFi 2 yet.

    • It contains everything the original flow did, except the TCP Listener process group now references a parameter instead of the removed variable.

    • This is still a NiFi 1 flow definition, so do not load this file directly into NiFi 2. Use an empty NiFi 1 instance to review the modification made by the tool. The flow definition will appear unchanged, but variables will be replaced with parameters of the same name, which are now referenced by the processor instead.

    • Changes in the syntax: Variables were previously referenced using ${}, whereas parameters are now referenced using #{}.

      • NiFi 1 variables: ${variable_name}

      • NiFi 2 parameters: #{parameter_name}

    migrated_Elastic_template.json
    • A modified NiFi 1 flow definition, which is not compatible with NiFi 2 yet.

    • It contains everything the original flow definition did, but the Elastic process group now references a parameter instead of the removed variable.

  2. Validate the Stage 1 variable migration output for the migrated_TCP_Listener_template.json file.
    1. Check the new flow definition in a NiFi 1 instance to verify that the flow matches your expectations.
      1. Start your NiFi 1 instance.

      2. Create a new process group.

      3. Load sourceVersion/migrated_output/migrated_TCP_Listener_template.json to NiFi 1.

      4. Remove the ‘migrated_‘ prefix from the process group name.

      5. Confirm that variables were correctly converted to parameters.

    2. Review the activity_log.json file and check for any manual-change-requests or manual-validation-requests.
    3. If there are manual-change-requests or manual-validation-requests to handle, follow these steps:
      1. Make the modifications on the NiFi canvas, indicated by the manual-change-requests or manual-validation-requests in the sourceVersion/activity_log.json.

      2. Right-click on the TCPListener process group

      3. Select Download flow definition without external services.

      4. Save the file as migrated_TCP_Listener_template.json and overwrite the one in the sourceVersion/migrated_output folder.

    4. If no requests are present in the log:
      1. Make a backup of the original TCP_Listener_flow_definition.json.

      2. Right-click on the TCPListener process group

      3. Select Download flow definition without external services.

      4. Save file as migrated_TCP_Listener_template.json and overwrite the one in the sourceVersion/migrated_output folder.

    5. Proceed to running Stage 1 variable migration on the Elastic_template.json.

    At this stage, the flow definition no longer contains variables and uses parameters instead. If the flow meets your expectations, you can either run a full variable migration to validate your flow in NiFi 2 or proceed with migrating the components.

  3. Validate the Stage 1 variable migration output for the Elastic_template.json.
    1. Check the new flow definition in a NiFi 1 instance to verify that the flow matches your expectations.
      1. Start your NiFi 1 instance.

      2. Create a new process group.

      3. Load sourceVersion/migrated_output/migrated_Elastic_template.json to NiFi 1.

      4. Remove the ‘migrated_‘ prefix from the process group name.

      5. Confirm that variables were correctly converted to parameters.

    2. Review the activity_log.json file and check for any manual-change-requests or manual-validation-requests.
    3. If there are manual-change-requests or manual-validation-requests to handle, follow these steps:
      1. Make the modifications on the NiFi canvas, indicated by the manual-change-requests or manual-validation-requests in the sourceVersion/activity_log.json.

      2. Right-click on the Elastic process group

      3. Select Download flow definition without external services.

      4. Save the file as migrated_Elastic_template.json and overwrite the one in the sourceVersion/migrated_output folder.

    4. If no requests are present in the log:
      1. Right-click on the Elastic process group

      2. Select Download flow definition without external services.

      3. Save file as migrated_Elastic_template.json and overwrite the one in the sourceVersion/migrated_output folder.

    At this stage, both flow definitions no longer contain variables and use parameters instead. If the flow meets your expectations, you can either run a full variable migration to validate your flow in NiFi 2 or proceed with migrating the components.

  4. Run full variable migration (Stage 1 and 2) using the following command.
    bin/migration.sh nifi migrate-variables \
    -i /etc/migration-tool-input \
    -od /etc/migration-tool-output/variables

    This generates a sourceVersion and targetVersion folders that contain the output files of the migration.

    variables
    └── sourceVersion
    │    ├── activity_log.json
    │    └── migrated_output
    │        ├── migrated_Elastic_template.json
    │        └── migrated_TCP_Listener_template.json
    └──targetVersion
        ├── activity_log.json
        └── migrated_output
            ├── migrated_Elastic_template.json
            └── migrated_TCP_Listener_template.json
    

    The contents of the previously generated sourceVersion folder will be overwritten, but this is not a concern as the tool generates the same sourceVersion output as before.

    Additionally, a targetVersion directory is created, containing the output files of the Stage 2 part of the migration.

    migrated_Elastic_template.json
    • NiFi 2-compatible flow in terms of variables

    • You can load it into NiFi 2 and validate it

    migrated_TCP_Listener_template.json
    • NiFi 2-compatible flow in terms of variables

    • You can load it into NiFi 2 and validate it

    activity_log.json
    • List of all actions performed during this stage of the migration.

  5. Proceed with component migration using the input folder from step 3.