Cloudera Docs

Migrating variables using template.xml as input

Learn how to use the Cloudera Flow Management Migration Tool to convert variables to parameters and parameter contexts for template files.

Example flows for migrating variables

The following NiFi flow is used to demonstrate both variable and component migration. It shows how variables are used within process groups and how they are referenced by individual components.

Flow definition file:

  • TCP_Listener_template.xml

    The file consists of process group TCP Listener (ID: b41940d7-0194-1000-42fc-458834630567).

TCP Listener Process Group

This process group contains the following simple flow:

The process group defines a variable called TCP Listener Port, which is referenced by the ListenTCPRecord processor.

The example guides you through variable migration and maintaining a clear activity log. While this example template is simple, the step-by-step approach shows how this method improves clarity for more complex migrations. In real-world scenarios, you have to define a migration strategy based on your flow’s structure. Individual template migration may not always be necessary.

Copy the TCP_Listener_template.xml file to the Migration Tool’s input folder (/etc/migration-tool-input).

  1. Run Stage 1 variable migration on the TCP_Listener_template.xml file, using the following command. 
    bin/migration.sh nifi migrate-variables \
-i /etc/migration-tool-input/TCP_Listener_template.xml \
-od /etc/migration-tool-output/variables \
--sourceCompatibleOutput

    This generates a sourceVersion folder that contains the output files of the Stage 1 migration.

    variables
└── sourceVersion
    ├── activity_log.json
    └── migrated_output
        └── migrated_TCP_Listener_template.json
    activity_log.json

    • Log of all actions performed during this stage of the migration.

    • The following were changes made during the template migration:

      • A new parameter context was created with a new parameter called TCP Listener Port, which replaces the corresponding variable.

      • The TCP Listener Port variable was removed.

    migrated_TCP_Listener_template.json

    • A modified NiFi 1 flow, which is not compatible with NiFi 2 yet.

    • It contains everything the original flow did, except the TCP Listener process group now references a parameter instead of the removed variable.

    • This is still a NiFi 1 flow definition, so do not load this file directly into NiFi 2. Use an empty NiFi 1 instance to review the modification made by the tool. The flow definition will appear unchanged, but variables will be replaced with parameters of the same name, which are now referenced by the processor instead.

    • Changes in syntax: Variables were previously referenced using ${}, whereas parameters are now referenced using #{}.

      • NiFi 1 variables: ${variable_name}

      • NiFi 2 parameters: #{parameter_name}

  2. Validate the Stage 1 variable migration output for the migrated_TCP_Listener_template.json file.
    1. Check the new flow definition in a NiFi 1 instance to verify that the flow matches your expectations.

      1. Start your NiFi 1 instance.

      2. Create a new process group.

      3. Load sourceVersion/migrated_output/migrated_TCP_Listener_template.json to NiFi 1.

      4. Remove the ‘migrated_‘ prefix from the process group name.

      5. Confirm that variables were correctly converted to parameters.

    2. Review the activity_log.json file and check for any manual-change-requests or manual-validation-requests.
    3. If there are manual-change-requests or manual-validation-requests to handle, follow these steps:

      1. Make the modifications on the NiFi canvas, indicated by the manual-change-requests or manual-validation-requests in the sourceVersion/activity_log.json.

      2. Right-click on the TCPListener process group

      3. Select Download flow definition without external services.

      4. Save the file as migrated_TCP_Listener_template.json and overwrite the one in the sourceVersion/migrated_output folder.

    4. If no requests are present in the log, proceed to the next step.

    At this stage, the flow definition no longer contains variables and uses parameters instead. If the flow meets your expectations, you can either run a full variable migration to validate your flow in NiFi 2 or proceed with migrating the components.

  3. Run full variable migration (Stage 1 and 2) on the TCP_Listener_template.
    1. Copy the migrated_TCP_Listener_template.json file from Step 2 into the input folder (/etc/migration-tool-input) and rename it to TCP_Listener_template.json for clarity.
    2. Make a backup of the output folder (/etc/migration-tool-output/variables) before running the next migration step.
    3. Run the following command. 
      bin/migration.sh nifi migrate-variables \
-i /etc/migration-tool-input/TCP_Listener_template.json \
-od /etc/migration-tool-output/variables

    The contents of the previously generated sourceVersion folder will be overwritten, but this is not a concern as the tool generates the same sourceVersion output as before.

    Additionally, a targetVersion directory is created, containing the output files of the Stage 2 part of the migration.

    migrated_TCP_Listener_template.json

    • NiFi 2-compatible flow in terms of variables

    • You can load it into NiFi 2 and validate it

    activity_log.json

    • List of all actions performed during this stage of the migration.

  4. Proceed with component migration using the input folder from step 3.