Migrating variables using flow definition JSON as input

Learn how to use the Cloudera Flow Management Migration Tool to convert variables to parameters and parameter contexts from flow definition JSON files.

Example flows for migrating variables

The following NiFi flow demonstrates both variable and component migration. It shows how variables are used within process groups and how they are referenced by individual components.

Flow definition file:

TCP_Listener_flow_definition.json
This file contains the TCP Listener process group (ID: b41940d7-0194-1000-42fc-458834630567)

TCP Listener Process Group

This process group contains the following simple flow:

The process group defines a variable called TCP Listener Port, which is referenced by the ListenTCPRecord processor.

The example guides you through the variable migration process and shows how to maintain a clear activity log. Although the example uses a simple flow, the step-by-step approach can be applied to more complex scenarios. In real-world scenarios, Cloudera recommends defining a migration strategy based on the structure of each flow. In some cases, individual flow definition migration may not be necessary.

Copy the TCP_Listener_flow_definition.json file to the Migration Tool’s input folder (/etc/migration-tool-input).

Run Stage 1 variable migration on TCP_Listener_flow_definition.json, using the following command.
```
bin/migration.sh nifi migrate-variables \
-i /etc/migration-tool-input/TCP_Listener_flow_definition.json \
-od /etc/migration-tool-output/variables \
--sourceCompatibleOutput
```
This generates a sourceVersion folder that contains the output files of the Stage 1 migration.
```
variables
└── sourceVersion
    ├── activity_log.json
    └── migrated_output
        └── migrated_TCP_Listener.json
```
activity_log.json
- Log of all actions performed during this stage of the migration.
- The following changes were made during the variable migration:
  
  A new parameter context was created with a new parameter called TCP Listener Port, which replaces the corresponding variable.
  
  The TCP Listener Port variable was removed.
migrated_TCP_Listener.json
- A modified flow definition, which is not compatible with NiFi 2 yet.
- It contains everything the original flow definition did, except the TCP Listener process group now references a parameter instead of the removed variable.
- This is still a NiFi 1 flow definition, so do not load this file directly into NiFi 2. Use an empty NiFi 1 instance to review the modification made by the tool. The flow definition will appear unchanged, but variables will be replaced with parameters of the same name, which are now referenced by the processor instead.
- Changes in the syntax: Variables were previously referenced using ${}, whereas parameters are now referenced using #{}.
  
  NiFi 1 variables: ${variable_name}
  
  NiFi 2 parameters: #{parameter_name}
note

The file name is composed of the migrated prefix and the name of the top-level process group and not the original file name.
Validate the Stage 1 variable migration output for the TCP_Listener_flow_definition.json.
1. Check the new flow definition in a NiFi 1 instance to verify that the flow matches your expectations.
  1. Start your NiFi instance.
  2. Create a new process group.
  3. Load sourceVersion/migrated_output/migrated_TCP_Listener.json to NiFi 1.
  4. Remove the ‘migrated_‘ prefix from the process group name.
  5. Confirm that variables were correctly converted to parameters.
2. Review the activity_log.json file and check for any manual-change-requests or manual-validation-requests.
3. If there are manual-change-requests or manual-validation-requests to handle, follow these steps:
  1. Make the modifications on the NiFi canvas, indicated by the manual-change-requests or manual-validation-requests in the sourceVersion/activity_log.json.
  2. Right-click on the TCPListener process group.
  3. Select Download flow definition without external services.
  Save the file as migrated_TCP_Listener.json and overwrite the one in the sourceVersion/migrated_output folder.
4. If no requests are present in the log, proceed to the next step.
At this stage, the flow definition no longer contains variables and uses parameters instead. If the flow meets your expectations, you can either run a full variable migration to validate your flow in NiFi 2 or proceed with migrating the components.
Run full variable migration (Stage 1 and 2) on TCP_Listener_flow_definition.json.
1. Move migrated_TCP_Listener.json from Step 2 into the input folder (/etc/migration-tool-input) and rename it to TCP_Listener_flow_definition.json for clarity.
2. Make a backup of the output folder (/etc/migration-tool-output/variables) before running the next migration step.
  
  note
  
  This step is crucial because the same output folder is used as in the previous migration, and its contents will be overwritten. To retain a record of changes made at each stage, it is better to create a backup.
```
bin/migration.sh nifi migrate-variables \
-i /etc/migration-tool-input/TCP_Listener_flow_definition.json \
-od /etc/migration-tool-output/variables
```
The contents of the previously generated sourceVersion folder will be overwritten, but this is not a concern as the tool generates the same sourceVersion output as before.

Additionally, a targetVersion directory is created, containing the output files of the Stage 2 part of the migration.
migrated_TCP_Listener.json
- NiFi 2-compatible flow definition in terms of variables
- You can load it into NiFi 2 and validate it
activity_log.json
- List of all actions performed during this stage of the migration
Proceed with component migration using the input folder from step 3.