Migrating variables

Learn how to use the Cloudera Flow Management Migration Tool to convert variables to parameters and parameter contexts.

Example flow for migrating variables

The following NiFi flow is used to demonstrate both variable and component migration. It illustrates how variables are used within process groups and how they are referenced by individual components in the NiFi flow.

The flow consists of two process groups:

TCP Listener (ID: b41940d7-0194-1000-42fc-458834630567)
Elastic (ID: b42881c7-0194-1000-3cdf-1bd453a0ed0f)

At the root level, the flow is structured as follows:

TCP Listener Process Group

This process group contains the following simple flow:

The process group defines a variable called TCP Listener Port, which is referenced by the ListenTCPRecord processor.

Elastic Process Group

This process group also contains a simple flow:

The process group defines a variable called Elasticsearch Index. This variable is referenced in the PutElasticsearchJson processor’s "Index"property.

Your goal is to migrate the variables used in Cloudera Flow Management 2.1.7.1000 powered by NiFi 1 to parameters used in Cloudera Flow Management 4.0.0 powered by NiFi 2.

The example guides you through variable migration one process group at a time, simplifying the process and maintaining a clear activity log. While this example flow is simple, the step-by-step approach shows how this method improves clarity for more complex migrations. In real-world scenarios, you have to define a migration strategy based on your flow’s structure, individual process group migration may not always be necessary. Migrating a parent process group automatically applies the changes recursively to its descendants.

Stop all processors and disable all controller services in NiFi.
Stop NiFi.
Copy the flow.json.gz file from NiFi’s conf directory to the Migration Tool’s input folder (/etc/migration-tool-input).
Unzip the file to obtain flow.json.

Run Stage 1 variable migration on the TCP Listener process group, using the following command.
```
bin/migration.sh nifi migrate-variables \
-i /etc/migration-tool-input/flow.json \
-od /etc/migration-tool-output/variables \
-pgid <process_group_id> \
--stage STAGE_1
```
This generates a v1 folder that contains the output files of the Stage 1 migration.
```
variables
└── v1
    ├── activity_log.json
    └── migrated_flow.json
```
activity_log.json
- Log of all actions performed during this stage of the migration.
- The following were changes made during the template migration:
  
  A new parameter context was created with a new parameter called TCP Listener Port, which replaces the corresponding variable.
  
  The TCP Listener Port variable was removed.
migrated_flow.json
- A modified NiFi 1 flow, which is not compatible with NiFi 2 yet.
- It contains everything the original flow did, except the TCP Listener process group now references a parameter instead of the removed variable.
Validate the Stage 1 variable migration output for the TCP Listener process group.
1. Load the migrated_flow.json into a NiFi 1 instance and check the flow.
  1. Ensure NiFi is not running.
  2. Go to NiFi's conf directory and back up the flow.json.gz and flow.xml.gz files.
  3. Delete the original flow.json.gz and flow.xml.gz files.
  4. Rename the migrated_flow.json to flow.json and compress it using gzip to create flow.json.gz.
  5. Copy the newly created flow.json.gz file to NiFi's conf directory.
  6. Start NiFi and check the flow.
2. Review the activity_log.json file and check for any manual-change-requests or manual-inspection-requests. If none are present, proceed to running Stage 1 variable migration on the Elastic process group.
  1. If manual changes are necessary, load the migrated_flow.json to NiFi, update and validate it.
  2. Once the flow is validated and meets expectations, continue with the next step using the new flow.json.gz file (unzipped to produce flow.json).
Run Stage 1 variable migration on the Elastic process group.
1. Move migrated_flow.json from Step 2 into the input folder (/etc/migration-tool-input) and rename it to flow.json for clarity.
2. Make a backup of the output folder (/etc/migration-tool-output/variables) before running the next migration step.
  
  note
  This step is crucial because the same output folder is used as in the previous migration, and its contents will be overwritten. To retain a record of changes made at each stage, it is better to create a backup.
3. Run Stage 1 variable migration on the Elastic process group using the following command.
```
bin/migration.sh nifi migrate-variables \
-i /etc/migration-tool-input/flow.json \
-od /etc/migration-tool-output/variables \
-pgid b42881c7-0194-1000-3cdf-1bd453a0ed0f \
--stage STAGE_1
```
  This generates a v1 folder that contains the two output files of the Stage 1 migration.
  activity_log.json
  Log of all actions performed during this stage of the migration.
  
  The following were changes made during the template migration:
  
  A new parameter context was created with a new parameter called Elasticsearch Index, which replaces the corresponding variable of the same name.
  
  The new parameter is referenced from the PutElasticsearchJson processor.
  
  The Elasticsearch Index variable was removed.
  migrated_flow.json
  A modified NiFi 1 flow, which is not compatible with NiFi 2 yet.
  
  It contains everything the original flow did, but the Elastic process group now references a parameter instead of the removed variable.
Validate the Stage 1 variable migration output for the Elastic process group.
1. Load the migrated_flow.json into a NiFi 1 instance and check the flow.
2. Review the activity_log.json file and check for any manual-change-requests ormanual-inspection-requests. If none are present, proceed to the next step.
3. If manual changes are necessary, update the migrated_flow.json on the NiFi canvas after loading it. Once the flow is validated and meets expectations, continue with the next step.
At this stage, both process groups no longer contain variables and use parameters instead. If the flow meets your expectations, you can either run a full variable migration to validate your flow in NiFi 2 or proceed with migrating the components.
Run full variable migration (Stage 1 and 2) by using the following command.
```
bin/migration.sh nifi migrate-variables \
-i /etc/migration-tool-input/flow.json \
-od /etc/migration-tool-output/variables
```
The contents of the previously generated v1 folder will be overwritten, but this is not a concern as the tool generates the same v1 output as before.

Additionally, a v2 directory is created, containing the output files of the Stage 2 part of the migration.
migrated_flow.json
- NiFi 2-compatible flow in terms of variables
- You can load it into NiFi 2 and validate it
activity_log.json
- List of all actions performed during this stage of the migration.
Stage 2 migration allows you to validate the result of the variable migration process in NiFi 2. However, component migration should not be performed on the v2/migrated_flow.json because the input of component migration must be a NiFi 1 flow.
Proceed with component migration using the migrated_flow.json output from step 3.