Migrating variables using a directory with flow definition JSON files as input

Learn how to use the Cloudera Flow Management Migration Tool to convert variables to parameters and parameter contexts from a directory of flow definition JSON files.

Example flows for migrating variables

The input directory contains the following flow definition files:

  • TCP_Listener_flow_definition.json

    This file contains the TCP Listener process group (ID: b41940d7-0194-1000-42fc-458834630567)

  • Elastic_flow_definition.json

    This file contains process group Elastic (ID: b42881c7-0194-1000-3cdf-1bd453a0ed0f)

TCP Listener Process Group

This process group contains the following simple flow:

The process group defines a variable called TCP Listener Port, which is referenced by the ListenTCPRecord processor.

Elastic Process Group

This process group also contains a simple flow:

The process group defines a variable called Elasticsearch Index, which is referenced in the PutElasticsearchJson processor’s Index property.

Your goal is to migrate the variables used in Cloudera Flow Management 2.1.7.2000 powered by NiFi 1 to parameters used in Cloudera Flow Management 4.10.0 powered by NiFi 2.

The example guides you through the variable migration process and shows how to maintain a clear activity log. Although the example uses simple flow definitions, the step-by-step approach can be applied to more complex scenarios. In real-world scenarios, Cloudera recommends defining a migration strategy based on your flow’s structure.

Copy the TCP_Listener_flow_definition.json and Elastic_flow_definition.json files to the Migration Tool’s input folder (/etc/migration-tool-input).

  1. Run Stage 1 variable migration on the MigrationTool input directory (/etc/migration-tool-input) using the following command.
    bin/migration.sh nifi migrate-variables \
    -i /etc/migration-tool-input \
    -od /etc/migration-tool-output/variables \
    --sourceCompatibleOutput

    This generates a sourceVersion folder that contains the output files of the Stage 1 migration.

    variables
    └── sourceVersion
        ├── activity_log.json
        └── migrated_output
            ├── migrated_Elastic.json
            └── migrated_TCP_Listener.json
    activity_log.json
    • Log of all actions performed during this stage of the migration.

    • The following changes were made during the variable migration:

      • A new parameter context was created with a new parameter called TCP Listener Port, which replaces the corresponding variable.
      • The TCP Listener Port variable was removed.
      • A new parameter context was created with a new parameter called Elasticsearch Index, which replaces the corresponding variable of the same name.
      • The new parameter is referenced from the PutElasticsearchJson processor.
      • The Elasticsearch Index variable was removed.
    migrated_TCP_Listener.json
    • A modified flow definition, which is not compatible with NiFi 2 yet.

    • It contains everything the original flow definition did, except the TCP Listener process group now references a parameter instead of the removed variable.

    • This is still a NiFi 1 flow definition, so do not load this file directly into NiFi 2. Use an empty NiFi 1 instance to review the modification made by the tool. The flow definition will appear unchanged, but variables will be replaced with parameters of the same name, which are now referenced by the processor instead.

    • Changes in the syntax: Variables were previously referenced using ${}, whereas parameters are now referenced using #{}.

      • NiFi 1 variables: ${variable_name}
      • NiFi 2 parameters: #{parameter_name}
    migrated_Elastic.json
    • A modified NiFi 1 flow definition, which is not compatible with NiFi 2 yet.

    • It contains everything the original flow definition did, but the Elastic process group now references a parameter instead of the removed variable.

  2. Validate the Stage 1 variable migration output for TCP_Listener_flow_definition.json.
    1. Check the new flow definition in a NiFi 1 instance to verify that the flow matches your expectations.
      1. Start your NiFi instance.

      2. Create a new process group.

      3. Load sourceVersion/migrated_output/migrated_TCP_Listener.json to NiFi 1.

      4. Remove the ‘migrated_‘ prefix from the process group name.

      5. Confirm that variables were correctly converted to parameters.

    2. Review the activity_log.json file and check for any manual-change-requests or manual-validation-requests.
    3. If there are manual-change-requests or manual-validation-requests to handle, follow these steps:
      1. Make the modifications on the NiFi canvas, indicated by the manual-change-requests or manual-validation-requests in the sourceVersion/activity_log.json.

      2. Right-click on the TCPListener process group.

      3. Select Download flow definition without external services.

      4. Save the file as migrated_TCP_Listener.json and overwrite the one in the sourceVersion/migrated_output folder.

    4. If no requests are present in the log,
      1. Make a backup of the original TCP_Listener_flow_definition.json file.

      2. Right-click on the TCPListener process group.

      3. Select Download flow definition without external services.

      4. Save the file as migrated_TCP_Listener.json and overwrite the one in the sourceVersion/migrated_output folder.

    5. Proceed to running Stage 1 variable migration on Elastic_flow_definition.json.
  3. Validate the Stage 1 variable migration output for Elastic_flow_definition.json.
    1. Check the new flow definition in a NiFi 1 instance to verify that the flow matches your expectations.
      1. Start your NiFi instance.

      2. Create a new process group.

      3. Load sourceVersion/migrated_output/migrated_Elastic.json to NiFi 1.

      4. Remove the ‘migrated_‘ prefix from the process group name.

      5. Confirm that variables were correctly converted to parameters.

    2. Review the activity_log.json file and check for any manual-change-requests or manual-validation-requests.
    3. If there are manual-change-requests or manual-validation-requests to handle, follow these steps:
      1. Make the modifications on the NiFi canvas, indicated by the manual-change-requests or manual-validation-requests in the sourceVersion/activity_log.json.

      2. Right-click on the Elastic process group.

      3. Select Download flow definition without external services.

      4. Save the file as migrated_Elastic.json and overwrite the one in the sourceVersion/migrated_output folder.

    4. If no requests are present in the log,
      1. Right-click on the Elastic process group.

      2. Select Download flow definition without external services.

      3. Save the file as migrated_Elastic.json and overwrite the one in the sourceVersion/migrated_output folder.

    At this point, both flow definitions have been updated to replace variables with parameters. If the result aligns with your expectations, you can either perform a full variable migration to validate the flow in NiFi 2 or continue with migrating the components.

  4. Run full variable migration (Stage 1 and 2) using the following command.
    bin/migration.sh nifi migrate-variables \
    -i /etc/migration-tool-input \
    -od /etc/migration-tool-output/variables

    The contents of the previously generated sourceVersion folder will be overwritten, but this is not a concern as the tool generates the same sourceVersion output as before.

    Additionally, a targetVersion directory is created, containing the output files of the Stage 2 part of the migration.

    variables
    └── sourceVersion
    │    ├── activity_log.json
    │    └── migrated_output
    │        ├── migrated_Elastic.json
    │        └── migrated_TCP_Listener.json
    └──targetVersion
        ├── activity_log.json
        └── migrated_output
            ├── migrated_Elastic.json
            └── migrated_TCP_Listener.json
    
    migrated_Elastic.json
    • NiFi 2-compatible flow definition in terms of variables

    • You can load it into NiFi 2 and validate it

    migrated_TCP_Listener.json
    • NiFi 2-compatible flow definition in terms of variables

    • You can load it into NiFi 2 and validate it

    activity_log.json
    • List of all actions performed during this stage of the migration

  5. Proceed with component migration using the input folder from step 3.