Migrating components

Learn how to use the Cloudera Flow Management Migration Tool to migrate components to parameters and parameter contexts.

Example flow for migrating components🔗

The following NiFi flow is used to demonstrate both variable and component migration.

It consists of two process groups:

TCP Listener (ID: b41940d7-0194-1000-42fc-458834630567)
Elastic (ID: b42881c7-0194-1000-3cdf-1bd453a0ed0f)

At the root level, the flow is structured as follows:

TCP Listener Process Group

This process group contains the following simple flow:

The process group defines a variable called TCP Listener Port, which is referenced by the ListenTCPRecord processor.

Elastic Process Group

This process group also contains a simple flow:

The process group defines a variable called Elasticsearch Index. This variable is referenced in the PutElasticsearchJson processor’s "Index" property.

Your goal is to migrate the components used in Cloudera Flow Management 2.1.7.1000 powered by NiFi 1 to NiFi 2-compatible components used in Cloudera Flow Management 4.0.0 powered by NiFi 2.

The example guides you through component migration one process group at a time, simplifying the process and maintaining a clear activity log. While this example flow is simple, the step-by-step approach shows how this method improves clarity for more complex migrations. In real-world scenarios, you have to define a migration strategy based on your flow’s structure, individual process group migration may not always be necessary. Migrating a parent process group automatically applies the changes recursively to its descendants.

Move the migrated_flow.json file, the output of variable migration from Step 3, into the input folder (/etc/migration-tool-input) and rename it to flow.json for clarity.

This is a NiFi-1 compatible flow that no longer contains variables.
Run Stage 1 component migration on the TCP Listener process group using the following command.
```
bin/migration.sh nifi migrate-components \
-i /etc/migration-tool-input/flow.json \
-od /etc/migration-tool-output/components \
-pgid b41940d7-0194-1000-42fc-458834630567 \
--stage STAGE_1
```
This generates a v1 folder that contains the output files of the Stage 1 migration.
```
components
└── v1
    ├── activity_log.json
    └── migrated_flow.json
```
activity_log.json
- The log describes all the actions that were performed for this stage of the process group migration.
- Log of all actions performed during this stage of the process group migration.
migrated_flow.json
- A modified NiFi 1 flow, which is not compatible with NiFi 2 yet.
- It contains everything the original flow did, except the TCP Listener process group was modified as described in the activity log.
Validate the Stage 1 component migration output for the TCP Listener process group.
1. Load the migrated_flow.json into a NiFi 1 instance and check the flow.
2. Review the activity_log.json file and check for any manual-change-requests ormanual-inspection-requests. If none are present, proceed to the next step.
  In this example, you can see the following information in the activity log:
```
{
	"sequence" : 2,
	"type" : "change-info",
	"subject" : "b41966ad-0194-1000-a08d-a92489457356",
	"message" : "Component [org.apache.nifi.processors.standard.ListenTCPRecord] has been deprecated (NIFI-13509)",
	"context" : {
  	"rule" : "36227b60-75f0-40dc-8caf-a2ec577aa54c"
}
```
  Search for the value of the “subject” element in NiFi’s search box.
  You are directed to the ListenTCPRecord processor. No manual modifications are needed at this stage. However, it is important to note that the ListenTCPRecord processor is deprecated and is not available in NiFi 2. Once the full component migration (Stage 1 and Stage 2) is complete, instructions will be provided on how to handle this deprecation.
3. If manual changes are necessary, update the migrated_flow.json on the NiFi canvas after loading it. Once the flow is validated and meets expectations, continue with the next step.
Run Stage 1 component migration on the Elastic process group.
1. Move migrated_flow.json from Step 2 into the input folder (/etc/migration-tool-input) and rename it to flow.json for clarity.
2. Make a backup of the output folder (/etc/migration-tool-output/components) before running the next migration step.
  
  note
  
  This step is crucial because the same output folder is used as in the previous migration, and its contents will be overwritten. To retain a record of changes made at each stage, it is better to create a backup.
3. Run Stage 1 component migration on the Elastic process group using the following command.
```
bin/migration.sh nifi migrate-components \
-i /etc/migration-tool-input/flow.json \
-od /etc/migration-tool-output/components \
-pgid b42881c7-0194-1000-3cdf-1bd453a0ed0f \
--stage STAGE_1
```
  This generates a v1 folder that contains the two output files of the Stage 1 migration.
  activity_log.json
  Log of all actions performed during this stage of the migration.
  migrated_flow.json
  A modified NiFi 1 flow, which is not compatible with NiFi 2 yet.
  
  It contains everything the original flow did, except the Elastic process group was modified with the actions described in the activity log.
Validate the Stage 1 component migration output for the Elastic process group.
1. Load the migrated_flow.json into a NiFi 1 instance and check the flow.
2. Review the activity_log.json file.
  It contains a change-info entry that informs you of changes to a NiFi 2 processor, identified by the subject ID.
```
{
	"sequence" : 2,
	"type" : "change-info",
	"subject" : "b428aad6-0194-1000-d73d-9f2332a59f04",
	"message" : "Property [Max JSON Field String Length] has been added (NIFI-12343); Property [put-es-json-not_found-is-error] has been renamed to [put-es-not_found-is-error] (NIFI-12255); Property [put-es-json-error-documents] has been removed (NIFI-12255); Relationships [success] has been renamed to [original]; Relationship [successful] has been added (NIFI-12255);",
	"context" : {
  	"rule" : "6501a693-b524-4d1a-b11e-69747c2651e8"
}
```
  To identify the processor affected by these changes, search for the value of the “subject” element in NiFi’s search box.
  
  You are directed to the PutElasticsearchJson processor. No manual modifications are needed at this stage. However, it is important to note that changes will be applied to the PutElasticsearchJson processor during Stage 2 of the migration. Once the full component migration (Stage 1 and Stage 2) is complete, instructions will be provided on how to handle this change.
At this stage, you have completed Stage 1 of both variable and component migration for the process groups in your flow. After reviewing the logs, you confirmed that no manual changes were needed. You can proceed with a full component migration using the migrated_flow.json from Step 4.
Run full component migration (Stage 1 and 2).
1. Move migrated_flow.json from Step 4 into the input folder (/etc/migration-tool-input) and rename it to flow.json for clarity.
2. Make a backup of the output folder (/etc/migration-tool-output/components) before running the next migration step.
  
  note
  
  This step is crucial because the same output folder is used as in the previous migration, and its contents will be overwritten. To retain a record of changes made at each stage, it is better to create a backup.
3. Run the full component migration using the following command.
```
bin/migration.sh nifi migrate-components \
-i /etc/migration-tool-input/flow.json \
-od /etc/migration-tool-output/components
```
  This generates the following output:
```
components
├── v1
│   ├── activity_log.json
│   └── migrated_flow.json
└── v2
    ├── activity_log.json
    └── migrated_flow.json
```
  The contents of the previously generated v1 folder will be overwritten. The contents represent the NiFi 1-compatible version of the migrated flow. Since the root process group only contained the two process groups on which you already performed Stage 1 component migration, the activity log will only include the same entries as those in Steps 2 and 4.
  
  Additionally, a v2 directory is created, containing the output files of the Stage 2 part of the migration:
  migrated_flow.json
  NiFi 2-compatible flow in terms of variables
  
  You can load it into NiFi 2 and validate it
  activity_log.json
  List of all actions performed during this stage of the migration, and any manual steps that you need to perform.
Validate the Stage 2 component migration output.
1. Load the v2/migrated_flow.json into a NiFi 2 instance and check the flow.
2. Review the v2/activity_log.json file.
  In this example, you can see the following manual-change-request entry.
```
{
	"sequence" : 3,
	"type" : "manual-change-request",
	"subject" : "b428aad6-0194-1000-d73d-9f2332a59f04",
	"message" : "New relationship [original] has been added, it has to be connected to a downstream processor or terminated.",
	"context" : {
  	"rule" : "6501a693-b524-4d1a-b11e-69747c2651e8"
}
```
  To identify the processor affected by these changes, search for the value of the “subject” element in NiFi’s search box. It refers to the PutElasticsearchJson processor. You will find an unbound "original" relationship that needs to be connected to a downstream processor or terminated. Make the necessary modifications manually. Once completed, this change request is resolved.
  
  You can see another manual-change-request entry in the Activity Log.
```
{
	"sequence" : 6,
	"type" : "manual-change-request",
	"subject" : "b41966ad-0194-1000-a08d-a92489457356",
	"message" : "The component has been deprecated. It is suggested to replace it with a combination of [org.apache.nifi.processors.standard.ListenTCP] and [org.apache.nifi.processors.standard.ConvertRecord] processors",
	"context" : {
  	"rule" : "36227b60-75f0-40dc-8caf-a2ec577aa54c"
}
```
  To identify the processor affected by these changes, search for the value of the “subject” element in NiFi’s search box. It refers to the ListenTCPRecord processor. Open the TCP Listener process group. You can see that the processor is marked with dashed borders and its version number still shows that of the NiFi 1 instance. It means that it is a "ghost processor", not available in NiFi 2. The log message suggests replacing it with a combination of ListenTCP and ConvertRecord processors.
  
  Instantiate and configure the ListenTCP and ConvertRecord processors, connect them appropriately, and then remove the ghost processor.
  
  Besides these changes there are no further manual-change-requests or manual-inspection-requests in the activity log.
Wrap up the migration.
1. Review the completed flow to ensure it meets your use cases and that all necessary changes have been made.
2. Enable the controller services associated with the flow in the NiFi 2 instance.
3. Once the controller services are enabled, start the processors in your migrated NiFi 2 flow.