Cloudera Docs

Pre-upgrade check for NiFi

Starting with Cloudera Flow Management 2.1.7.1000 (Service Pack 1), you can run a pre-upgrade check to help you prepare for the transition to NiFi 2.

The pre-upgrade check script is specifically designed to highlight necessary steps and make you aware of any breaking changes that will not be automatically handled by Cloudera’s NiFi Migration Tooling. Its primary purpose is to ensure a smooth upgrade by informing you of potential issues before you start your NiFi 2 upgrade.

Pre-upgrade check for NiFi in Cloudera Manager

Cloudera Manager allows you to easily run a pre-upgrade check for NiFi. This process helps identify any potential issues before upgrading. Follow the below steps to perform the pre-upgrade check directly from Cloudera Manager.

  1. From Cloudera Manager, click the Clusters tab in the left-hand navigation.
  2. Click NiFi in the list of services to display the NiFi service page.
  3. Click the Actions drop-down next to the service name and select the Check NiFi Before Upgrade To NiFi 2.0 option.
  4. Click Check NiFi Before Upgrade To NiFi 2.0 again.

    The script runs across all NiFi nodes, providing a comprehensive list of items to address before starting an upgrade to NiFi 2.

  5. Review this list thoroughly to ensure a smooth upgrade process.

Here is an example of the output:

Tue Sep 24 19:37:22 UTC 2024
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
Using -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck as conf dir
Using scripts/service-commands/pre_upgrade_check.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck
CMF_CONF_DIR=
Script pre_upgrade_check.sh started work!
CMF_PACKAGE_DIR = /opt/cloudera/cm-agent/service
NIFI_WORKING_DIRECTORY = /var/lib/nifi
Path to csv file = /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck/scripts/common-scripts/resources/CFM4-Deprecated-components.csv
Path to flow.json.gz = /var/lib/nifi/flow.json.gz
Path to bootstrap.conf = /var/lib/nifi/config_backup/bootstrap.conf
Custom java home = 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Check Before Upgrade to CFM with NiFi 2.0 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Using Python 3 or newer version (3.8.12)
Script pre_upgrade_check.py started work!
Passed arguments:
Path to json flow file = /var/lib/nifi/flow.json.gz
Path to deprecated components CSV file = /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck/scripts/common-scripts/resources/CFM4-Deprecated-components.csv
Path to components with kerberos properties CSV file = /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck/scripts/common-scripts/resources/NiFi-1.x-components-with-legacy-Kerberos-properties.csv
Path to bootstrap conf = /var/lib/nifi/config_backup/bootstrap.conf
********************
Validating arguments:
You are running pre upgrade check before upgrade to NiFi 2.0 for NiFi managed by Cloudera Manager.
Path to flow.json.gz file is valid!
Path to deprecated components CSV file is valid!
Path to components with kerberos properties CSV file is valid!
Path to bootstrap.conf file is valid!
********************
Parsing bootstrap.conf:
********************

Reading and parsing /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck/scripts/common-scripts/resources/CFM4-Deprecated-components.csv:
********************

Reading and parsing /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck/scripts/common-scripts/resources/NiFi-1.x-components-with-legacy-Kerberos-properties.csv:
********************
Reading and parsing flow.json.gz:
********************

********************
******************** CHECKING THAT UPGRADE PATH IS SUPPORTED
******************** (It is required to be on CFM 2.1.7+ before upgrading to CFM 4.0.0+)
********************
Get current CFM version from bootstrap.conf and check it's valid to upgrade to CFM with NiFi 2.0 (CFM-4.0.0.0 or newer)
Property value for 'working.dir' = /opt/cloudera/parcels/CFM-2.1.7.1000-42/NIFI
Current CFM version could be upgraded to CFM with NiFi 2.0 (CFM-4.0.0.0 or newer).
********************

********************
******************** CHECKING THAT JAVA 21 IS USED
******************** (Java 21 is the minimum Java version required for NiFi and NiFi Registry 2.0)
********************
Get 'java' property value from bootstrap.conf
Property value for 'java' = java
Getting java home from JAVA_HOME in env variables.
Executing command: /usr/java/jdk1.8.0_232-cloudera/bin/java -version 2>&1 | awk -F[\"\.] -v OFS=. 'NR==1{print $2}'
Command output: 1

WARN: Your current Java version is too low for upgrading to NiFi 2.0 (CFM-4.0.0.0). Please install Java version 21 on each host with NiFi and follow the upgrade documentation for NiFi 2.0 (CFM-4.0.0.0). NOTE: Do not change the JDK for the current NiFi. It should only be changed during the upgrade to NiFi 2.0 (CFM-4.0.0.0).

********************
******************** CHECKING FOR REMOVED COMPONENTS
******************** Some components have been removed in NiFi 2.0.
******************** You should switch to the recommended option before upgrading.
******************** If not, those components will be showing as ghost components in NiFi.
********************
ERROR: Component identifier: '8d74d462-7c52-35d1-ad1b-665696503ea2' of type 'org.apache.nifi.processors.gettcp.GetTCP', group 'org.apache.nifi' and artifact 'nifi-tcp-nar' is no longer supported.
ERROR: Component identifier: 'f78dba93-e6ec-38d7-b5e4-fdbcd09fd831' of type 'org.apache.nifi.processors.pulsar.pubsub.PublishPulsar', group 'io.streamnative.connectors' and artifact 'nifi-pulsar-nar' is removed, but it is planned to be reintroduced later. Please check the available CFM-4.x releases.
ERROR: Component identifier: '2e2edf20-3efc-388a-ac96-943b717e5252' of type 'org.apache.nifi.processors.twitter.GetTwitter', group 'org.apache.nifi' and artifact 'nifi-social-media-nar' is no longer supported. Please consider using 'ConsumeTwitter' instead.
ERROR: Component identifier: 'a0766b6b-140e-31a8-a1fb-80f10459f870' of type 'org.apache.nifi.kerberos.KeytabCredentialsService', group 'org.apache.nifi' and artifact 'nifi-kerberos-credentials-service-nar' is no longer supported. Please consider using 'KerberosKeytabUserService' instead.
********************

********************
******************** CHECKING FOR REMOVED SCRIPTING LANGUAGES
******************** Some scripting languages have been removed in NiFi 2.0.
******************** You should rewrite your scripted components using Groovy as language or leverage the new Python API in NiFi 2.0.
******************** If not, those components will be showing as invalid in NiFi 2.0.
********************
ERROR: Component identifier: 'e0334fb0-738b-3000-bdfc-fee3e59f539d' of type 'org.apache.nifi.processors.script.InvokeScriptedProcessor', group 'org.apache.nifi' and artifact 'nifi-scripting-nar' has deprecated 'Script Engine' value 'ruby'.
ERROR: Component identifier: 'e9d304ef-6c58-39ea-aa27-c01a6d5718fe' of type 'org.apache.nifi.processors.script.ExecuteScript', group 'org.apache.nifi' and artifact 'nifi-scripting-nar' has deprecated 'Script Engine' value 'python'.
********************

********************
******************** CHECKING FOR VARIABLES
******************** Variables no longer exist in NiFi 2.0 and are replaced by Parameters and Parameter Contexts.
********************
ERROR: Component identifier: 'e65e16f2-d5ba-32b4-a04e-1ae97c44da02' contains 2 variable(s).
ERROR: Component identifier: '1428d513-770b-3c26-94c9-9641d57050c1' contains 3 variable(s).
********************

********************
******************** CHECKING FOR EVENT DRIVEN SCHEDULING STRATEGY
******************** Event Driven Scheduling Strategy no longer exists. Use Time Driven scheduling strategy instead.
********************
No component is configured with Event Driven scheduling strategy.
********************

********************
******************** CHECKING FOR TEMPLATES
******************** Templates no longer exist in NiFi 2.0. The use of NiFi Registry is recommended for versioning flows.
********************
ERROR: Component identifier: '6c387684-c2f2-3bbe-b485-8b6593cda6ba' with 'componentType' 'TEMPLATE'.
********************

********************
******************** CHECKING FOR INCORRECT KERBEROS CONFIGURATION
******************** NiFi components with Kerberos configuration that are available in NiFi 2.0 should be configured with the 'Kerberos User Service'
********************
ERROR: Component identifier: 'a716b969-9163-3e70-a162-b819b9ec4b63' of type 'org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_2_6', group 'org.apache.nifi' and artifact 'nifi-kafka-2-6-nar' has configured Kerberos properties. Please configure Kerberos using 'KerberosUserService', as this is the only configuration supported in NiFi 2.0.
ERROR: Component identifier: '0d650d47-5136-303c-a694-678a027b27cf' of type 'org.apache.nifi.processors.parquet.PutParquet', group 'org.apache.nifi' and artifact 'nifi-parquet-nar' has configured Kerberos properties. Please configure Kerberos using 'KerberosUserService', as this is the only configuration supported in NiFi 2.0.
ERROR: Component identifier: '86c88c6a-369f-3209-9a79-dd6168c28b48' of type 'org.apache.nifi.processors.hadoop.PutHDFS', group 'org.apache.nifi' and artifact 'nifi-hadoop-nar' has configured Kerberos properties. Please configure Kerberos using 'KerberosUserService', as this is the only configuration supported in NiFi 2.0.
********************

****************************************************************************************************
Script finished work successfully
ERROR: You should not upgrade until the above errors are taken care of!

Manual pre-upgrade check for NiFi managed by Cloudera Manager

You can perform a manual pre-upgrade check from the command line on NiFi nodes even if NiFi is managed by Cloudera Manager. This approach provides administrators with greater control and is ideal for scenarios that require or prefer a manual process. Follow the below steps to perform a pre-upgrade check directly from the command line.

  1. Switch the user to NiFi using su nifi.
  2. Find the location of these files.
    1. pre_upgrade_check.py
    2. CFM4-Deprecated-components.csv
    3. NiFi-1.x-components-with-legacy-Kerberos-properties.csv
  3. Identify the ${NIFI_WORKING_DIRECTORY} environment variable, which can be found in stdout.log during NiFi startup.
  4. Use Python3 to run the script with the following command, replacing the placeholders with the actual paths. 
    python3 [***/path/to***]/pre_upgrade_check.py \
"${NIFI_WORKING_DIRECTORY}/flow.json.gz" \
"[***/path/to***]/CFM4-Deprecated-components.csv" \
"[***/path/to***]/NiFi-1.x-components-with-legacy-Kerberos-properties.csv" \
"[***NIFI_WORKING_DIRECTORY***]/config_backup/bootstrap.conf" \
"${CUSTOM_JAVA_HOME}"
  5. If necessary, specify the ${CUSTOM_JAVA_HOME} argument.
    This should be set if the JAVA_HOME environment variable for the "nifi" user is different from the JAVA_HOME you are using to run the NiFi service.
  6. Once all file paths are updated in the command, run it to start the pre-upgrade check.

Manual pre-upgrade check for standalone NiFi (not managed by Cloudera Manager)

You can perform a manual pre-upgrade check from the command line on NiFi nodes in environments where NiFi is not managed by Cloudera Manager. Follow the below steps to perform a standalone pre-upgrade check directly from the command line.

  1. Download the pre-upgrade check script and the associated CSV files listing the components deprecated in NiFi 2 and the components with updated Kerberos properties.
    These files are part of the Cloudera Service Descriptor (CSD) package available behind the paywall.
    1. Download the NiFi CSD file and unzip it.
    2. Locate the required files.
      • scripts/service-commands/pre_upgrade_check.py
      • scripts/common-scripts/resources/CFM4-Deprecated-components.csv
      • scripts/common-scripts/resources/NiFi-1.x-components-with-legacy-Kerberos-properties.csv
  2. After downloading the files, run the pre-upgrade check script using the following command. 
    su nifi
python3 [***path_to***]/pre_upgrade_check.py \
"[***path_to***]/flow.json.gz" \
"[***path_to***]/CFM4-Deprecated-components.csv" \
"[***path_to***]/NiFi-1.x-components-with-legacy-Kerberos-properties.csv" \
"[***path_to***]/bootstrap.conf" \ 
"" \ 
"[***path_to***]/nifi-framework-nar-<version>.nar" \ 
"[***path_to***]/nifi.sh"

    For example:

    su nifi
python3 /tmp/scripts/service-commands/pre_upgrade_check.py \ 
/var/lib/nifi/flow.json.gz \
"/tmp/scripts/common-scripts/resources/CFM4-Deprecated-components.csv" \
"/tmp/scripts/common-scripts/resources/NiFi-1.x-components-with-legacy-Kerberos-properties.csv" \ 
"/tmp/nifi-1.26.0.2.1.7.1000-25/conf/bootstrap.conf" \ 
"" \
"/tmp/nifi-1.26.0.2.1.7.1000-25/lib/nifi-framework-nar-1.26.0.2.1.7.1000-25.nar" \
"/tmp/nifi-1.26.0.2.1.7.1000-25/bin/nifi.sh"

    The script generates a detailed report highlighting any issues that need to be resolved before upgrading to NiFi 2.

  3. Review this report thoroughly to ensure a smooth upgrade process.