Pre-upgrade check for NiFi
Cloudera Flow Management allows you to run a pre-upgrade check to help you prepare for the transition to NiFi 2. The pre-upgrade check script is designed to highlight any preparation steps needed and make you aware of any breaking changes that will not be automatically handled by the Cloudera Flow Management Migration Tool.
Running a NiFi pre-upgrade check in Cloudera Manager
Cloudera Manager allows you to easily run a pre-upgrade check for NiFi. This process helps identify any potential issues before upgrading. Follow the below steps to perform the pre-upgrade check directly from Cloudera Manager.
Here is an example of the output:
Tue Sep 24 19:37:22 UTC 2024 JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera Using -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS Using /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck as conf dir Using scripts/service-commands/pre_upgrade_check.sh as process script CONF_DIR=/var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck CMF_CONF_DIR= Script pre_upgrade_check.sh started work! CMF_PACKAGE_DIR = /opt/cloudera/cm-agent/service NIFI_WORKING_DIRECTORY = /var/lib/nifi Path to csv file = /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck/scripts/common-scripts/resources/CFM4-Deprecated-components.csv Path to flow.json.gz = /var/lib/nifi/flow.json.gz Path to bootstrap.conf = /var/lib/nifi/config_backup/bootstrap.conf Custom java home = %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Check Before Upgrade to CFM with NiFi 2.0 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Using Python 3 or newer version (3.8.12) Script pre_upgrade_check.py started work! Passed arguments: Path to json flow file = /var/lib/nifi/flow.json.gz Path to deprecated components CSV file = /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck/scripts/common-scripts/resources/CFM4-Deprecated-components.csv Path to components with kerberos properties CSV file = /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck/scripts/common-scripts/resources/NiFi-1.x-components-with-legacy-Kerberos-properties.csv Path to bootstrap conf = /var/lib/nifi/config_backup/bootstrap.conf ******************** Validating arguments: You are running pre upgrade check before upgrade to NiFi 2.0 for NiFi managed by Cloudera Manager. Path to flow.json.gz file is valid! Path to deprecated components CSV file is valid! Path to components with kerberos properties CSV file is valid! Path to bootstrap.conf file is valid! ******************** Parsing bootstrap.conf: ******************** Reading and parsing /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck/scripts/common-scripts/resources/CFM4-Deprecated-components.csv: ******************** Reading and parsing /var/run/cloudera-scm-agent/process/60-nifi-NIFI_NODE-PreUpgradeCheck/scripts/common-scripts/resources/NiFi-1.x-components-with-legacy-Kerberos-properties.csv: ******************** Reading and parsing flow.json.gz: ******************** ******************** ******************** CHECKING THAT UPGRADE PATH IS SUPPORTED ******************** (It is required to be on CFM 2.1.7+ before upgrading to CFM 4.0.0+) ******************** Get current CFM version from bootstrap.conf and check it's valid to upgrade to CFM with NiFi 2.0 (CFM-4.0.0.0 or newer) Property value for 'working.dir' = /opt/cloudera/parcels/CFM-2.1.7.1000-42/NIFI Current CFM version could be upgraded to CFM with NiFi 2.0 (CFM-4.0.0.0 or newer). ******************** ******************** ******************** CHECKING THAT JAVA 21 IS USED ******************** (Java 21 is the minimum Java version required for NiFi and NiFi Registry 2.0) ******************** Get 'java' property value from bootstrap.conf Property value for 'java' = java Getting java home from JAVA_HOME in env variables. Executing command: /usr/java/jdk1.8.0_232-cloudera/bin/java -version 2>&1 | awk -F[\"\.] -v OFS=. 'NR==1{print $2}' Command output: 1 WARN: Your current Java version is too low for upgrading to NiFi 2.0 (CFM-4.0.0.0). Please install Java version 21 on each host with NiFi and follow the upgrade documentation for NiFi 2.0 (CFM-4.0.0.0). NOTE: Do not change the JDK for the current NiFi. It should only be changed during the upgrade to NiFi 2.0 (CFM-4.0.0.0). ******************** ******************** CHECKING FOR REMOVED COMPONENTS ******************** Some components have been removed in NiFi 2.0. ******************** You should switch to the recommended option before upgrading. ******************** If not, those components will be showing as ghost components in NiFi. ******************** ERROR: Component identifier: '8d74d462-7c52-35d1-ad1b-665696503ea2' of type 'org.apache.nifi.processors.gettcp.GetTCP', group 'org.apache.nifi' and artifact 'nifi-tcp-nar' is no longer supported. ERROR: Component identifier: 'f78dba93-e6ec-38d7-b5e4-fdbcd09fd831' of type 'org.apache.nifi.processors.pulsar.pubsub.PublishPulsar', group 'io.streamnative.connectors' and artifact 'nifi-pulsar-nar' is removed, but it is planned to be reintroduced later. Please check the available CFM-4.x releases. ERROR: Component identifier: '2e2edf20-3efc-388a-ac96-943b717e5252' of type 'org.apache.nifi.processors.twitter.GetTwitter', group 'org.apache.nifi' and artifact 'nifi-social-media-nar' is no longer supported. Please consider using 'ConsumeTwitter' instead. ERROR: Component identifier: 'a0766b6b-140e-31a8-a1fb-80f10459f870' of type 'org.apache.nifi.kerberos.KeytabCredentialsService', group 'org.apache.nifi' and artifact 'nifi-kerberos-credentials-service-nar' is no longer supported. Please consider using 'KerberosKeytabUserService' instead. ******************** ******************** ******************** CHECKING FOR REMOVED SCRIPTING LANGUAGES ******************** Some scripting languages have been removed in NiFi 2.0. ******************** You should rewrite your scripted components using Groovy as language or leverage the new Python API in NiFi 2.0. ******************** If not, those components will be showing as invalid in NiFi 2.0. ******************** ERROR: Component identifier: 'e0334fb0-738b-3000-bdfc-fee3e59f539d' of type 'org.apache.nifi.processors.script.InvokeScriptedProcessor', group 'org.apache.nifi' and artifact 'nifi-scripting-nar' has deprecated 'Script Engine' value 'ruby'. ERROR: Component identifier: 'e9d304ef-6c58-39ea-aa27-c01a6d5718fe' of type 'org.apache.nifi.processors.script.ExecuteScript', group 'org.apache.nifi' and artifact 'nifi-scripting-nar' has deprecated 'Script Engine' value 'python'. ******************** ******************** ******************** CHECKING FOR VARIABLES ******************** Variables no longer exist in NiFi 2.0 and are replaced by Parameters and Parameter Contexts. ******************** ERROR: Component identifier: 'e65e16f2-d5ba-32b4-a04e-1ae97c44da02' contains 2 variable(s). ERROR: Component identifier: '1428d513-770b-3c26-94c9-9641d57050c1' contains 3 variable(s). ******************** ******************** ******************** CHECKING FOR EVENT DRIVEN SCHEDULING STRATEGY ******************** Event Driven Scheduling Strategy no longer exists. Use Time Driven scheduling strategy instead. ******************** No component is configured with Event Driven scheduling strategy. ******************** ******************** ******************** CHECKING FOR TEMPLATES ******************** Templates no longer exist in NiFi 2.0. The use of NiFi Registry is recommended for versioning flows. ******************** ERROR: Component identifier: '6c387684-c2f2-3bbe-b485-8b6593cda6ba' with 'componentType' 'TEMPLATE'. ******************** ******************** ******************** CHECKING FOR INCORRECT KERBEROS CONFIGURATION ******************** NiFi components with Kerberos configuration that are available in NiFi 2.0 should be configured with the 'Kerberos User Service' ******************** ERROR: Component identifier: 'a716b969-9163-3e70-a162-b819b9ec4b63' of type 'org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_2_6', group 'org.apache.nifi' and artifact 'nifi-kafka-2-6-nar' has configured Kerberos properties. Please configure Kerberos using 'KerberosUserService', as this is the only configuration supported in NiFi 2.0. ERROR: Component identifier: '0d650d47-5136-303c-a694-678a027b27cf' of type 'org.apache.nifi.processors.parquet.PutParquet', group 'org.apache.nifi' and artifact 'nifi-parquet-nar' has configured Kerberos properties. Please configure Kerberos using 'KerberosUserService', as this is the only configuration supported in NiFi 2.0. ERROR: Component identifier: '86c88c6a-369f-3209-9a79-dd6168c28b48' of type 'org.apache.nifi.processors.hadoop.PutHDFS', group 'org.apache.nifi' and artifact 'nifi-hadoop-nar' has configured Kerberos properties. Please configure Kerberos using 'KerberosUserService', as this is the only configuration supported in NiFi 2.0. ******************** **************************************************************************************************** Script finished work successfully ERROR: You should not upgrade until the above errors are taken care of!
Running a manual pre-upgrade check for NiFi managed by Cloudera Manager
You can perform a manual pre-upgrade check from the command line on NiFi nodes even if NiFi is managed by Cloudera Manager. This approach provides administrators with greater control and is ideal for scenarios that require or prefer a manual process. Follow the below steps to perform a pre-upgrade check directly from the command line.
Running a manual pre-upgrade check for standalone NiFi (not managed by Cloudera Manager)
You can perform a manual pre-upgrade check from the command line on NiFi nodes in environments where NiFi is not managed by Cloudera Manager. Follow the below steps to perform a standalone pre-upgrade check directly from the command line.