Upgrading a NiFi cluster
The NiFi software configuration described in this procedure applies to both standalone and cluster node instances of NiFi software.
Download the latest NiFi release.
Install the new release in /opt. Your directory structure should look similar to:
/ |--> opt/ | |--> nifi-current_version | | |--> LICENSE | | |--> NOTICE | | |--> README | | |--> bin/ | | |--> conf/ | | |--> docs/ | | |--> lib/ | | |--> work/ | | | |--> nifi-1.0.0 | | |--> LICENSE | | |--> NOTICE | | |--> README | | |--> bin/ | | |--> conf/ | | |--> docs/ | | |--> lib/ | | |--> work/ | | | |--> Configuration_resources/ | | |--> authorized-users.xml | | |--> flow.xml.gz | | |--> templates/ | | |--> custom_lib/ | |--> database_repository/ |--> content_repository_1/ |--> content_repository_N/ |--> flowfile_repository/ |--> NiFi-logs/ |--> provenance_repository_1/ |--> provenance_repository_N/
Save the
authorized-users.xml
from your older NiFi version, so that NiFi 1.0.0 can use it to populate the values of theauthorizations.xml
andusers.xml
files.Using the values already configured in the following files inside the previous NiFi conf directory, update the corresponding lines in the same files under target NiFi directory (nifi-1.0.0):
bootstrap.conf
logback.xml
nifi.properties
Note Ensure that you make no typos when you are configuring the various paths to your existing repos and the path to the
flow.xml.gz
file in thenifi.properties
file in the newer release.Ensure that you have the same Run As user configured in the
bootstrap.conf
file.If no new lines were added to these files, you can copy them directly from an older version to a newer. If you do copy the
nifi.properties
file, update thenifi.version
number.
Verify that all file and directory ownerships for your target NiFi directory match what you set on the previous version.
Stop the NCM and all nodes running the previous NiFi version.
Note NiFi 1.0.0 implements zero-master clustering and there is no NCM in NiFi 1.0.0.
It is important to stop everything. You cannot upgrade and restart one node at a time. You could end up with mismatched versions of NiFi connected to your cluster NCM.
It is OK to stop your NiFi cluster while files still exist in your dataflows.
Before you start the upgraded NiFi nodes, add the location of your
authorized-users.xml
file toauthorizations.xml
, if you want to map your previous authorized users to the new NiFi authentication model.<authorizer> <identifier>file-provider</identifier> <class>org.apache.nifi.authorization.FileAuthorizer</class> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Users File">./conf/users.xml</property> <property name="Initial Admin Identity"></property> <property name="Legacy Authorized Users File"></property>
To support NiFi's new zero-master clustering model, ensure that the following properties are set on each node in your cluster:
nifi.web.http.port=<node port>
nifi.cluster.is.node=true
nifi.cluster.node.address=<fully qualified hostname of the node>
nifi.cluster.node.protocol.port=<node protocol port>
nifi.state.management.embedded.zookeeper.start=true
nifi.state.management.provider.cluster=zk-provider
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
nifi.zookeeper.connect.string=<A comma-separated list of host:port pairs to connect to ZooKeeper. For example, my-zk-server1:2181,my-zk-server2:2181,my-zk-server3:2183>
Configuration details can be found in the Clustering Configuration, the Cluster Common Properties, the Cluster Node Properties, and the ZooKeeper Properties sections of Hortonworks DataFlow Administration.
Set the following NiFi properties to ensure that the cluster coordinator and other nodes have time to select the correct dataflow:
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=<number of NiFi nodes in the cluster>
Start the upgraded NiFi nodes.
Note Check the
nifi-app.log
for errors and for any nodes that fail to join the upgraded cluster.If you configured NiFi to run as a service, ensure that any path or links for that service are updated to point at the newly installed version executables.
Configure the
state-management.xml
andzookeeper.properties
file on every node with the following information:Specify whether you want to use an embedded or external ZooKeeper server.
If you are using an embedded server, identify the nodes on which you want to run the Zookper server. For these nodes, be sure to also edit the state management porition of the
nifi.properties
file.
See the Hortonworks DataFlow Administration for additional information on state management and using the ZooKeeper server.
If you want to start using LDAP user authentication, configure the
login-identity-providers.xml
file, and update thenifi.properties
file.If you want to setup NiFi JVM lifecycle event notifications, configure the
bootstrap-notification.services.xml
file and update the relevant settings in thebootstrap.conf
file.Once you have verified that the new version has started, is processing data, and can be accessed via the UI, you can delete the older version.