Configuring flow.snapshot for Stateless NiFi connectors

Learn about the options you have in Streams Messaging Manager (SMM) when configuring the flow.snapshot Stateless NiFi connector property as well as the Save and Enhance feature that can be used to automatically import NiFi parameters into the connector configuration.

NiFi dataflows within Kafka Connect are deployed using the Stateless NiFi Source and Sink connectors. When you deploy your custom developed dataflows with the Stateless NiFi connectors, you must configure the flow.snapshot property. This property specifies the dataflow that is run within the connector.

Because the value of this property is the dataflow itself, the value you add to the property is not a simple string or integer. Instead, the value is the full contents of a flow definition JSON file, which depending on the dataflow, can be considerable in size.

As a result of this, the property is handled in a unique way by SMM. You can use multiple configuration methods to specify value. Additionally, you can choose to automatically enhance (amend) the configuration with the NiFi parameters that are available within the flow definition.

Configuration methods

You can use the following configuration methods when specifying a value for flow.snapshot. The following instructions assume that a flow definition JSON file is available to you and that you are logged in to the Connect tab of the SMM UI.

note

If the flow definition is larger than 64 KB, the following message appears:

Too large config, to view flow.snapshot click the edit button...

In a case like this, the dataflow is added, but you are not able to edit it directly in the Properties section. Instead, you must click and edit or review the flow definition in the modal window that appears.

Copy and paste the contents of the flow definition

This method is similar to configuring any other connector property. The only difference is that the value you specify is considerably longer than other property value.

Open the flow definition in an editor.
Copy the full contents of the file.
Paste the contents of the file in the value field of the flow.snapshot property.

Upload the flow definition

This method involves uploading the flow definition from your own machine. This method is convenient if the flow definition JSON is available on the machine you are using to access the SMM UI.

Click next to flow.snapshot.
Click Browse... in the modal window that appears.
Select the flow definition file that you want to use.
Click Save or Save and Enhance.

Reference a file path

The flow.snapshot property accepts file paths. This means that instead of copy-pasting or uploading the full contents of the flow definition JSON, you can also specify a location on disk where the file is available. Cloudera recommends that you use this method if the flow definition is larger than 1 MB.

Deploy the flow definition file on all Kafka Connect hosts.
important
Ensure that the file is deployed to all hosts at the same location and that the user that Kafka Connect is running as, which defaults to kafka, has read access to the file.
Set the file path as the value of the flow.snapshot property.

Using Save and Enhance

NiFi dataflows that you run within Kafka Connect are in the majority of cases parameterized. You can assign values to the parameters available within the dataflow on the SMM UI during connector configuration. This is done by adding each parameter, with specific prefixes, in the Properties section. However, this can take a considerable amount of time if done manually. This is especially true if the dataflow has many parameters and multiple parameter contexts.

In cases like this, you can choose to enhance the configuration by clicking the Save and Enhance option after a value for the flow.snapshot property is added. If this option is used, SMM parses the dataflow specified in flow.snapshot, extracts all parameters that are available in the dataflow, and adds them to properties of the connector. The Save and Enhance option is available on the modal window that appears after you click next to the property.