Cloudera Docs

Monitor data flows

As data travels through your data flows in the Cloudera Flow Management system, it is important to understand how these data flows are performing. Monitoring information can help you to assess if a data flow requires more resources. Learn how you can check and observe your flows in Cloudera Flow Management.

There are multiple ways to monitor the NiFi flows to check on errors, memory usage, CPU usage, data flow statistics, and so on.

Status bar

The Status bar provides information about the overall health of the system. You can find it at the top of the NiFi UI, under the component toolbar. It contains important statistics about the current health of NiFi. The number of active threads can indicate how hard NiFi is working, and the queued statistics indicate how many FlowFiles are queued across the entire flow, and the total size of those FlowFiles.

If the NiFi instance is in a cluster, we will also see an indicator here telling us how many nodes are in the cluster and how many are currently connected. In this case, the number of active threads and the queue size are indicative of the sum of all nodes that are currently connected.

Component statistics

Each processor, process group, and remote process group on the canvas provides fine-grained statistics about their operations. Connections and process groups provide information about the amount of data in their queues. Processors provide information on how much data they processed in the last five minutes. This is a rolling window and allows you to see things like the number of FlowFiles that have been consumed by a processor, as well as the number of FlowFiles that have been emitted by the processor. The connections between processors also expose the number of items that are currently queued.

To see historical values for these metrics, right-click on a component and choose the View Status History menu item. This shows you a graph that spans the time since NiFi was started, or up to 24 hours, whichever is less. You can extend or reduce the amount of time that is shown by changing the configuration in the properties file.

Queue interaction

You can view the FlowFiles queued in a connection when necessary. Right-click a connection, and click List queue in the context menu to open the queue listing. You can check the list even when the source and destination processors are actively running.

Additionally, details for a Flowfile in the listing can be viewed by clicking Details in the left-most column. From here, the FlowFile details and attributes are available along with the buttons for downloading or viewing the content. If the source or destination of the connection is actively running, there is a chance that the desired FlowFile is no longer in the active queue.

You can delete FlowFiles queued in a connection. Right-click the connection, and click Empty queue in the context menu to initiate the removal of the FlowFiles. You can remove FlowFiles even if the source and destination processors are actively running.

Bulletins

If a processor logs anything as a WARNING or ERROR, you see a Bulletin Indicator displayed in the top-right corner of the processor. This indicator resembles a sticky note and is displayed for five minutes after the event occurs. Hovering over the bulletin provides information about the incident so that you do not have to go through log messages to find it. You can also change the log level at which bulletins occur in the Settings tab of the Configure dialog for a processor.

When the framework generates a bulletin, a bulletin indicator is highlighted at the top-right of the screen. The bulletin board shows the latest WARNING and ERROR messages that are generated by NiFi processors in real time. To access the bulletin board, click Bulletin Board in the Global Menu at the top-right corner of the NiFi UI. Clicking this option takes you to the bulletin board where you can see all bulletins that occur across the NiFi instance and can filter based on the message, name, ID, ad group ID. It refreshes automatically and you can also disable it. You can navigate to the component with a bulletin by clicking the component UUID.

NiFi summary page

NiFi Summary provides information about all of the components on the canvas in a tabular format. It also provides system diagnostics that include disk usage, CPU utilization, Java heap, and garbage collection information. To access the summary page, click Summary in the Global Menu at the top-right corner of the NiFi UI. Clicking this option takes you to the summary page, where you can get information about all the components of that particular NiFi instance or cluster. You can monitor the following components in the NiFi summary UI:

  • Processors
  • Input ports
  • Output ports
  • Remote process groups
  • Connections
  • Process groups

Depending on the component, you can filter by name, type, or UIR.

For more in-depth monitoring information, see the Apache NiFi User Guide.