ConsumeMQTT

Description:

Subscribes to a topic and receives messages from an MQTT broker

Additional Details...

Tags:

subscribe, MQTT, IOT, consume, listen

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Broker URIBroker URIThe URI(s) to use to connect to the MQTT broker (e.g., tcp://localhost:1883). The 'tcp', 'ssl', 'ws' and 'wss' schemes are supported. In order to use 'ssl', the SSL Context Service property must be set. When a comma-separated URI list is set (e.g., tcp://localhost:1883,tcp://localhost:1884), the processor will use a round-robin algorithm to connect to the brokers on connection failure.
Supports Expression Language: true (will be evaluated using variable registry only)
MQTT Specification VersionMQTT Specification Versionv3 AUTO
  • v3 AUTO Start with v3.1.1 and fallback to v3.1.0 if not supported by a broker
  • v5.0
  • v3.1.1
  • v3.1.0
The MQTT specification version when connecting with the broker. See the allowable value descriptions for more details.
UsernameUsernameUsername to use when connecting to the broker
Supports Expression Language: true (will be evaluated using variable registry only)
PasswordPasswordPassword to use when connecting to the broker
Sensitive Property: true
SSL Context ServiceSSL Context ServiceController Service API:
SSLContextService
Implementations: StandardRestrictedSSLContextService
StandardSSLContextService		The SSL Context Service used to provide client certificate information for TLS/SSL connections.
Session stateSession stateClean Session
  • Clean Session Client and Server discard any previous session and start a new one. This session lasts as long as the network connection. State data associated with this session is not reused in any subsequent session
  • Resume Session Server resumes communications with the client based on state from the current session (as identified by the ClientID). The client and server store the session after the client and server are disconnected. After the disconnection of a session that was not a clean session, the server stores further QoS 1 and QoS 2 messages that match any subscriptions that the client had at the time of disconnection as part of the session state
Whether to start a fresh or resume previous flows. See the allowable value descriptions for more details.
Session Expiry IntervalSession Expiry Interval24 hrsAfter this interval the broker will expire the client and clear the session state.

This Property is only considered if all of the following conditions are met:
  • The [MQTT Specification Version] Property has a value of "v5.0".
  • The [Session state] Property has a value of "Resume Session".
Client IDClient IDMQTT client ID to use. If not set, a UUID will be generated.
Supports Expression Language: true (will be evaluated using variable registry only)
Group IDGroup IDMQTT consumer group ID to use. If group ID not set, client will connect as individual consumer.
Topic FilterTopic FilterThe MQTT topic filter to designate the topics to subscribe to.
Supports Expression Language: true (will be evaluated using variable registry only)
Quality of Service (QoS)Quality of Service(QoS)0 - At most once
  • 0 - At most once Best effort delivery. A message won’t be acknowledged by the receiver or stored and redelivered by the sender. This is often called “fire and forget” and provides the same guarantee as the underlying TCP protocol.
  • 1 - At least once Guarantees that a message will be delivered at least once to the receiver. The message can also be delivered more than once
  • 2 - Exactly once Guarantees that each message is received only once by the counterpart. It is the safest and also the slowest quality of service level. The guarantee is provided by two round-trip flows between sender and receiver.
The Quality of Service (QoS) to receive the message with. Accepts values '0', '1' or '2'; '0' for 'at most once', '1' for 'at least once', '2' for 'exactly once'.
Record Readerrecord-readerController Service API:
RecordReaderFactory
Implementations: Syslog5424Reader
CEFReader
ReaderLookup
CiscoEmblemSyslogMessageReader
CSVReader
GrokReader
SyslogReader
JsonTreeReader
JsonPathReader
XMLReader
AvroReader
JASN1Reader
ExcelReader
ParquetReader
EBCDICRecordReader
WindowsEventLogReader
IPFIXReader
ScriptedReader		The Record Reader to use for parsing received MQTT Messages into Records.
Record Writerrecord-writerController Service API:
RecordSetWriterFactory
Implementations: JsonRecordSetWriter
ParquetRecordSetWriter
CSVRecordSetWriter
ScriptedRecordSetWriter
XMLRecordSetWriter
FreeFormTextRecordSetWriter
AvroRecordSetWriter
RecordSetWriterLookup		The Record Writer to use for serializing Records before writing them to a FlowFile.
Add attributes as fieldsadd-attributes-as-fieldstrue
  • true
  • false
If setting this property to true, default fields are going to be added in each record: _topic, _qos, _isDuplicate, _isRetained.

This Property is only considered if the [Record Reader] Property has a value specified.
Message Demarcatormessage-demarcatorWith this property, you have an option to output FlowFiles which contains multiple messages. This property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple messages. This is an optional property ; if not provided, and if not defining a Record Reader/Writer, each message received will result in a single FlowFile. To enter special character such as 'new line' use CTRL+Enter or Shift+Enter depending on the OS.
Supports Expression Language: true (will be evaluated using variable registry only)
Connection Timeout (seconds)Connection Timeout (seconds)30Maximum time interval the client will wait for the network connection to the MQTT server to be established. The default timeout is 30 seconds. A value of 0 disables timeout processing meaning the client will wait until the network connection is made successfully or fails.
Keep Alive Interval (seconds)Keep Alive Interval (seconds)60Defines the maximum time interval between messages sent or received. It enables the client to detect if the server is no longer available, without having to wait for the TCP/IP timeout. The client will ensure that at least one message travels across the network within each keep alive period. In the absence of a data-related message during the time period, the client sends a very small "ping" message, which the server will acknowledge. A value of 0 disables keepalive processing in the client.
Last Will MessageLast Will MessageThe message to send as the client's Last Will.
Last Will TopicLast Will TopicThe topic to send the client's Last Will to.

This Property is only considered if the [Last Will Message] Property has a value specified.
Last Will RetainLast Will Retainfalse
  • true
  • false
Whether to retain the client's Last Will.

This Property is only considered if the [Last Will Message] Property has a value specified.
Last Will QoS LevelLast Will QoS Level0 - At most once
  • 0 - At most once Best effort delivery. A message won’t be acknowledged by the receiver or stored and redelivered by the sender. This is often called “fire and forget” and provides the same guarantee as the underlying TCP protocol.
  • 1 - At least once Guarantees that a message will be delivered at least once to the receiver. The message can also be delivered more than once
  • 2 - Exactly once Guarantees that each message is received only once by the counterpart. It is the safest and also the slowest quality of service level. The guarantee is provided by two round-trip flows between sender and receiver.
QoS level to be used when publishing the Last Will Message.

This Property is only considered if the [Last Will Message] Property has a value specified.
Max Queue SizeMax Queue SizeThe MQTT messages are always being sent to subscribers on a topic regardless of how frequently the processor is scheduled to run. If the 'Run Schedule' is significantly behind the rate at which the messages are arriving to this processor, then a back up can occur in the internal queue of this processor. This property specifies the maximum number of messages this processor will hold in memory at one time in the internal queue. This data would be lost in case of a NiFi restart.

Relationships:

NameDescription
parse.failureIf a message cannot be parsed using the configured Record Reader, the contents of the message will be routed to this Relationship as its own individual FlowFile.
MessageThe MQTT message output

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
record.countThe number of records received
mqtt.brokerMQTT broker that was the message source
mqtt.topicMQTT topic on which message was received
mqtt.qosThe quality of service for this message.
mqtt.isDuplicateWhether or not this message might be a duplicate of one which has already been received.
mqtt.isRetainedWhether or not this message was from a current publisher, or was "retained" by the server as the last message published on the topic.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

System Resource Considerations:

ResourceDescription
MEMORYThe 'Max Queue Size' specifies the maximum number of messages that can be hold in memory by NiFi by a single instance of this processor. A high value for this property could represent a lot of data being stored in memory.

See Also:

PublishMQTT