ConsumeAzureEventHub

Description:

Receives messages from Microsoft Azure Event Hubs with checkpointing to ensure consistent event processing. Checkpoint tracking avoids consuming a message multiple times and enables reliable resumption of processing in the event of intermittent network failures. Checkpoint tracking requires external storage and provides the preferred approach to consuming messages from Azure Event Hubs. In clustered environment, ConsumeAzureEventHub processor instances form a consumer group and the messages are distributed among the cluster nodes (each message is processed on one cluster node only).

Tags:

azure, microsoft, cloud, eventhub, events, streaming, streams

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display Name	API Name	Default Value	Allowable Values	Description
Event Hub Namespace	event-hub-namespace			The namespace that the Azure Event Hubs is assigned to. This is generally equal to <Event Hub Names>-ns. Supports Expression Language: true (will be evaluated using variable registry only)
Event Hub Name	event-hub-name			The name of the event hub to pull messages from. Supports Expression Language: true (will be evaluated using variable registry only)
Service Bus Endpoint	Service Bus Endpoint	Azure	Azure Azure China Azure Germany Azure US Government	To support namespaces not in the default windows.net domain.
Transport Type	Transport Type	AMQP	AMQP AMQP_WEB_SOCKETS	Advanced Message Queuing Protocol Transport Type for communication with Azure Event Hubs
Shared Access Policy Name	event-hub-shared-access-policy-name			The name of the shared access policy. This policy must have Listen claims. Supports Expression Language: true (will be evaluated using variable registry only)
Shared Access Policy Key	event-hub-shared-access-policy-primary-key			The key of the shared access policy. Either the primary or the secondary key can be used. Sensitive Property: true
Use Azure Managed Identity	use-managed-identity	false	true false	Choose whether or not to use the managed identity of Azure VM/VMSS
Consumer Group	event-hub-consumer-group	$Default		The name of the consumer group to use. Supports Expression Language: true (will be evaluated using variable registry only)
Consumer Hostname	event-hub-consumer-hostname			DEPRECATED: This property is no longer used. Supports Expression Language: true (will be evaluated using variable registry only)
Record Reader	record-reader		Controller Service API: RecordReaderFactory Implementations: EBCDICRecordReader XMLReader AvroReader CiscoEmblemSyslogMessageReader ReaderLookup JsonPathReader JASN1Reader GrokReader JsonTreeReader WindowsEventLogReader YamlTreeReader IPFIXReader CEFReader SyslogReader ExcelReader ParquetReader ScriptedReader CSVReader Syslog5424Reader	The Record Reader to use for reading received messages. The event hub name can be referred by Expression Language '${eventhub.name}' to access a schema.
Record Writer	record-writer		Controller Service API: RecordSetWriterFactory Implementations: RecordSetWriterLookup ScriptedRecordSetWriter JsonRecordSetWriter AvroRecordSetWriter CSVRecordSetWriter ParquetRecordSetWriter XMLRecordSetWriter FreeFormTextRecordSetWriter	The Record Writer to use for serializing Records to an output FlowFile. The event hub name can be referred by Expression Language '${eventhub.name}' to access a schema. If not specified, each message will create a FlowFile.
Initial Offset	event-hub-initial-offset	End of stream	Start of stream End of stream	Specify where to start receiving messages if offset is not yet stored in the checkpoint store.
Prefetch Count	event-hub-prefetch-count	300		No Description Provided. Supports Expression Language: true (will be evaluated using variable registry only)
Batch Size	event-hub-batch-size	10		The number of messages to process within a NiFi session. This parameter affects throughput and consistency. NiFi commits its session and Event Hubs checkpoints after processing this number of messages. If NiFi session is committed, but fails to create an Event Hubs checkpoint, then it is possible that the same messages will be received again. The higher number, the higher throughput, but possibly less consistent. Supports Expression Language: true (will be evaluated using variable registry only)
Message Receive Timeout	event-hub-message-receive-timeout	1 min		The amount of time this consumer should wait to receive the Batch Size before returning. Supports Expression Language: true (will be evaluated using variable registry only)
Checkpoint Strategy	checkpoint-strategy	Azure Blob Storage	Azure Blob Storage Component State	Specifies which strategy to use for storing and retrieving partition ownership and checkpoint information for each partition.
Storage Account Name	storage-account-name			Name of the Azure Storage account to store event hub consumer group state. Supports Expression Language: true (will be evaluated using variable registry only) This Property is only considered if the [Checkpoint Strategy] Property has a value of "Azure Blob Storage".
Storage Account Key	storage-account-key			The Azure Storage account key to store event hub consumer group state. Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only) This Property is only considered if the [Checkpoint Strategy] Property has a value of "Azure Blob Storage".
Storage SAS Token	storage-sas-token			The Azure Storage SAS token to store Event Hub consumer group state. Always starts with a ? character. Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only) This Property is only considered if the [Checkpoint Strategy] Property has a value of "Azure Blob Storage".
Storage Container Name	storage-container-name			Name of the Azure Storage container to store the event hub consumer group state. If not specified, event hub name is used. Supports Expression Language: true (will be evaluated using variable registry only) This Property is only considered if the [Checkpoint Strategy] Property has a value of "Azure Blob Storage".
Proxy Configuration Service	proxy-configuration-service		Controller Service API: ProxyConfigurationService Implementation: StandardProxyConfigurationService	Specifies the Proxy Configuration Controller Service to proxy network requests. Supported proxies: HTTP + AuthN This Property is only considered if the [Transport Type] Property has a value of "AMQP_WEB_SOCKETS".

Relationships:

Name	Description
success	FlowFiles received from Event Hub.
parse.failure	If a message from event hub cannot be parsed using the configured Record Reader or failed to be written by the configured Record Writer, the contents of the message will be routed to this Relationship as its own individual FlowFile.

Reads Attributes:

None specified.

Writes Attributes:

Name	Description
eventhub.enqueued.timestamp	The time (in milliseconds since epoch, UTC) at which the message was enqueued in the event hub
eventhub.offset	The offset into the partition at which the message was stored
eventhub.sequence	The sequence number associated with the message
eventhub.name	The name of the event hub from which the message was pulled
eventhub.partition	The name of the partition from which the message was pulled
eventhub.property.*	The application properties of this message. IE: 'application' would be 'eventhub.property.application'

State management:

Scope	Description
LOCAL, CLUSTER	Local state is used to store the client id. Cluster state is used to store partition ownership and checkpoint information when component state is configured as the checkpointing strategy.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

System Resource Considerations:

None specified.