GetHDFSEvents 2.3.0.4.10.0.0-147

Bundle
org.apache.nifi | nifi-hadoop-nar
Description
This processor polls the notification events provided by the HdfsAdmin API. Since this uses the HdfsAdmin APIs it is required to run as an HDFS super user. Currently there are six types of events (append, close, create, metadata, rename, and unlink). Please see org.apache.hadoop.hdfs.inotify.Event documentation for full explanations of each event. This processor will poll for new events based on a defined duration. For each event received a new flow file will be created with the expected attributes and the event itself serialized to JSON and written to the flow file's content. For example, if event.type is APPEND then the content of the flow file will contain a JSON file containing the information about the append event. If successful the flow files are sent to the 'success' relationship. Be careful of where the generated flow files are stored. If the flow files are stored in one of processor's watch directories there will be a never ending flow of events. It is also important to be aware that this processor must consume all events. The filtering must happen within the processor. This is because the HDFS admin's event notifications API does not have filtering.
Tags
events, filesystem, hadoop, inotify, notifications
Input Requirement
FORBIDDEN
Supports Sensitive Dynamic Properties
false
Properties
State Management
Scopes Description
CLUSTER The last used transaction id is stored. This is used
Relationships
Name Description
success A flow file with updated information about a specific event will be sent to this relationship.
Writes Attributes
Name Description
mime.type This is always application/json.
hdfs.inotify.event.type This will specify the specific HDFS notification event type. Currently there are six types of events (append, close, create, metadata, rename, and unlink).
hdfs.inotify.event.path The specific path that the event is tied to.
See Also