TriggerClouderaHiveMetaStoreEvent 2.3.0.4.10.0.0-147

Bundle
com.cloudera | nifi-cdf-hive-nar
Description
This component uses the Hive client version 3.1.3000.7.3.1.400-76. The processor is capable to trigger different type of events in the HiveMetaStore and generate notifications. The metastore action to be executed is determined from the incoming path and event type attributes. The supported event type values are 'put' in case of data insertion or 'delete' in case of data removal. The notifications should be enabled in the metastore configuration to generate them e.g.: the 'hive.metastore.transactional.event.listeners' should have a proper listener configured, for instance 'org.apache.hive.hcatalog.listener.DbNotificationListener'.
Tags
delete, event, hive, insert, metastore, notification, partition
Input Requirement
Supports Sensitive Dynamic Properties
false
  • Additional Details for TriggerClouderaHiveMetaStoreEvent 2.3.0.4.10.0.0-147

    TriggerHiveMetaStoreEvent

    Event Types

    The processor supports the following event types:

    Put

    The put event type represents a file or directory insertion. The metastore action depends on the target table. If the table is partitioned the processor will parse the target partition values from the path e.g.: if the inserted file’s path is ‘hdfs://localhost:9000/user/hive/warehouse/table/a=5/b=10/file’ then the partition values are coming from the ‘/a=5/b=10/’ path part -> [5,10]. If the parsed partition does not exist then an ‘APPEND PARTITION’ event is going to be called and the new partition will be registered in the metastore. If the partition exists only an ‘INSERT’ event will be called.

    Delete

    The delete event type represents a ‘DROP PARTITION’ metastore action. The processor will parse the target partition values from the path e.g.: if the deleted directory’s path is ‘hdfs://localhost:9000/user/hive/warehouse/table/a=5/b=10/’ then the partition values are coming from the ‘/a=5/b=10/’ path part -> [5,10]. If the table is not partitioned then no metastore action will be executed.

    Any other event type will cause an error and will be directed to the failure relation of the processor.

Properties
Relationships
Name Description
success A FlowFile is routed to this relationship after the data ingestion was successful.
failure A FlowFile is routed to this relationship if the data ingestion failed and retrying the operation will also fail, such as an invalid data or schema.
Writes Attributes
Name Description
metastore.notification.event The event type of the triggered notification.