UpdateDeltaLakeTable 2.3.0.4.10.0.0-147

Bundle
com.cloudera | nifi-cdf-deltalake-nar
Description
Updates a DeltaLake table with information of new added files.
Tags
cloud, deltalake, deltatable, parquet, storage
Input Requirement
REQUIRED
Supports Sensitive Dynamic Properties
false
  • Additional Details for UpdateDeltaLakeTable 2.3.0.4.10.0.0-147

    UpdateDeltaLakeTable

    Description

    UpdateDeltaLakeTable can be used to update an existing DeltaLake table when a new file is added to a Hadoop directory. The URL of the added file can be set in the Data File URL property. It supports expression language and the default value is ${hadoop.file.url} i.e. expects the URL in the hadoop.file.url attribute of the incoming FlowFile.

    Partition support

    The processor is capable of writing data to a partitioned delta table. The partition names and values are derived from the input file path and are matched with the partitions defined on the delta table.

    E.g. let’s assume the table is partitioned by “year” and “month” and the input file path is:

    ".../folder1/year=2020/month=11/filename.extension"
    

    In this example the input file will be put to year=2020 and month=11 partitions in the delta table.

Properties
Relationships
Name Description
success DeltaLake table successfully updated
failure DeltaLake table update failed