PutIceberg
Description
Iceberg is a high-performance format for huge analytic tables.
The PutIceberg processor is capable of pushing data into Iceberg tables using different types of Iceberg catalog implementations.
Commit retry properties
Iceberg supports multiple concurrent writes using optimistic concurrency.
The processor's commit retry implementation is using exponential backoff with jitter and scale factor 2, and provides the following properties to configure the behaviour according to its usage.
-
Number Of Commit Retries (default: 10) - Number of retries that the processor is going to try to commit the new data files.
-
Minimum Commit Wait Time (default: 100 ms) - Minimum time that the processor is going to wait before each commit attempt.
-
Maximum Commit Wait Time (default: 2 sec) - Maximum time that the processor is going to wait before each commit attempt.
-
Maximum Commit Duration (default: 30 sec) - Maximum duration that the processor is going to wait before failing the current processor event's commit.
The NiFi side retry logic is built on top of the Iceberg commit retry logic which can be configured through table properties. See more: Table behavior properties