This processor uses Iceberg API to parse and load CDC (Change Data Capture) records into Iceberg tables. The incoming data sets are parsed with Record Reader Controller Service and ingested into an Iceberg table using the configured catalog service and provided table information. A CDC operation type and before/after records need to be specified in the incoming record to identify the change which is applied to Iceberg table. The processor requires Iceberg 2.0 tables. The target Iceberg table should already exist and it must have the same schema which before/after records have in the incoming CDC record. To avoid 'small file problem' it is recommended pre-appending a MergeRecord processor.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|
| Catalog Service | catalog-service | | Controller Service API:
IcebergCatalogService
Implementations: HiveCatalogService
HadoopCatalogService | Specifies the Controller Service to use for handling references to table’s metadata files. |
| Catalog Namespace | catalog-namespace | | | The namespace of the catalog.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| Table Name | table-name | | | The name of the Iceberg table to write to.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| File Format | file-format | | | File format to use when writing Iceberg data files. If not set, then the 'write.format.default' table property will be used, default value is parquet.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| Record Type | Record Type | Debezium | - Debezium
 - GoldenGate
- Custom
| Specifies the type of the incoming CDC record. In case Custom record type is chosen, the used operation type values need to be specified for insert, delete and update. |
| Record Reader | record-reader | | Controller Service API:
RecordReaderFactory
Implementations: Syslog5424Reader
CEFReader
ReaderLookup
CiscoEmblemSyslogMessageReader
CSVReader
GrokReader
SyslogReader
JsonTreeReader
JsonPathReader
XMLReader
AvroReader
JASN1Reader
ExcelReader
ParquetReader
EBCDICRecordReader
WindowsEventLogReader
IPFIXReader
ScriptedReader | Specifies the Controller Service to use for parsing incoming data and determining the data's schema. |
| Insert Operation Type | Insert Operation Type | | | Specifies the operation type values used which denote an insert operation in the incoming record.
This Property is only considered if the [Record Type] Property has a value of "Custom". |
| Delete Operation Type | Delete Operation Type | | | Specifies the operation type values used which denote a delete operation in the incoming record.
This Property is only considered if the [Record Type] Property has a value of "Custom". |
| Update Operation Type | Update Operation Type | | | Specifies the operation type values used which denote an update operation in the incoming record.
This Property is only considered if the [Record Type] Property has a value of "Custom". |
| Operation RecordPath | Operation RecordPath | | | This property denotes a RecordPath that will be evaluated against each incoming Record in order to determine the operation type. The RecordPath must evaluate to one of the valid Iceberg Operation Types, or the incoming FlowFile will be routed to failure.
This Property is only considered if the [Record Type] Property has a value of "Custom". |
| Before Data RecordPath | Before Data RecordPath | | | This property denotes a RecordPath that will be evaluated against each incoming Record and marks the record state before the operation.
This Property is only considered if the [Record Type] Property has a value of "Custom". |
| After Data RecordPath | After Data RecordPath | | | This property denotes a RecordPath that will be evaluated against each incoming Record and marks the record state after the operation.
This Property is only considered if the [Record Type] Property has a value of "Custom". |
| Failure Strategy | Failure Strategy | Route to Failure | - Route to Failure
- Rollback Session
| If one or more Records cannot be processed or the operation can not be applied to Iceberg table, specifies how to handle the failure |
| Maximum File Size | maximum-file-size | | | The maximum size that a file can be, if the file size is exceeded a new file will be generated with the remaining data. If not set, then the 'write.target-file-size-bytes' table property will be used, default value is 512 MB.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| Kerberos User Service | kerberos-user-service | | Controller Service API:
KerberosUserService
Implementations: KerberosKeytabUserService
KerberosPasswordUserService
KerberosTicketCacheUserService | Specifies the Kerberos User Controller Service that should be used for authenticating with Kerberos. |
| Number of Commit Retries | number-of-commit-retries | 10 | | Number of times to retry a commit before failing.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| Minimum Commit Wait Time | minimum-commit-wait-time | 100 ms | | Minimum time to wait before retrying a commit.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| Maximum Commit Wait Time | maximum-commit-wait-time | 2 sec | | Maximum time to wait before retrying a commit.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| Maximum Commit Duration | maximum-commit-duration | 30 sec | | Total retry timeout period for a commit.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |