Retrieves Change Data Capture (CDC) events from a MySQL database. CDC Events include INSERT, UPDATE, DELETE operations. Events are output as either a group of a specified number of events (the default is 1 so each event becomes its own flow file) or grouped as a full transaction (BEGIN to COMMIT). All events are ordered by the time at which the operation occurred. NOTE: If the processor is stopped before the specified number of events have been written to a flow file, the partial flow file will be output in order to maintain the consistency of the event stream.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|
| MySQL Nodes | capture-change-mysql-hosts | | | A list of hostname (and optional port) entries corresponding to nodes in a MySQL cluster. The entries should be comma separated using a colon (if the port is to be specified) such as host1:port,host2:port,.... For example mysql.myhost.com:3306. The port need not be specified, when omitted the default MySQL port value of 3306 will be used. This processor will attempt to connect to the hosts in the list in order. If one node goes down and failover is enabled for the cluster, then the processor will connect to the active node (assuming its node entry is specified in this property).
Supports Expression Language: true (will be evaluated using variable registry only) |
| MySQL Driver Class Name | capture-change-mysql-driver-class | com.mysql.jdbc.Driver | | The class name of the MySQL database driver class
Supports Expression Language: true (will be evaluated using variable registry only) |
| MySQL Driver Location(s) | capture-change-mysql-driver-locations | | | Comma-separated list of files/folders and/or URLs containing the MySQL driver JAR and its dependencies (if any). For example '/var/tmp/mysql-connector-java-5.1.38-bin.jar'
This property expects a comma-separated list of resources. Each of the resources may be of any of the following types: directory, URL, file.
Supports Expression Language: true (will be evaluated using variable registry only) |
| Username | capture-change-mysql-username | | | Username to access the MySQL cluster
Supports Expression Language: true (will be evaluated using variable registry only) |
| Password | capture-change-mysql-password | | | Password to access the MySQL cluster
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only) |
| Event Processing Strategy | events-per-flowfile-strategy | Max Events Per FlowFile | - Max Events Per FlowFile
 - One Transaction Per FlowFile
| Specifies the strategy to use when writing events to FlowFile(s), such as 'Max Events Per FlowFile' |
| Events Per FlowFile | number-of-events-per-flowfile | 1 | | Specifies how many events should be written to a single FlowFile. If the processor is stopped before the specified number of events has been written,the events will still be written as a FlowFile before stopping.
Supports Expression Language: true (will be evaluated using variable registry only)
This Property is only considered if the [Event Processing Strategy] Property has a value of "Max Events Per FlowFile". |
| Server ID | capture-change-mysql-server-id | | | The client connecting to the MySQL replication group is actually a simplified replica (server), and the Server ID value must be unique across the whole replication group (i.e. different from any other Server ID being used by any primary or replica). Thus, each instance of CaptureChangeMySQL must have a Server ID unique across the replication group. If the Server ID is not specified, it defaults to 65535.
Supports Expression Language: true (will be evaluated using variable registry only) |
| Database/Schema Name Pattern | capture-change-mysql-db-name-pattern | | | A regular expression (regex) for matching databases (or schemas, depending on your RDBMS' terminology) against the list of CDC events. The regex must match the database name as it is stored in the RDBMS. If the property is not set, the database name will not be used to filter the CDC events. NOTE: DDL events, even if they affect different databases, are associated with the database used by the session to execute the DDL. This means if a connection is made to one database, but the DDL is issued against another, then the connected database will be the one matched against the specified pattern. |
| Table Name Pattern | capture-change-mysql-name-pattern | | | A regular expression (regex) for matching CDC events affecting matching tables. The regex must match the table name as it is stored in the database. If the property is not set, no events will be filtered based on table name. |
| Max Wait Time | capture-change-mysql-max-wait-time | 30 seconds | | The maximum amount of time allowed for a connection to be established, zero means there is effectively no limit.
Supports Expression Language: true (will be evaluated using variable registry only) |
| Distributed Map Cache Client - unused | capture-change-mysql-dist-map-cache-client | | Controller Service API:
DistributedMapCacheClient
Implementations: RedisDistributedMapCacheClientService
HazelcastMapCacheClient
DistributedMapCacheClientService
HBase_1_1_2_ClientMapCacheService
HBase_2_ClientMapCacheService
CassandraDistributedMapCache
CouchbaseMapCacheClient | This is a legacy property that is no longer used to store table information, the processor will handle the table information (column names, types, etc.) |
| Retrieve All Records | capture-change-mysql-retrieve-all-records | true | | Specifies whether to get all available CDC events, regardless of the current binlog filename and/or position. If binlog filename and position values are present in the processor's State, this property's value is ignored. This allows for 4 different configurations: 1) If binlog data is available in processor State, that is used to determine the start location and the value of Retrieve All Records is ignored. 2) If no binlog data is in processor State, then Retrieve All Records set to true means start at the beginning of the binlog history. 3) If no binlog data is in processor State and Initial Binlog Filename/Position are not set, then Retrieve All Records set to false means start at the end of the binlog history. 4) If no binlog data is in processor State and Initial Binlog Filename/Position are set, then Retrieve All Records set to false means start at the specified initial binlog file/position. To reset the behavior, clear the processor state (refer to the State Management section of the processor's documentation). |
| Include Begin/Commit Events | capture-change-mysql-include-begin-commit | false | | Specifies whether to emit events corresponding to a BEGIN or COMMIT event in the binary log. Set to true if the BEGIN/COMMIT events are necessary in the downstream flow, otherwise set to false, which suppresses generation of these events and can increase flow performance. |
| Include DDL Events | capture-change-mysql-include-ddl-events | false | | Specifies whether to emit events corresponding to Data Definition Language (DDL) events such as ALTER TABLE, TRUNCATE TABLE, e.g. in the binary log. Set to true if the DDL events are desired/necessary in the downstream flow, otherwise set to false, which suppresses generation of these events and can increase flow performance. |
| State Update Interval | capture-change-mysql-state-update-interval | 0 seconds | | DEPRECATED. This property is no longer used and exists solely for backward compatibility purposes. Indicates how often to update the processor's state with binlog file/position values. A value of zero means that state will only be updated when the processor is stopped or shutdown. If at some point the processor state does not contain the desired binlog values, the last flow file emitted will contain the last observed values, and the processor can be returned to that state by using the Initial Binlog File, Initial Binlog Position, and Initial Sequence ID properties.
Supports Expression Language: true (will be evaluated using variable registry only) |
| Initial Sequence ID | capture-change-mysql-init-seq-id | | | Specifies an initial sequence identifier to use if this processor's State does not have a current sequence identifier. If a sequence identifier is present in the processor's State, this property is ignored. Sequence identifiers are monotonically increasing integers that record the order of flow files generated by the processor. They can be used with the EnforceOrder processor to guarantee ordered delivery of CDC events.
Supports Expression Language: true (will be evaluated using variable registry only) |
| Initial Binlog Filename | capture-change-mysql-init-binlog-filename | | | Specifies an initial binlog filename to use if this processor's State does not have a current binlog filename. If a filename is present in the processor's State or "Use GTID" property is set to false, this property is ignored. This can be used along with Initial Binlog Position to "skip ahead" if previous events are not desired. Note that NiFi Expression Language is supported, but this property is evaluated when the processor is configured, so FlowFile attributes may not be used. Expression Language is supported to enable the use of the Variable Registry and/or environment properties.
Supports Expression Language: true (will be evaluated using variable registry only) |
| Initial Binlog Position | capture-change-mysql-init-binlog-position | | | Specifies an initial offset into a binlog (specified by Initial Binlog Filename) to use if this processor's State does not have a current binlog filename. If a filename is present in the processor's State or "Use GTID" property is false, this property is ignored. This can be used along with Initial Binlog Filename to "skip ahead" if previous events are not desired. Note that NiFi Expression Language is supported, but this property is evaluated when the processor is configured, so FlowFile attributes may not be used. Expression Language is supported to enable the use of the Variable Registry and/or environment properties.
Supports Expression Language: true (will be evaluated using variable registry only) |
| Use Binlog GTID | capture-change-mysql-use-gtid | false | | Specifies whether to use Global Transaction ID (GTID) for binlog tracking. If set to true, processor's state of binlog file name and position is ignored. The main benefit of using GTID is to have much reliable failover than using binlog filename/position. |
| Initial Binlog GTID | capture-change-mysql-init-gtid | | | Specifies an initial GTID to use if this processor's State does not have a current GTID. If a GTID is present in the processor's State or "Use GTID" property is set to false, this property is ignored. This can be used to "skip ahead" if previous events are not desired. Note that NiFi Expression Language is supported, but this property is evaluated when the processor is configured, so FlowFile attributes may not be used. Expression Language is supported to enable the use of the Variable Registry and/or environment properties.
Supports Expression Language: true (will be evaluated using variable registry only) |
| SSL Mode | SSL Mode | DISABLED | - DISABLED
- PREFERRED
- REQUIRED
- VERIFY_IDENTITY
| SSL Mode used when SSL Context Service configured supporting certificate verification options |
| SSL Context Service | SSL Context Service | | Controller Service API:
SSLContextService
Implementations: StandardRestrictedSSLContextService
StandardSSLContextService | SSL Context Service supporting encrypted socket communication
This Property is only considered if the [SSL Mode] Property is set to one of the following values: [PREFERRED], [VERIFY_IDENTITY], [REQUIRED] |