CaptureChangeMySQL

CaptureChangeMySQL 2.0.0

Bundle: org.apache.nifi | nifi-cdc-mysql-nar
Description: Retrieves Change Data Capture (CDC) events from a MySQL database. CDC Events include INSERT, UPDATE, DELETE operations. Events are output as either a group of a specified number of events (the default is 1 so each event becomes its own flow file) or grouped as a full transaction (BEGIN to COMMIT). All events are ordered by the time at which the operation occurred. NOTE: If the processor is stopped before the specified number of events have been written to a flow file, the partial flow file will be output in order to maintain the consistency of the event stream.
Tags: cdc, event, jdbc, mysql, sql, transaction
Input Requirement: FORBIDDEN
Supports Sensitive Dynamic Properties: false

Properties

Database/Schema Name Pattern
A regular expression (regex) for matching databases (or schemas, depending on your RDBMS' terminology) against the list of CDC events. The regex must match the database name as it is stored in the RDBMS. If the property is not set, the database name will not be used to filter the CDC events. NOTE: DDL events, even if they affect different databases, are associated with the database used by the session to execute the DDL. This means if a connection is made to one database, but the DDL is issued against another, then the connected database will be the one matched against the specified pattern.

Display Name

Database/Schema Name Pattern

Description

A regular expression (regex) for matching databases (or schemas, depending on your RDBMS' terminology) against the list of CDC events. The regex must match the database name as it is stored in the RDBMS. If the property is not set, the database name will not be used to filter the CDC events. NOTE: DDL events, even if they affect different databases, are associated with the database used by the session to execute the DDL. This means if a connection is made to one database, but the DDL is issued against another, then the connected database will be the one matched against the specified pattern.

API Name

capture-change-mysql-db-name-pattern

Expression Language Scope

Not Supported

Sensitive

false

Required

false
Distributed Map Cache Client - unused
This is a legacy property that is no longer used to store table information, the processor will handle the table information (column names, types, etc.)

Display Name

Distributed Map Cache Client - unused

Description

This is a legacy property that is no longer used to store table information, the processor will handle the table information (column names, types, etc.)

API Name

capture-change-mysql-dist-map-cache-client

Service Interface

org.apache.nifi.distributed.cache.client.DistributedMapCacheClient

Service Implementations

org.apache.nifi.hazelcast.services.cacheclient.HazelcastMapCacheClient

org.apache.nifi.distributed.cache.client.MapCacheClientService

org.apache.nifi.redis.service.RedisDistributedMapCacheClientService

org.apache.nifi.redis.service.SimpleRedisDistributedMapCacheClientService

Expression Language Scope

Not Supported

Sensitive

false

Required

false
MySQL Driver Class Name
The class name of the MySQL database driver class

Display Name

MySQL Driver Class Name

Description

The class name of the MySQL database driver class

API Name

capture-change-mysql-driver-class

Default Value

com.mysql.jdbc.Driver

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

true
MySQL Driver Location(s)
Comma-separated list of files/folders and/or URLs containing the MySQL driver JAR and its dependencies (if any). For example '/var/tmp/mysql-connector-java-5.1.38-bin.jar'

Display Name

MySQL Driver Location(s)

Description

Comma-separated list of files/folders and/or URLs containing the MySQL driver JAR and its dependencies (if any). For example '/var/tmp/mysql-connector-java-5.1.38-bin.jar'

API Name

capture-change-mysql-driver-locations

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

false
MySQL Nodes
A list of hostname (and optional port) entries corresponding to nodes in a MySQL cluster. The entries should be comma separated using a colon (if the port is to be specified) such as host1:port,host2:port,.... For example mysql.myhost.com:3306. The port need not be specified, when omitted the default MySQL port value of 3306 will be used. This processor will attempt to connect to the hosts in the list in order. If one node goes down and failover is enabled for the cluster, then the processor will connect to the active node (assuming its node entry is specified in this property).

Display Name

MySQL Nodes

Description

A list of hostname (and optional port) entries corresponding to nodes in a MySQL cluster. The entries should be comma separated using a colon (if the port is to be specified) such as host1:port,host2:port,.... For example mysql.myhost.com:3306. The port need not be specified, when omitted the default MySQL port value of 3306 will be used. This processor will attempt to connect to the hosts in the list in order. If one node goes down and failover is enabled for the cluster, then the processor will connect to the active node (assuming its node entry is specified in this property).

API Name

capture-change-mysql-hosts

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

true
Include Begin/Commit Events
Specifies whether to emit events corresponding to a BEGIN or COMMIT event in the binary log. Set to true if the BEGIN/COMMIT events are necessary in the downstream flow, otherwise set to false, which suppresses generation of these events and can increase flow performance.
Display Name

Include Begin/Commit Events

Description

Specifies whether to emit events corresponding to a BEGIN or COMMIT event in the binary log. Set to true if the BEGIN/COMMIT events are necessary in the downstream flow, otherwise set to false, which suppresses generation of these events and can increase flow performance.

API Name

capture-change-mysql-include-begin-commit

Default Value

false

Allowable Values
- true
- false
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Include DDL Events
Specifies whether to emit events corresponding to Data Definition Language (DDL) events such as ALTER TABLE, TRUNCATE TABLE, e.g. in the binary log. Set to true if the DDL events are desired/necessary in the downstream flow, otherwise set to false, which suppresses generation of these events and can increase flow performance.
Display Name

Include DDL Events

Description

Specifies whether to emit events corresponding to Data Definition Language (DDL) events such as ALTER TABLE, TRUNCATE TABLE, e.g. in the binary log. Set to true if the DDL events are desired/necessary in the downstream flow, otherwise set to false, which suppresses generation of these events and can increase flow performance.

API Name

capture-change-mysql-include-ddl-events

Default Value

false

Allowable Values
- true
- false
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Initial Binlog Filename
Specifies an initial binlog filename to use if this processor's State does not have a current binlog filename. If a filename is present in the processor's State or "Use GTID" property is set to false, this property is ignored. This can be used along with Initial Binlog Position to "skip ahead" if previous events are not desired. Note that NiFi Expression Language is supported, but this property is evaluated when the processor is configured, so FlowFile attributes may not be used. Expression Language is supported to enable the use of the environment properties.

Display Name

Initial Binlog Filename

Description

Specifies an initial binlog filename to use if this processor's State does not have a current binlog filename. If a filename is present in the processor's State or "Use GTID" property is set to false, this property is ignored. This can be used along with Initial Binlog Position to "skip ahead" if previous events are not desired. Note that NiFi Expression Language is supported, but this property is evaluated when the processor is configured, so FlowFile attributes may not be used. Expression Language is supported to enable the use of the environment properties.

API Name

capture-change-mysql-init-binlog-filename

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

false
Initial Binlog Position
Specifies an initial offset into a binlog (specified by Initial Binlog Filename) to use if this processor's State does not have a current binlog filename. If a filename is present in the processor's State or "Use GTID" property is false, this property is ignored. This can be used along with Initial Binlog Filename to "skip ahead" if previous events are not desired. Note that NiFi Expression Language is supported, but this property is evaluated when the processor is configured, so FlowFile attributes may not be used. Expression Language is supported to enable the use of the environment properties.

Display Name

Initial Binlog Position

Description

Specifies an initial offset into a binlog (specified by Initial Binlog Filename) to use if this processor's State does not have a current binlog filename. If a filename is present in the processor's State or "Use GTID" property is false, this property is ignored. This can be used along with Initial Binlog Filename to "skip ahead" if previous events are not desired. Note that NiFi Expression Language is supported, but this property is evaluated when the processor is configured, so FlowFile attributes may not be used. Expression Language is supported to enable the use of the environment properties.

API Name

capture-change-mysql-init-binlog-position

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

false
Initial Binlog GTID
Specifies an initial GTID to use if this processor's State does not have a current GTID. If a GTID is present in the processor's State or "Use GTID" property is set to false, this property is ignored. This can be used to "skip ahead" if previous events are not desired. Note that NiFi Expression Language is supported, but this property is evaluated when the processor is configured, so FlowFile attributes may not be used. Expression Language is supported to enable the use of the environment properties.

Display Name

Initial Binlog GTID

Description

Specifies an initial GTID to use if this processor's State does not have a current GTID. If a GTID is present in the processor's State or "Use GTID" property is set to false, this property is ignored. This can be used to "skip ahead" if previous events are not desired. Note that NiFi Expression Language is supported, but this property is evaluated when the processor is configured, so FlowFile attributes may not be used. Expression Language is supported to enable the use of the environment properties.

API Name

capture-change-mysql-init-gtid

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

false
Initial Sequence ID
Specifies an initial sequence identifier to use if this processor's State does not have a current sequence identifier. If a sequence identifier is present in the processor's State, this property is ignored. Sequence identifiers are monotonically increasing integers that record the order of flow files generated by the processor. They can be used with the EnforceOrder processor to guarantee ordered delivery of CDC events.

Display Name

Initial Sequence ID

Description

Specifies an initial sequence identifier to use if this processor's State does not have a current sequence identifier. If a sequence identifier is present in the processor's State, this property is ignored. Sequence identifiers are monotonically increasing integers that record the order of flow files generated by the processor. They can be used with the EnforceOrder processor to guarantee ordered delivery of CDC events.

API Name

capture-change-mysql-init-seq-id

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

false
Max Wait Time
The maximum amount of time allowed for a connection to be established, zero means there is effectively no limit.

Display Name

Max Wait Time

Description

The maximum amount of time allowed for a connection to be established, zero means there is effectively no limit.

API Name

capture-change-mysql-max-wait-time

Default Value

30 seconds

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

true
Table Name Pattern
A regular expression (regex) for matching CDC events affecting matching tables. The regex must match the table name as it is stored in the database. If the property is not set, no events will be filtered based on table name.

Display Name

Table Name Pattern

Description

A regular expression (regex) for matching CDC events affecting matching tables. The regex must match the table name as it is stored in the database. If the property is not set, no events will be filtered based on table name.

API Name

capture-change-mysql-name-pattern

Expression Language Scope

Not Supported

Sensitive

false

Required

false
Password
Password to access the MySQL cluster

Display Name

Password

Description

Password to access the MySQL cluster

API Name

capture-change-mysql-password

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

true

Required

false
Retrieve All Records
Specifies whether to get all available CDC events, regardless of the current binlog filename and/or position. If binlog filename and position values are present in the processor's State, this property's value is ignored. This allows for 4 different configurations: 1) If binlog data is available in processor State, that is used to determine the start location and the value of Retrieve All Records is ignored. 2) If no binlog data is in processor State, then Retrieve All Records set to true means start at the beginning of the binlog history. 3) If no binlog data is in processor State and Initial Binlog Filename/Position are not set, then Retrieve All Records set to false means start at the end of the binlog history. 4) If no binlog data is in processor State and Initial Binlog Filename/Position are set, then Retrieve All Records set to false means start at the specified initial binlog file/position. To reset the behavior, clear the processor state (refer to the State Management section of the processor's documentation).
Display Name

Retrieve All Records

Description

Specifies whether to get all available CDC events, regardless of the current binlog filename and/or position. If binlog filename and position values are present in the processor's State, this property's value is ignored. This allows for 4 different configurations: 1) If binlog data is available in processor State, that is used to determine the start location and the value of Retrieve All Records is ignored. 2) If no binlog data is in processor State, then Retrieve All Records set to true means start at the beginning of the binlog history. 3) If no binlog data is in processor State and Initial Binlog Filename/Position are not set, then Retrieve All Records set to false means start at the end of the binlog history. 4) If no binlog data is in processor State and Initial Binlog Filename/Position are set, then Retrieve All Records set to false means start at the specified initial binlog file/position. To reset the behavior, clear the processor state (refer to the State Management section of the processor's documentation).

API Name

capture-change-mysql-retrieve-all-records

Default Value

true

Allowable Values
- true
- false
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Server ID
The client connecting to the MySQL replication group is actually a simplified replica (server), and the Server ID value must be unique across the whole replication group (i.e. different from any other Server ID being used by any primary or replica). Thus, each instance of CaptureChangeMySQL must have a Server ID unique across the replication group. If the Server ID is not specified, it defaults to 65535.

Display Name

Server ID

Description

The client connecting to the MySQL replication group is actually a simplified replica (server), and the Server ID value must be unique across the whole replication group (i.e. different from any other Server ID being used by any primary or replica). Thus, each instance of CaptureChangeMySQL must have a Server ID unique across the replication group. If the Server ID is not specified, it defaults to 65535.

API Name

capture-change-mysql-server-id

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

false
Use Binlog GTID
Specifies whether to use Global Transaction ID (GTID) for binlog tracking. If set to true, processor's state of binlog file name and position is ignored. The main benefit of using GTID is to have much reliable failover than using binlog filename/position.
Display Name

Use Binlog GTID

Description

Specifies whether to use Global Transaction ID (GTID) for binlog tracking. If set to true, processor's state of binlog file name and position is ignored. The main benefit of using GTID is to have much reliable failover than using binlog filename/position.

API Name

capture-change-mysql-use-gtid

Default Value

false

Allowable Values
- true
- false
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Username
Username to access the MySQL cluster

Display Name

Username

Description

Username to access the MySQL cluster

API Name

capture-change-mysql-username

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

false
Event Processing Strategy
Specifies the strategy to use when writing events to FlowFile(s), such as 'Max Events Per FlowFile'
Display Name

Event Processing Strategy

Description

Specifies the strategy to use when writing events to FlowFile(s), such as 'Max Events Per FlowFile'

API Name

events-per-flowfile-strategy

Default Value

MAX_EVENTS_PER_FLOWFILE

Allowable Values
- Max Events Per FlowFile
- One Transaction Per FlowFile
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Events Per FlowFile
Specifies how many events should be written to a single FlowFile. If the processor is stopped before the specified number of events has been written,the events will still be written as a FlowFile before stopping.
Display Name

Events Per FlowFile

Description

Specifies how many events should be written to a single FlowFile. If the processor is stopped before the specified number of events has been written,the events will still be written as a FlowFile before stopping.

API Name

number-of-events-per-flowfile

Default Value

1

Expression Language Scope

Environment variables defined at JVM level and system properties

Sensitive

false

Required

true

Dependencies
- Event Processing Strategy is set to any of [MAX_EVENTS_PER_FLOWFILE]
SSL Context Service
SSL Context Service supporting encrypted socket communication
Display Name

SSL Context Service

Description

SSL Context Service supporting encrypted socket communication

API Name

SSL Context Service

Service Interface

org.apache.nifi.ssl.SSLContextService

Service Implementations

org.apache.nifi.ssl.StandardRestrictedSSLContextService

org.apache.nifi.ssl.StandardSSLContextService

Expression Language Scope

Not Supported

Sensitive

false

Required

false

Dependencies
- SSL Mode is set to any of [PREFERRED, VERIFY_IDENTITY, REQUIRED]
SSL Mode
SSL Mode used when SSL Context Service configured supporting certificate verification options
Display Name

SSL Mode

Description

SSL Mode used when SSL Context Service configured supporting certificate verification options

API Name

SSL Mode

Default Value

DISABLED

Allowable Values
- DISABLED
- PREFERRED
- REQUIRED
- VERIFY_IDENTITY
Expression Language Scope

Not Supported

Sensitive

false

Required

true

State Management

Scopes	Description
CLUSTER	Information such as a 'pointer' to the current CDC event in the database is stored by this processor, such that it can continue from the same location if restarted.

Restrictions

Required Permission	Explanation
reference remote resources	Database Driver Location can reference resources over HTTP

Relationships

Name	Description
success	Successfully created FlowFile from SQL query result set.

Writes Attributes

Name	Description
cdc.sequence.id	A sequence identifier (i.e. strictly increasing integer value) specifying the order of the CDC event flow file relative to the other event flow file(s).
cdc.event.type	A string indicating the type of CDC event that occurred, including (but not limited to) 'begin', 'insert', 'update', 'delete', 'ddl' and 'commit'.
mime.type	The processor outputs flow file content in JSON format, and sets the mime.type attribute to application/json