HadoopDBCPConnectionPool

Description:

Provides a Database Connection Pooling Service for Hadoop related JDBC services. This service requires that the Database Driver Location(s) contains some version of a hadoop-common JAR, or a shaded JAR that shades hadoop-common.

Tags:

dbcp, jdbc, database, connection, pooling, store, hadoop

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Database Connection URLDatabase Connection URLA database connection URL used to connect to a database. May contain database system name, host, port, database name and some parameters. The exact syntax of a database connection URL is specified by your DBMS.
Supports Expression Language: true (will be evaluated using variable registry only)
Database Driver Class NameDatabase Driver Class NameDatabase driver class name
Supports Expression Language: true (will be evaluated using variable registry only)
Database Driver Location(s)database-driver-locationsComma-separated list of files/folders and/or URLs containing the driver JAR and its dependencies (if any). For example '/var/tmp/phoenix-client.jar'. NOTE: It is required that the resources specified by this property provide the classes from hadoop-common, such as Configuration and UserGroupInformation.

This property expects a comma-separated list of resources. Each of the resources may be of any of the following types: directory, URL, file.

Supports Expression Language: true (will be evaluated using variable registry only)
Hadoop Configuration Resourceshadoop-config-resourcesA file, or comma separated list of files, which contain the Hadoop configuration (core-site.xml, etc.). Without this, Hadoop will search the classpath, or will revert to a default configuration. Note that to enable authentication with Kerberos, the appropriate properties must be set in the configuration files.

This property expects a comma-separated list of file resources.

Supports Expression Language: true (will be evaluated using variable registry only)
Kerberos User Servicekerberos-user-serviceController Service API:
KerberosUserService
Implementations: KerberosKeytabUserService
KerberosPasswordUserService
KerberosTicketCacheUserService
Specifies the Kerberos User Controller Service that should be used for authenticating with Kerberos
Kerberos Credentials Servicekerberos-credentials-serviceController Service API:
KerberosCredentialsService
Implementation: KeytabCredentialsService
Specifies the Kerberos Credentials Controller Service that should be used for authenticating with Kerberos
Kerberos PrincipalKerberos PrincipalKerberos principal to authenticate as. Requires nifi.kerberos.krb5.file to be set in your nifi.properties
Supports Expression Language: true (will be evaluated using variable registry only)
Kerberos KeytabKerberos KeytabKerberos keytab associated with the principal. Requires nifi.kerberos.krb5.file to be set in your nifi.properties

This property requires exactly one file to be provided..

Supports Expression Language: true (will be evaluated using variable registry only)
Kerberos PasswordKerberos PasswordKerberos password associated with the principal.
Sensitive Property: true
Database UserDatabase UserThe user for the database
Supports Expression Language: true (will be evaluated using variable registry only)
PasswordPasswordThe password for the database user
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only)
Max Wait TimeMax Wait Time500 millisThe maximum amount of time that the pool will wait (when there are no available connections) for a connection to be returned before failing, or -1 to wait indefinitely.
Supports Expression Language: true (will be evaluated using variable registry only)
Max Total ConnectionsMax Total Connections8The maximum number of active connections that can be allocated from this pool at the same time, or negative for no limit.
Supports Expression Language: true (will be evaluated using variable registry only)
Validation queryValidation-queryValidation query used to validate connections before returning them. When connection is invalid, it get's dropped and new valid connection will be returned. Note!! Using validation might have some performance penalty.
Supports Expression Language: true (will be evaluated using variable registry only)
Minimum Idle Connectionsdbcp-min-idle-conns0The minimum number of connections that can remain idle in the pool, without extra ones being created, or zero to create none.
Supports Expression Language: true (will be evaluated using variable registry only)
Max Idle Connectionsdbcp-max-idle-conns8The maximum number of connections that can remain idle in the pool, without extra ones being released, or negative for no limit.
Supports Expression Language: true (will be evaluated using variable registry only)
Max Connection Lifetimedbcp-max-conn-lifetime-1The maximum lifetime in milliseconds of a connection. After this time is exceeded the connection will fail the next activation, passivation or validation test. A value of zero or less means the connection has an infinite lifetime.
Supports Expression Language: true (will be evaluated using variable registry only)
Time Between Eviction Runsdbcp-time-between-eviction-runs-1The number of milliseconds to sleep between runs of the idle connection evictor thread. When non-positive, no idle connection evictor thread will be run.
Supports Expression Language: true (will be evaluated using variable registry only)
Minimum Evictable Idle Timedbcp-min-evictable-idle-time30 minsThe minimum amount of time a connection may sit idle in the pool before it is eligible for eviction.
Supports Expression Language: true (will be evaluated using variable registry only)
Soft Minimum Evictable Idle Timedbcp-soft-min-evictable-idle-time-1The minimum amount of time a connection may sit idle in the pool before it is eligible for eviction by the idle connection evictor, with the extra condition that at least a minimum number of idle connections remain in the pool. When the not-soft version of this option is set to a positive value, it is examined first by the idle connection evictor: when idle connections are visited by the evictor, idle time is first compared against it (without considering the number of idle connections in the pool) and then against this soft option, including the minimum idle connections constraint.
Supports Expression Language: true (will be evaluated using variable registry only)

Dynamic Properties:

Supports Sensitive Dynamic Properties: No

Dynamic Properties allow the user to specify both the name and value of a property.

NameValueDescription
The name of a Hadoop configuration property.The value of the given Hadoop configuration property.These properties will be set on the Hadoop configuration after loading any provided configuration files.
Supports Expression Language: true (will be evaluated using variable registry only)

State management:

This component does not store state.

Restricted:

Required PermissionExplanation
reference remote resourcesDatabase Driver Location can reference resources over HTTP

System Resource Considerations:

None specified.