Configuring your target processor

In a data flow that is pulling data from a Snowflake database, configure the ExecuteSQLRecord processor to handle data pooling from remote tables. To do this, launch the processor configuration window and provide the configurations appropriate for your use case.

Configure ExecuteSQLRecord.

The ExecuteSQLRecord processor handles data pooling from the remote tables. To load balance the execution of the SQL queries across the NiFi nodes, you should update the configuration of the relationship between the ListDatabaseTables processor and the ExecuteSQLRecord processor and define a round robin load balancing strategy. This way the flow files generated on the primary node by the ListDatabaseTables processors will be shuffled across the nodes of the NiFi cluster to distribute the workload.

  1. Right-click ExecuteSQLRecord and click Configure Processor.
  2. For Database Connection Pooling Service, select your DBCPConnectionPool controller service you defined earlier.
  3. For SQL select query, if you want to extract all the data from the remote tables, you can use the SQL query:
    SELECT * FROM ${db.table.fullname}
  4. For Record Writer, select the Avro writer controller service you defined earlier.
  5. For Normalize Table/Column Names, you may want to set this property to true in case some of the column names are not compatible with Avro specifications.

After you have configured your target processor, proceed to confirming your data flow success.