PutSnowflakeStreaming 2.11.0.4.13.0.0-948

Bundle
com.cloudera | nifi-cdf-snowflake-processors-nar
Description
Writes NiFi records to a Snowflake table using the Snowpipe Streaming API, enabling low-latency, continuous data ingestion without staging files.
Tags
database, snowflake, snowpipe, streaming
Input Requirement
REQUIRED
Supports Sensitive Dynamic Properties
false
  • Additional Details for PutSnowflakeStreaming 2.11.0.4.13.0.0-948

    PutSnowflakeStreaming

    Overview

    PutSnowflakeStreaming ingests NiFi records into a Snowflake table using the Snowpipe Streaming API. Unlike file-based ingestion, Snowpipe Streaming appends rows directly to a channel without staging files, resulting in lower latency and more predictable end-to-end delivery times. PLEASE NOTE: There are important restrictions regarding the processor’s connections and the flow design. Please see Additional Details for more information.

    Ingestion Flow

    The processor operates in three modes, and every trigger chooses which one to execute:

    1. Source — This is the default mode. A single FlowFile is dequeued from the input connection, its records are parsed by the configured Record Reader, and the rows are appended to the Snowflake streaming channel. The FlowFile is then routed to the pending relationship with offset metadata attached. The Snowflake client buffers the rows in the background and sends them to Snowflake asynchronously.

    2. Commit — Every Commit Interval, the processor queries Snowflake for the latest committed offset. FlowFiles in the pending queue whose offset has been confirmed are routed to success (all records written) or failure (a read error occurred mid-file, so only a partial set of records was sent).

    3. Cleanup — Periodically, the processor resolves pending FlowFiles that do not belong to the current active channel (e.g. offloaded from another cluster node).

    Connection Requirements

    1. Because FlowFiles pass through the pending relationship before being confirmed by Snowflake, that relationship must be looped back to the processor directly. Additionally, the connection must be configured with FirstInFirstOutPrioritizer to preserve ordering — the processor relies on FlowFiles being processed in the order they were appended. The connection must also be in the default Do not load balance mode.

    2. The processor must have a single input connection, and it must be configured with FirstInFirstOutPrioritizer too.

    [input (FIFO)] ───► PutSnowflakeStreaming ───► [success]
                              ▲           │   └──► [failure]
                              └───────────┘
                               [pending (FIFO)]
    

    Recovery

    When the processor starts (or is re-scheduled after a failure), it enters a recovery phase before normal processing begins. During recovery, it:

    • Re-opens the Snowflake streaming channel and retrieves the latest committed offset (stable offset just after opening the channel and before appending any records).
    • Transfers already-committed pending FlowFiles to success or failure.
    • Re-sends uncommitted pending FlowFiles (fully or partially, as it is necessary) and transfers them to pending again.

    The recovery process handles the pending FlowFiles that belong to the current client/channel only. The client/channel id is generated at first run, stored in the processor’s state and attached to every input FlowFiles as an attribute. FlowFiles with other client ids are handled by the cleanup process.

    Error Handling

    • FlowFile read error — If the FlowFile content is not readable/parsable at all (e.g. due to data format mismatch), the processor routes the FlowFile to failure directly.

    • Record read error (partial delivery) — If parsing a record fails mid-FlowFile (e.g. data type mismatch with the NiFi-side record schema), the processor stops reading at that point. Records already appended are still sent to Snowflake. Once those records are committed, the FlowFile is routed to failure (rather than success). Please note that the processor stops reading at the first error encountered. It does not attempt to continue reading the remaining records in the FlowFile.

    • Client-side ingestion error — If the Snowflake channel rejects a batch of rows on the client side, the session is rolled back and the FlowFile remains in the input queue. When the rejection is caused by exceeding the request rate limit, the channel remains open and the processor waits for the rate limit to be released. For other client-side ingestion errors, the channel is closed and re-initialized later. The processor yields and uses an exponential backoff strategy before retrying.

    • Server-side ingestion error - Snowflake-side errors (e.g. data type mismatch with the table schema) are NOT reported back to the client at record level. This is a limitation of the Snowpipe Streaming API. Therefore, the processor cannot guarantee that all records in a FlowFile are successfully ingested even if the FlowFile is routed to success. In every commit phase, the processor creates an error log if any records failed since the last commit. The log includes the number of failed records and information about the last error. For more details, the service-side error table needs to be checked.

    • Commit timeout — If the oldest pending FlowFile has been waiting longer than Commit Timeout without Snowflake confirming its offset, the processor treats the channel as stuck, raises an error, and closes the channel. It will re-initialize on the next trigger and perform recovery to resolve the pending FlowFiles.

    • Uncommitted offsets during cleanup - The cleanup process never re-sends FlowFiles because those are not originated from the current client/channel (offloaded from another cluster node or created by a previous processor run after which the state was cleared). The cleanup process just makes a one-time decision based on the latest committed offset of the given channel and routes the FlowFiles to success or failure accordingly.

    • FlowFile attributes on failure — Failed FlowFiles carry the following attributes to aid diagnosis:

      Attribute Description
      snowflake.errorMessage Human-readable description of what went wrong
      snowflake.errorIndex 1-based record index where a read error occurred (set only for partial delivery failures)

    Delivery Guarantees

    The processor aims for exactly-once delivery of records to Snowflake. The key mechanisms that enable this are:

    • Offset tokens — Every batch of rows appended to the channel carries a unique, monotonically increasing offset token derived from FlowFile index (generated by the processor), and record index (record position in the given FlowFile). Snowflake uses these tokens to track progress and expose the latest committed offset via getLatestCommittedOffsetToken().

    • Recovery on restart — When the processor (re-)starts, it queries the latest committed offset from Snowflake before doing anything else. FlowFiles already confirmed as committed are transferred to success or failure without being re-sent. FlowFiles, whose offset was not yet committed, are re-sent from scratch (or from the last committed record for partially committed FlowFiles), so no rows are lost.

    • Recovery after rollback — If processing is rolled back, for example due to rate limiting, a forced stop, or a system crash, some records from the FlowFile may already have been sent to Snowflake while others have not. When the FlowFile is picked up again, the processor checks whether it was in progress and resumes processing from the record where it left off. This ensures that no rows are duplicated or lost after rollback.

    • FIFO ordering — The required FirstInFirstOutPrioritizer on the input and pending connections guarantees that FlowFiles are appended and confirmed in the same order they arrived, which is a prerequisite for the offset-based commit tracking to work correctly.

    Exceptions to exactly-once:

    • Record read error mid-FlowFile: Records before the error are sent to Snowflake exactly once, while records from the error point onward are never sent. The FlowFile is routed to failure to signal the partial delivery.

    • Server-side ingestion error: These errors are not reported back to the client at record level, and the latest committed offset still progresses even if the record is not successfully ingested. As a result, a FlowFile routed to success does not guarantee that all records have been ingested. Snowflake-side monitoring is also needed to detect and resolve these errors.

    SSL and Proxy Settings

    PutSnowflakeStreaming does not support configuring SSL or proxy behavior through NiFi’s usual controller services, such as SSL Context Service or Proxy Configuration Service.

    The processor creates a Snowflake Snowpipe Streaming Java client directly. The Java API used by the processor does not provide options for configuring SSL context or proxy settings. As a result, the processor cannot translate NiFi controller-service settings into equivalent Snowflake client configuration.

Properties
State Management
Scopes Description
LOCAL Stores the generated, unique client id specific to this processor instance in the cluster.
Relationships
Name Description
failure FlowFiles that could not be processed due to an error, or that were only partially written (an input error occurred mid-record).
pending FlowFiles that have been appended to the Snowflake streaming channel but whose commit has not yet been confirmed. They are re-queued and transferred to success or failure once Snowflake acknowledges the commit. The connection must be looped back to the processor to continue processing these FlowFiles and FirstInFirstOutPrioritizer must be configured to ensure the FlowFiles are processed properly.
success FlowFiles whose records were fully appended and confirmed committed by Snowflake.
Writes Attributes
Name Description
snowflake.isPending Set to true when records have been appended to the Snowflake streaming channel but their commit has not been confirmed. Removed before routing committed FlowFiles to success or failure.
snowflake.channelName Name of the Snowflake streaming channel to which records from the FlowFile were appended.
snowflake.offsetToken Offset token for the last record appended from the FlowFile. Used to compare against Snowflake's latest committed offset.
snowflake.appendTimestamp Epoch timestamp in milliseconds when the FlowFile records were appended to the Snowflake streaming channel. Used for commit timeout tracking.
snowflake.recoveryId Identifier of the recovery attempt that re-sent the FlowFile after restart or channel reinitialization.
snowflake.errorMessage Error details written when the FlowFile is routed to failure.
snowflake.errorIndex Record index associated with snowflake.errorMessage. The index is 1-based and it is set to 0 when no record could be read.