OpenTelemetryReportingTask 2.11.0.4.13.0.0-948

Bundle
com.cloudera | nifi-cdf-opentelemetry-nar
Description
Sends NiFi and JVM metrics and status to an OpenTelemetry endpoint that can be used for external monitoring of the system.
Tags
metrics, opentelemetry, reporting, telemetry, time series
Input Requirement
Supports Sensitive Dynamic Properties
false
  • Additional Details for OpenTelemetryReportingTask 2.11.0.4.13.0.0-948

    OpenTelemetryReportingTask

    Adding The Reporting Task to NiFi

    To add an OpenTelemetry Reporting Task to NiFi, open Controller Settings –> Reporting Tasks, then click the [+] button and select OpenTelemetryReportingTask from the list.

    Exported Metrics

    The OpenTelemetry Reporting Task reports metrics for:

    • Connections
    • Ports (Input and Output)
    • Process Groups
    • Processors
    • Remote Process Groups
    • Java Virtual Machine
    • CPU
    • Storage

    Metric Time Window

    Most component metrics are not cumulative since NiFi started. The throughput metrics for Connections, Ports, Process Groups, Processors, and Remote Process Groups describe activity over a rolling 5-minute window. This is a NiFi default (the same window NiFi uses for the In / Out / Read / Write statistics shown in its UI) and cannot be configured from this reporting task.

    For example, connection.input.count reports how many FlowFiles entered the connection during the last 5 minutes, not since the flow began. When a component is idle for more than 5 minutes, these metrics fall back toward 0.

    A subset of metrics are instead point-in-time snapshots captured when the reporting task runs (for example, current queue depth, active thread counts, run/transmission status, and configured thresholds). The Type column in each component’s metrics table below indicates which is which.

    Note: JVM, CPU, and Storage metrics are instantaneous readings or cumulative counters and are not subject to this 5-minute window.

    Connections

    Metrics

    The Type column indicates how each metric is measured:

    • Windowed — the value covers only the last 5 minutes of activity. This is a NiFi default (the same rolling window NiFi shows in its UI) and cannot be configured from this reporting task; the value falls back toward 0 when the component is idle.
    • Snapshot — a point-in-time reading captured when the reporting task runs.
    Metric Name Description Unit Type
    connection.input.bytes Size of Items Input bytes Windowed
    connection.input.count Count of Items Input items Windowed
    connection.output.bytes Size of Items Output bytes Windowed
    connection.output.count Count of Items Output items Windowed
    connection.queued.bytes Size of Items Queued bytes Snapshot
    connection.queued.bytes.max Max Size of Items Queued bytes Snapshot
    connection.queued.count Count of Items Queued items Snapshot
    connection.queued.count.max Max Count of Items Queued items Snapshot
    connection.queued.duration.total Total Duration of Queued Items milliseconds Snapshot
    connection.queued.duration.max Max Duration of Queued Items milliseconds Snapshot
    connection.backpressure.threshold.bytes The maximum size of data in bytes that can be queued in this connection before it applies back pressure. bytes Snapshot
    connection.backpressure.threshold.objects The configured maximum number of FlowFiles that can be queued in this connection before it applies back pressure. items Snapshot
    connection.loadbalance.status.load_balance_not_configured 1 if the connection does not have a configured load balance setting. Otherwise, 0. binary Snapshot
    connection.loadbalance.status.load_balance_active 1 if the connection is load balancing across the cluster. Otherwise, 0. binary Snapshot
    connection.loadbalance.status.load_balance_inactive 1 if the connection is not load balancing across the cluster. Otherwise, 0. binary Snapshot

    Attributes

    Each Connection metric includes the following attributes:

    Attribute Description
    id The unique ID of the connection
    name The name of the connection
    type The fixed value of connection
    source.id The unique ID of the component sending FlowFiles to this connection
    source.name The name of the component sending FlowFiles to this connection
    destination.id The unique ID of the component receiving FlowFiles from this connection
    destination.name The name of the component receiving FlowFiles from this connection
    group.id The unique ID of the Process Group this Connection resides in

    Ports (Input and Output)

    Metrics

    The Type column indicates how each metric is measured:

    • Windowed — the value covers only the last 5 minutes of activity. This is a NiFi default (the same rolling window NiFi shows in its UI) and cannot be configured from this reporting task; the value falls back toward 0 when the component is idle.
    • Snapshot — a point-in-time reading captured when the reporting task runs.
    Metric Name Description Unit Type
    port.thread.count.active Number of Active Threads threads Snapshot
    port.bytes.received Number of Bytes Received bytes Windowed
    port.bytes.sent Number of Bytes Sent bytes Windowed
    port.flowfiles.received Number of FlowFiles Received flowfiles Windowed
    port.flowfiles.sent Number of FlowFiles Sent flowfiles Windowed
    port.input.bytes Size of Items Input bytes Windowed
    port.input.count Count of Items Input items Windowed
    port.output.bytes Size of Items Output bytes Windowed
    port.output.count Count of Items Output items Windowed

    Attributes

    Each Port metric includes the following attributes:

    Attribute Description
    id The unique ID of the port
    name The name of the port
    type port-input or port-output
    group.id The unique ID of the Process Group that this Port resides in

    Process Groups

    Metrics

    The Type column indicates how each metric is measured:

    • Windowed — the value covers only the last 5 minutes of activity. This is a NiFi default (the same rolling window NiFi shows in its UI) and cannot be configured from this reporting task; the value falls back toward 0 when the component is idle.
    • Snapshot — a point-in-time reading captured when the reporting task runs.
    Metric Name Description Unit Type
    processgroup.thread.count.active Number of Active Threads threads Snapshot
    processgroup.thread.count.stateless Number of Stateless Threads threads Snapshot
    processgroup.thread.count.terminated Number of Terminated Threads threads Snapshot
    processgroup.bytes.read Number of Bytes Read bytes Windowed
    processgroup.bytes.received Number of Bytes Received bytes Windowed
    processgroup.bytes.transferred Number of Bytes Transferred bytes Windowed
    processgroup.bytes.sent Number of Bytes Sent bytes Windowed
    processgroup.bytes.written Number of Bytes Written bytes Windowed
    processgroup.flowfiles.received Number of FlowFiles Received flowfiles Windowed
    processgroup.flowfiles.sent Number of FlowFiles Sent flowfiles Windowed
    processgroup.flowfiles.transferred Number of FlowFiles Transferred flowfiles Windowed
    processgroup.input.count Number of Items Input items Windowed
    processgroup.input.content.size Size of Items Input bytes Windowed
    processgroup.output.count Number of Items Output items Windowed
    processgroup.output.content.size Size of Items Output bytes Windowed
    processgroup.queued.count Number of Items Queued items Snapshot
    processgroup.queued.content.size Size of Items Queued bytes Snapshot
    processgroup.time.processing Time Spent Processing nanoseconds Windowed

    Attributes

    Each Process Group metric includes the following attributes:

    Attribute Description
    id The unique ID of the Process Group
    name The name of the Process Group
    type The fixed value process-group

    Processors

    Metrics

    The Type column indicates how each metric is measured:

    • Windowed — the value covers only the last 5 minutes of activity. This is a NiFi default (the same rolling window NiFi shows in its UI) and cannot be configured from this reporting task; the value falls back toward 0 when the component is idle.
    • Snapshot — a point-in-time reading captured when the reporting task runs.
    • Cumulative — the value accumulates over the processor’s lifetime rather than resetting each window.
    Metric Name Description Unit Type
    processor.thread.count.active Number of Active Threads threads Snapshot
    processor.thread.count.terminated Number of Terminated Threads threads Snapshot
    processor.time.lineage.average Average Lineage Duration nanoseconds Windowed
    processor.invocations Number of Invocations invocations Windowed
    processor.bytes.read Number of Bytes Read bytes Windowed
    processor.bytes.received Number of Bytes Received bytes Windowed
    processor.bytes.sent Number of Bytes Sent bytes Windowed
    processor.bytes.written Number of Bytes Written bytes Windowed
    processor.flowfiles.received Number of FlowFiles Received flowfiles Windowed
    processor.flowfiles.removed Number of FlowFiles Removed flowfiles Windowed
    processor.flowfiles.sent Number of FlowFiles Sent flowfiles Windowed
    processor.input.count Number of Items Input items Windowed
    processor.input.content.size Size of Items Input bytes Windowed
    processor.output.count Number of Items Output items Windowed
    processor.output.content.size Size of Items Output bytes Windowed
    processor.time.processing Time Spent Processing nanoseconds Windowed
    processor.run.status.running 1 if running; 0 otherwise binary Snapshot
    processor.run.status.stopped 1 if stopped; 0 otherwise binary Snapshot
    processor.run.status.validating 1 if validating; 0 otherwise binary Snapshot
    processor.run.status.invalid 1 if invalid; 0 otherwise binary Snapshot
    processor.run.status.disabled 1 if disabled; 0 otherwise binary Snapshot
    processor.counter Value of the counter count Cumulative

    Attributes

    Each Processor metric includes the following attributes:

    Attribute Description
    id The unique ID of the processor
    name The name of the Processor
    type The fixed value processor
    execution.node Either ALL or PRIMARY, depending on how this Processor is configured to run
    group.id The unique ID of the Process Group that this Processor resides in
    Additional Attributes for Counters

    In addition to the attributes above, processor.counter metrics include the following:

    Attribute Description
    type The fixed value counter
    counter The name of the counter

    Remote Process Groups

    Metrics

    The Type column indicates how each metric is measured:

    • Windowed — the value covers only the last 5 minutes of activity. This is a NiFi default (the same rolling window NiFi shows in its UI) and cannot be configured from this reporting task; the value falls back toward 0 when the component is idle.
    • Snapshot — a point-in-time reading captured when the reporting task runs.
    Metric Name Description Unit Type
    remoteprocessgroup.thread.count.active Number of Active Threads threads Snapshot
    remoteprocessgroup.remote.port.count.active Number of Active Remote Ports ports Snapshot
    remoteprocessgroup.remote.port.count.inactive Number of Inactive Remote Ports ports Snapshot
    remoteprocessgroup.duration.lineage.average Average Lineage Duration nanoseconds Windowed
    remoteprocessgroup.refresh.age Time since Last Refresh milliseconds Snapshot
    remoteprocessgroup.received.count Number of Received Items items Windowed
    remoteprocessgroup.received.content.size Size of Received Items bytes Windowed
    remoteprocessgroup.sent.count Number of Sent Items items Windowed
    remoteprocessgroup.sent.content.size Size of Sent Items bytes Windowed
    remoteprocessgroup.transmission.status.transmitting 1 if the Remote Process Group is transmitting. Otherwise, 0. binary Snapshot
    remoteprocessgroup.transmission.status.nottransmitting 0 if the Remote Process Group is transmitting. Otherwise, 1. binary Snapshot

    Attributes

    Each Remote Process Group metric includes the following attributes:

    Attribute Description
    id The unique ID of the Remote Process Group
    name The name of the Remote Process Group
    group.id The unique ID of the Process Group that this Remote Process Group resides in
    authorization.issue The Authorization used to access the Remote Process Group
    target.uri The URI of the Remote Process Group
    type The fixed value remote-process-group

    JVM Metrics

    The OpenTelemetry Reporting Task exports the following JVM metrics:

    JVM Metric Name Description Unit
    jvm.memory.heap.used JVM Heap Used bytes
    jvm.memory.heap.committed JVM Heap Committed bytes
    jvm.memory.heap.max JVM Heap Max bytes
    jvm.memory.heap.init JVM Heap Init bytes
    jvm.memory.heap.usage JVM Heap Usage percentage
    jvm.memory.non-heap.usage JVM Non-Heap Usage percentage
    jvm.memory.total.init JVM Memory Total Init bytes
    jvm.memory.total.used JVM Memory Total Used bytes
    jvm.memory.total.max JVM Memory Total Max bytes
    jvm.memory.total.committed JVM Memory Total Committed bytes
    jvm.threads.count Number of Live Threads threads
    jvm.threads.deadlocks JVM Thread Deadlocks threads
    jvm.threads.daemon.count Number of Live Daemon Threads threads
    jvm.uptime Number of Seconds the JVM Process Has Been Running seconds
    jvm.file.descriptor.usage Percentage of Available File Descriptors Currently in Use percentage
    jvm.gc.<collector>.runs Total number of times that the <collector> garbage collection has run, where <collector> is replaced with the actual name of the garbage collector runs
    jvm.gc.<collector>.time Total amount of time that the <collector> garbage collection has been running, where <collector> is replaced with the actual name of the garbage collector milliseconds

    CPU Metrics

    CPU Metric Name Description Unit
    cores.available The Number of Available Cores for the Runtime cores
    cores.load Either the System Load Average or -1 If It Is Not Available threads

    Attributes

    Each CPU metric includes the following attributes:

    Attribute Description
    id The fixed value cpu
    name The name of the operating system
    architecture The architecture of the operating system
    version The version of the operating system

    Storage Metrics

    Storage metrics are reported for each repository (FlowFile, Content, Provenance).

    Storage Metric Name Description Unit
    storage.free The amount of free storage for the repository bytes
    storage.used The amount of storage used by the repository bytes

    Attributes

    Each storage metric includes the following attributes:

    Attribute Description
    id The unique identifier of the storage repository
    name Same as “id” and provided for consistency
    storage.type One of flowfile, content, or provenance
Properties
Dynamic Properties