Cloudera Docs

MiNiFi C++ agent updates

MiNiFi C++ agents has periodic software updates made available independent of the EFM releases.

This section lists major features and updates for the independent CEM MiNiFi C++ Agent release.

March 9, 2023

CEM MiNiFi C++ Agent - 1.23.02 release includes:

Added support for the following processors:

  • ListenUDP
  • PublishMQTT
  • ConsumeMQTT

Changes:

  • Python scripting can now work with multiple python minor versions.
  • Added support for multiple process groups.
  • Decreased the default C2 heartbeat frequency to once every 30 seconds.
  • Optimized memory usage in some high flow file count scenarios.
  • Added ProcessSession::remove to the Python scripting API.
  • Increased the default flow controller thread count (nifi.flow.engine.threads) to 5.
  • Improved error message on unsupported algorithms in HashContent.
  • ConsumeWindowsEventLog Identifier regexes are now precompiled in onSchedule.
  • Added the component UUID to the end of processor and controller service log lines.
  • Processor metrics configuration now supports regex matching, e.g. processorMetrics/Get.*Metrics.
  • Added support for new AWS regions: ap-southeast-3, me-central-1, us-iso-east-1, us-isob-east-1, us-iso-west-1.
  • Added session commit time metrics.
  • Improved error messages when minifi is unable to start.
  • Added a check that prevents running when volatile and persistent repositories are combined.
  • Listing processors are now enforced to run on a single thread. Previously you could configure them to run on multiple threads, but this could cause invalid output, while not improving performance.
  • Added caching of SID lookups in ConsumeWindowsEventLog. Cache entries are invalidated after 24 hours, on restart and on C2 flow updates.
  • Flow file reloading from the repository on startup is now done before starting flow execution. This eliminates the need for the flow file checkpoint.
  • Added automatic cleanup of orphaned content repository entries on startup as a safety measure.
  • Added default connection size limits of 2000 flow files or 100 MB. This does not change anything for deployments using EFM, since it generates a configuration with these settings configured.
  • Removed C2 metrics from the default configuration that are not used by EFM.
  • ConsumeWindowsEventLog can now handle UserData tags in the events when using simple or flattened JSON output format.
  • Improved ListFile performance.

Breaking changes:

  • The format of GetFile output attributes are changed to match ListFile and NiFi GetFile.
  • Added failure relationships to SQL processors.

Fixed issues:

  • Fixed a crash after C2 property updates.
  • Fixed a bug where the old flow was not restarted after a failed C2 flow update (publish).
  • Fixed a crash when empty flow files were read from python scripted processors.
  • Fixed a 1 byte leak on each SID lookup in ConsumeWindowsEventLog.
  • Fixed scheduling to retry after the yield period even if the processor trigger fails, and it would otherwise not be scheduled again (for example: cron daily).
  • Fixed a bug in ConsumeWindowsEventLog that caused output flow file attributes containing resolved identifiers to be missing when using JSON output format.
  • Fixed a bug that caused the flow file repository to keep growing with invalid entries if the flow file checkpoint is not empty, after a C2 flow update is published.
  • Added a workaround for a rocksdb limitation where compaction cycles are not run for long periods in some use cases. Minifi now forces a compaction every 2 minutes by default.
  • Fixed an issue with FilesystemRepository content repository where failed content file deletions were not properly handled. Now deletion failures are logged and retried later.

November 11, 2022

CEM MiNiFi C++ Agent - 1.22.10 release includes:

  • Added support for the following processors:
    • PutTCP
  • Added TLS support for ListenSyslog and ListenTCP.
  • Added agent ID to exported metrics to improve observability when multiple agents are monitored.
  • Windows binaries now have version metadata.
  • AWS and Azure processors can now run on multiple threads.
  • Various improvements to FilesystemRepository (content repository implementation).
  • Improved throughput in certain high IO load scenarios, for example, when ConsumeWindowsEventLog and MergeContent are used together in a flow.
  • Added Batch Size property to TailFile to limit resource usage spikes in high volume log collection scenarios.
  • The following issues are fixed:
    • Python scripting no longer requires the patchelf hack.
    • Fixed memory leaks in AppendHostInfo and ConsumeWindowsEventLog.
    • Fixed various issues with multi-gigabyte flow file handling.
    • Fixed an issue in MergeContent where bins were often not flushed until Max Bin Age expired, even if they were full.

September 1, 2022

CEM MiNiFi C++ Agent - 1.22.08 release includes:
  • Added support for the following processors:
    • CollectKubernetesPodMetrics (Linux only)
    • ExecuteScript (Tech Preview)
    • ExecutePythonProcessor (Tech Preview)
  • Flow files waiting in connections now have their metadata swapped out to disk.
  • Multiple instances of the same processor type now have separate metrics.
  • Added Python scripting tech preview, disabled by default.
  • Improved InvokeHTTP performance by using HTTP Keep-Alive.
  • Added push alert capability on certain errors.
  • The following issues are fixed:
    • Fixed date handling issues (cron scheduler, EL functions) in the Docker image release.
    • Fixed ListenHTTP causing high CPU usage when used with timer driven scheduling, and a 0 ms scheduling period.
    • Fixed an issue where duplicate processor names in a flow caused EFM to get stuck in a loop.
    • The system certificate store is now correctly detected on Debian and Ubuntu when using SSLContextService.

August 25, 2022

CEM MiNiFi C++ Agent - 1.22.06-h1 release includes:
  • Added support for the following processors:
    • FetchGCSObject, DeleteGCSObject, and ListGCSBucket for Google Cloud
    • ListenTCP
    • PostElasticsearch
  • The breaking changes are as follows:
    • LogAttribute Maximum Payload Line Length is now 0 by default. The value was 80 earlier.
    • Changed ConsumeWindowsEventLog output attributes to lowercase.separated.by.dots, to be consistent with other processors.
      • Users can use UpdateAttribute to emulate the old behavior.
    • Fixed a bug in ExtractText where "Include Capture Group 0" was interpreted as the reverse of the boolean value. Switched to the correct interpretation, i.e. true became false and false became true.
    • Fixed a bug in InvokeHTTP which made it try to send all flow file attributes as headers when "Attributes to Send" was empty or not set (introduced in 1.21.10).
  • The other changes are as follows:
    • Added a new InvokeHTTP attribute "Invalid HTTP Header Field Handling Strategy", which can prevent errors that occur when trying to send attributes with keys that would not be valid HTTP header names.
    • Added support for pushing assets from EFM to MiNiFi C++ agents. Requires EFM 1.4.1+ REST API, no UI yet.
    • Improved SSL/TLS error logging, including logging a warning message 2 weeks before a certificate expires.
    • Added optional C2 request compression, to reduce the network load of heartbeats. Activate with nifi.c2.rest.request.encoding=gzip in minifi.properties. Requires EFM 1.4.1+.
    • Added the capability to expose agent metrics through a Prometheus-compatible endpoint.
  • The following issues are fixed:
    • An issue which made ListenSyslog always fail validation in EFM (introduced in 1.22.04).
    • Various issues with CRON_DRIVEN scheduling, making it ready for production use.
    • Fixed an issue where EFM got stuck in a loop when a flow with more than one processor with the same name was published (affects 1.22.04 and 1.22.06).

May 4, 2022

CEM MiNiFi C++ Agent - 1.22.04 release includes:
  • Added support for the following processors:
    • ListFile (both Linux and Windows)
    • FetchFile (both Linux and Windows)
    • ProcFsMonitor (Linux only)
    • PutGCSObject (both Linux and Windows)
  • Improved the stability of ListenSyslog, and added support for running it on Windows.
  • When using the docker image, MiNiFi logs are now available from docker logs directly.
  • Added support for updating properties through EFM (CEM 1.4+).
  • The following issues are fixed:
    • MINIFICPP-1788: MergeContent now flushes old bins on expiration even without additional inputs.
    • MINIFICPP-1766: Fixed the bug that caused rollbacks when working with empty flowfiles.
    • MINIFICPP-1802: Removed unnecessary AWS calls on startup.
    • MINIFICPP-1773: Fixed the crash that could occur when using Provenance Repository in properties.
    • MINIFICPP-1675: Fixed the possible crash that could occur with large regex inputs.
  • Other small improvements and bug fixes.

March 22, 2022

CEM MiNiFi C++ Agent - 1.22.03 release includes:
  • TailFile is now capable of collecting logs from Kubernetes
  • Support added for KubernetesControllerService
  • Support added for FetchAzureBlobStorage processor
  • DefragmentText is now capable of handling multiple inputs simultaneously

February 4, 2022

CEM MiNiFi C++ Agent - 1.22.01 release includes:
  • Added support for the following processors:

    • DeleteAzureBlobStorage

    • DeleteAzureDataLakeStorage

    • FetchAzureDataLakeStorage

    • ListAzureDataLakeStorage

    • PutSplunkHTTP

    • QuerySplunkIndexingStatus

    • PutUDP

  • Added SASL/PLAIN mechanism support to Kafka processors
  • Fixed MINIFICPP-1692 - TLSSocket hangs when no more data is available

November 12, 2021

CEM MiNiFi C++ Agent - 1.21.10 release includes:
  • Dynamic loading of extensions

    Extensions are loaded at startup. The MSI installer also provides options to select the ones you want to use.

  • Added ReplaceText processor
  • Added RouteText processor
  • Added DefragementText processor
  • Improved Azure credential handling
  • The release is available in docker image format as well

August 30, 2021

CEM MiNiFi C++ Agent - 1.21.08 release includes:
  • AppendHostInfo is now supported on Windows
  • Added AttributesToJson processor
  • Added Funnels
  • Common binary for all unix systems

July 23, 2021

CEM MiNiFi C++ Agent - 1.21.06 release includes:
  • Agent's configuration fingerprinting. You can enable a property in the agent's configuration so that a fingerprint of its configuration and flow definition is added to the heartbeat. It can be used to detect local changes made to a single agent compared to other agents belonging to the same class.
  • Repository encryption. You can enable encryption of the repositories used to persist the data being processed by the agents.
  • Support dropped for Ubuntu 16 and Debian 9.
  • Support added for Ubuntu 20 and Debian 10.

June 2, 2021

CEM MiNiFi C++ Agent - 1.21.04 release includes:
  • Added PerformanceDataMonitor processor
  • Added ConsumeJournald processor
  • Added resource consumption data to heartbeat
  • Added 32 bit version of Windows installer
  • Fixed CVE-2021-33191

April 9, 2021

CEM MiNiFi C++ Agent - 1.21.03 release includes:
  • Fix for PublishKafka processor to properly support expression language
  • Rewrite and performance improvements for the SQL processors

March 8, 2021

CEM MiNiFi C++ Agent - 1.21.02 release includes:
  • Fix for PublishKafka processor
  • Outputting JSON data on one line in ConsumeWindowsEventLog processor

February 15, 2021

CEM MiNiFi C++ Agent - 1.21.01 release includes:
  • Support for JSON output in the Consume Windows Even Log processor
  • Full Expression Language support on Windows
  • Full S3 support (List, Fetch, Get, Put)

December 16, 2020

CEM MiNiFi C++ Agent - 1.20.11 release includes:
  • Optional integration with the Windows Certificate store to retrieve certificates to be used for the agent to EFM communication over HTTPS

    For more information, see Integrating with the Windows certificate store.

  • Optional capability to encrypt the flow definition when persisted on disk on the agent's host

    For more information, see Encrypt sensitive data.

November 16, 2020

CEM MiNiFi C++ Agent - 1.20.10 release includes:
  • PutS3 processor to support sending data into AWS S3
  • Improvements on the ListenHTTP processor
  • Adding SNI information into raw TCP information during TLS handshake
  • Adding support for FlowFile v3 format in MergeContent processor to keep payload and flow file attributes when sending data using InvokeHTTP
  • Adding support for :format() expression language function on Windows hosts

October 14, 2020

CEM MiNiFi C++ Agent - 1.20.09 release includes:
  • The content repository now uses an implementation based on RocksDB
  • Fix on the security events caused by the agent when running on Windows
  • Improvement on the memory footprint of the agent
  • You can encrypt the sensitive properties of the MiNiFi C++ agent's configuration