MiNiFi C++ agent updates

MiNiFi C++ agents has periodic software updates made available independent of the EFM releases.

This section lists major features and updates for the independent CEM MiNiFi C++ Agent release.

November 11, 2022

CEM MiNiFi C++ Agent - 1.22.10 release includes:

  • Added support for the following processors:
    • PutTCP
  • Added TLS support for ListenSyslog and ListenTCP.
  • Added agent ID to exported metrics to improve observability when multiple agents are monitored.
  • Windows binaries now have version metadata.
  • AWS and Azure processors can now run on multiple threads.
  • Various improvements to FilesystemRepository (content repository implementation).
  • Improved throughput in certain high IO load scenarios, for example, when ConsumeWindowsEventLog and MergeContent are used together in a flow.
  • Added Batch Size property to TailFile to limit resource usage spikes in high volume log collection scenarios.
  • The following issues are fixed:
    • Python scripting no longer requires the patchelf hack.
    • Fixed memory leaks in AppendHostInfo and ConsumeWindowsEventLog.
    • Fixed various issues with multi-gigabyte flow file handling.
    • Fixed an issue in MergeContent where bins were often not flushed until Max Bin Age expired, even if they were full.

September 1, 2022

CEM MiNiFi C++ Agent - 1.22.08 release includes:
  • Added support for the following processors:
    • CollectKubernetesPodMetrics (Linux only)
    • ExecuteScript (Tech Preview)
    • ExecutePythonProcessor (Tech Preview)
  • Flow files waiting in connections now have their metadata swapped out to disk.
  • Multiple instances of the same processor type now have separate metrics.
  • Added Python scripting tech preview, disabled by default.
  • Improved InvokeHTTP performance by using HTTP Keep-Alive.
  • Added push alert capability on certain errors.
  • The following issues are fixed:
    • Fixed date handling issues (cron scheduler, EL functions) in the Docker image release.
    • Fixed ListenHTTP causing high CPU usage when used with timer driven scheduling, and a 0 ms scheduling period.
    • Fixed an issue where duplicate processor names in a flow caused EFM to get stuck in a loop.
    • The system certificate store is now correctly detected on Debian and Ubuntu when using SSLContextService.

August 25, 2022

CEM MiNiFi C++ Agent - 1.22.06-h1 release includes:
  • Added support for the following processors:
    • FetchGCSObject, DeleteGCSObject, and ListGCSBucket for Google Cloud
    • ListenTCP
    • PostElasticsearch
  • The breaking changes are as follows:
    • LogAttribute Maximum Payload Line Length is now 0 by default. The value was 80 earlier.
    • Changed ConsumeWindowsEventLog output attributes to lowercase.separated.by.dots, to be consistent with other processors.
      • Users can use UpdateAttribute to emulate the old behavior.
    • Fixed a bug in ExtractText where "Include Capture Group 0" was interpreted as the reverse of the boolean value. Switched to the correct interpretation, i.e. true became false and false became true.
    • Fixed a bug in InvokeHTTP which made it try to send all flow file attributes as headers when "Attributes to Send" was empty or not set (introduced in 1.21.10).
  • The other changes are as follows:
    • Added a new InvokeHTTP attribute "Invalid HTTP Header Field Handling Strategy", which can prevent errors that occur when trying to send attributes with keys that would not be valid HTTP header names.
    • Added support for pushing assets from EFM to MiNiFi C++ agents. Requires EFM 1.4.1+ REST API, no UI yet.
    • Improved SSL/TLS error logging, including logging a warning message 2 weeks before a certificate expires.
    • Added optional C2 request compression, to reduce the network load of heartbeats. Activate with nifi.c2.rest.request.encoding=gzip in minifi.properties. Requires EFM 1.4.1+.
    • Added the capability to expose agent metrics through a Prometheus-compatible endpoint.
  • The following issues are fixed:
    • An issue which made ListenSyslog always fail validation in EFM (introduced in 1.22.04).
    • Various issues with CRON_DRIVEN scheduling, making it ready for production use.
    • Fixed an issue where EFM got stuck in a loop when a flow with more than one processor with the same name was published (affects 1.22.04 and 1.22.06).

May 4, 2022

CEM MiNiFi C++ Agent - 1.22.04 release includes:
  • Added support for the following processors:
    • ListFile (both Linux and Windows)
    • FetchFile (both Linux and Windows)
    • ProcFsMonitor (Linux only)
    • PutGCSObject (both Linux and Windows)
  • Improved the stability of ListenSyslog, and added support for running it on Windows.
  • When using the docker image, MiNiFi logs are now available from docker logs directly.
  • Added support for updating properties through EFM (CEM 1.4+).
  • The following issues are fixed:
    • MINIFICPP-1788: MergeContent now flushes old bins on expiration even without additional inputs.
    • MINIFICPP-1766: Fixed the bug that caused rollbacks when working with empty flowfiles.
    • MINIFICPP-1802: Removed unnecessary AWS calls on startup.
    • MINIFICPP-1773: Fixed the crash that could occur when using Provenance Repository in properties.
    • MINIFICPP-1675: Fixed the possible crash that could occur with large regex inputs.
  • Other small improvements and bug fixes.

March 22, 2022

CEM MiNiFi C++ Agent - 1.22.03 release includes:
  • TailFile is now capable of collecting logs from Kubernetes
  • Support added for KubernetesControllerService
  • Support added for FetchAzureBlobStorage processor
  • DefragmentText is now capable of handling multiple inputs simultaneously

February 4, 2022

CEM MiNiFi C++ Agent - 1.22.01 release includes:
  • Added support for the following processors:
    • DeleteAzureBlobStorage

    • DeleteAzureDataLakeStorage

    • FetchAzureDataLakeStorage

    • ListAzureDataLakeStorage

    • PutSplunkHTTP

    • QuerySplunkIndexingStatus

    • PutUDP

  • Added SASL/PLAIN mechanism support to Kafka processors
  • Fixed MINIFICPP-1692 - TLSSocket hangs when no more data is available

November 12, 2021

CEM MiNiFi C++ Agent - 1.21.10 release includes:
  • Dynamic loading of extensions

    Extensions are loaded at startup. The MSI installer also provides options to select the ones you want to use.

  • Added ReplaceText processor
  • Added RouteText processor
  • Added DefragementText processor
  • Improved Azure credential handling
  • The release is available in docker image format as well

August 30, 2021

CEM MiNiFi C++ Agent - 1.21.08 release includes:
  • AppendHostInfo is now supported on Windows
  • Added AttributesToJson processor
  • Added Funnels
  • Common binary for all unix systems

July 23, 2021

CEM MiNiFi C++ Agent - 1.21.06 release includes:
  • Agent's configuration fingerprinting. You can enable a property in the agent's configuration so that a fingerprint of its configuration and flow definition is added to the heartbeat. It can be used to detect local changes made to a single agent compared to other agents belonging to the same class.
  • Repository encryption. You can enable encryption of the repositories used to persist the data being processed by the agents.
  • Support dropped for Ubuntu 16 and Debian 9.
  • Support added for Ubuntu 20 and Debian 10.

June 2, 2021

CEM MiNiFi C++ Agent - 1.21.04 release includes:
  • Added PerformanceDataMonitor processor
  • Added ConsumeJournald processor
  • Added resource consumption data to heartbeat
  • Added 32 bit version of Windows installer
  • Fixed CVE-2021-33191

April 9, 2021

CEM MiNiFi C++ Agent - 1.21.03 release includes:
  • Fix for PublishKafka processor to properly support expression language
  • Rewrite and performance improvements for the SQL processors

March 8, 2021

CEM MiNiFi C++ Agent - 1.21.02 release includes:
  • Fix for PublishKafka processor
  • Outputting JSON data on one line in ConsumeWindowsEventLog processor

February 15, 2021

CEM MiNiFi C++ Agent - 1.21.01 release includes:
  • Support for JSON output in the Consume Windows Even Log processor
  • Full Expression Language support on Windows
  • Full S3 support (List, Fetch, Get, Put)

December 16, 2020

CEM MiNiFi C++ Agent - 1.20.11 release includes:
  • Optional integration with the Windows Certificate store to retrieve certificates to be used for the agent to EFM communication over HTTPS

    For more information, see Integrating with the Windows certificate store.

  • Optional capability to encrypt the flow definition when persisted on disk on the agent's host

    For more information, see Encrypt sensitive data.

November 16, 2020

CEM MiNiFi C++ Agent - 1.20.10 release includes:
  • PutS3 processor to support sending data into AWS S3
  • Improvements on the ListenHTTP processor
  • Adding SNI information into raw TCP information during TLS handshake
  • Adding support for FlowFile v3 format in MergeContent processor to keep payload and flow file attributes when sending data using InvokeHTTP
  • Adding support for :format() expression language function on Windows hosts

October 14, 2020

CEM MiNiFi C++ Agent - 1.20.09 release includes:
  • The content repository now uses an implementation based on RocksDB
  • Fix on the security events caused by the agent when running on Windows
  • Improvement on the memory footprint of the agent
  • You can encrypt the sensitive properties of the MiNiFi C++ agent's configuration