MiNiFi C++ Agent updates and fixed issues

July 30, 2024

CEM MiNiFi C++ Agent - 1.24.05 release includes:

New features:

• Packages for ARM64 Linux are now available in addition to x86_64
• Resource synchronization through C2 has been enabled
• Introduced agent-side sensitive and non-sensitive parameter support (not yet supported by EFM)
• Added an option for non-sync writes in RocksDB content repository
• Added SSLContextService support for NiFi Python processors

Bug fixes:

• Fixed issues with auto-terminated relationships in Python scripting
• Fixed issues with the AttributesToLog and AttributesToIgnore properties in LogAttribute
• The minifi.sh script now exits early on encountering an error
• ExecuteScript with Lua now works without requiring an onTrigger function
• Python processors now prioritize virtualenv packages over system packages

Other changes:

• Improved the throughput of Event Driven processors
• GetFile's Directory property now evaluates expression language
• Upgraded the Docker base image to Alpine Linux 3.19.1
• OpenSSL and cURL extensions have been moved to minifi-core
• Improved error handling in Python processors

Breaking changes:

• Moved from Quartz to Spring Cron Syntax, in line with NiFi
  • Configuration change is needed if the day of the week was set numerically: previously 1(SUN)-7(SAT) -> now 0(SUN)-7(SUN)
  • Cloudera recommends using Mon-Sun word abbreviations instead of numbers to specify the day of the week
• Moved ExecuteProcess from standard-processors to a separate extension, which is now included in the Linux extra-extensions package

April 30, 2024

CEM MiNiFi C++ Agent - 1.24.04 release includes:

New features and changes:

Fixed issues:

Breaking changes:

April 4, 2024

The CEM MiNiFi C++ Agent - 1.24.01-h1 release fixes a bug that could cause excessive disk I/O with MergeContent.

The CEM MiNiFi C++ Agent - 1.23.09-h1 release fixes a bug that could cause excessive disk I/O with MergeContent.

February 5, 2024

CEM MiNiFi C++ Agent - 1.24.01 release includes:

Added support for the following processors:

• AttributeRollingWindow
• SplitText
• PushGrafanaLokiREST
• PushGrafanaLokiGrpc
• JoltTransformJSON

New features and changes:

• Using "administrative yield duration" for onSchedule retry interval for consistency with NiFi ( the previous dedicated setting was "onSchedule retry interval").
• Moved to using Visual Studio 2022 to compile on Windows.
• Introduced network speed limit options to InvokeHTTP.
• Added debug command to MiNiFi Controller, similar to the C2 debug command.
• Updated AWS SDK and expanded support to new regions: ap-south-2, eu-central-2, and eu-south-2.
• Included markdown docs in the released binary packages.
• Sensitive processor and controller service properties are now stored encrypted in config.yml.
• GenerateFlowFile now reevaluates EL expressions in 'Custom Text' for each batch.

Fixed issues:

• Resolved issue with DefragmentText issue.
• Fixed memory leak of logger objects under various circumstances.
• Fixed issue with site-to-site issue, where a reply of TRANSACTION_FINISHED_BUT_DESTINATION_FULL, occurring when the destination connection had reached its backpressure threshold, was incorrectly treated as an error. This led to unnecessary data retransmission (retry) despite the successful transfer.

September 29, 2023

CEM MiNiFi C++ Agent - 1.23.09 release includes:

Added support for the following processors:

• PutSmb
• FetchSmb
• ListSmb

New features and changes:

• TLS 1.3 support
• TLS support for Prometheus Reporter
• Multipart upload support for PutS3Object
• Switched to systemd service management on Linux (from init script)
• GetTCP processor reworked
• MiNiFi Controller added to the docker image
• Maximum log entry length is now configurable using the 'max.log.entry.length' log property
• Properties can now be overridden by minifi command line arguments
• New System Load Avg field in C2 (EFM 2.0+) and Prometheus metrics
• Added missing System/Security/UserId to ConsumeWindowsEventLog JSON output
• Added RocksDB stats to Repository metrics
• Support for metrics in the application logs

Breaking changes:

• GetTCP has been reworked, many of its properties are changed

Fixed issues:

• Fixed MergeContent error handling issues
• Fixed failing PutS3Object on Windows
• Fixed C2 debug command (previously could return outdated logs)
• Fixed the system certificate store in SSLContextService (linux)
• Timer driven processors will now properly wait for next scheduled time after yielding
• MINIFI_HOME is now protected from mutual access

September 4, 2023

June 23, 2023

CEM MiNiFi C++ Agent - 1.23.06 release includes:

New features:

• ConsumeWindowsEventLog is now capable of reading Windows events from log files.
• Repository size metrics are now reported for every type of repository.

Fixes and improvements:

• Fixed premature wake-up of event-driven scheduled processors.
• Fixed MiNiFi controller crash during flow update.
• Cron-driven scheduling is no longer dependent on en_US.UTF-8 locale.
• Improved error messages related to RouteOnAttribute and ExpressionLanguage.
• Improved error messages related to configuration parsing.

April 28, 2023

CEM MiNiFi C++ Agent - 1.23.04 release includes:

Added support for the following processors:

• FetchOPCProcessor
• PutOPCProcessor

Changes:

• Start shipping prometheus extension for metrics export
• EL toDate can now parse RFC3339 dates
• Mapped StateManager to python and lua scripting to make stateful scripted processors possible
• Added 'Use Path Style Access' property to PutS3Object
• ConsumeWindowsEventLog: When mapping to flattened JSON, EventData subobjects are now prefixed with their path, and are no longer duplicated in the output

Breaking changes:

• ConsumeWindowsEventLog now fails on startup when the Output Format or JSON Format properties are invalid, instead of silently doing nothing.

Fixed issues:

• Fixed machineArch casing in the C2 protocol system info

April 4, 2023

CEM MiNiFi C++ Agent - 1.23.02-h1 restored compatibility with EFM 1.4.x (broken in 1.23.02).

Fixed issues:

Fixed an issue where using the time period format for specifying the heartbeat interval in minifi.properties resulted in EFM exceptions, and diminised agent management capabilities from EFM.

March 9, 2023

CEM MiNiFi C++ Agent - 1.23.02 release includes:

Added support for the following processors:

• ListenUDP
• PublishMQTT
• ConsumeMQTT

Changes:

• Python scripting can now work with multiple python minor versions.
• Added support for multiple process groups.
• Decreased the default C2 heartbeat frequency to once every 30 seconds.
• Optimized memory usage in some high flow file count scenarios.
• Added ProcessSession::remove to the Python scripting API.
• Increased the default flow controller thread count (nifi.flow.engine.threads) to 5.
• Improved error message on unsupported algorithms in HashContent.
• ConsumeWindowsEventLog Identifier regexes are now precompiled in onSchedule.
• Added the component UUID to the end of processor and controller service log lines.
• Processor metrics configuration now supports regex matching, e.g. processorMetrics/Get.*Metrics.
• Added support for new AWS regions: ap-southeast-3, me-central-1, us-iso-east-1, us-isob-east-1, us-iso-west-1.
• Added session commit time metrics.
• Improved error messages when minifi is unable to start.
• Added a check that prevents running when volatile and persistent repositories are combined.
• Listing processors are now enforced to run on a single thread. Previously you could configure them to run on multiple threads, but this could cause invalid output, while not improving performance.
• Added caching of SID lookups in ConsumeWindowsEventLog. Cache entries are invalidated after 24 hours, on restart and on C2 flow updates.
• Flow file reloading from the repository on startup is now done before starting flow execution. This eliminates the need for the flow file checkpoint.
• Added automatic cleanup of orphaned content repository entries on startup as a safety measure.
• Added default connection size limits of 2000 flow files or 100 MB. This does not change anything for deployments using EFM, since it generates a configuration with these settings configured.
• Removed C2 metrics from the default configuration that are not used by EFM.
• ConsumeWindowsEventLog can now handle UserData tags in the events when using simple or flattened JSON output format.
• Improved ListFile performance.

Breaking changes:

• The format of GetFile output attributes are changed to match ListFile and NiFi GetFile.
• Added failure relationships to SQL processors.

Fixed issues:

• Fixed a crash after C2 property updates.
• Fixed a bug where the old flow was not restarted after a failed C2 flow update (publish).
• Fixed a crash when empty flow files were read from python scripted processors.
• Fixed a one-byte leak on each SID lookup in ConsumeWindowsEventLog.
• Fixed scheduling to retry after the yield period even if the processor trigger fails, and it would otherwise not be scheduled again (for example: cron daily).
• Fixed a bug in ConsumeWindowsEventLog that caused output flow file attributes containing resolved identifiers to be missing when using JSON output format.
• Fixed a bug that caused the flow file repository to keep growing with invalid entries if the flow file checkpoint is not empty, after a C2 flow update is published.
• Added a workaround for a rocksdb limitation where compaction cycles are not run for long periods in some use cases. Minifi now forces a compaction every two minutes by default.
• Fixed an issue with FilesystemRepository content repository where failed content file deletions were not properly handled. Now deletion failures are logged and retried later.

November 11, 2022

CEM MiNiFi C++ Agent - 1.22.10 release includes:

Added support for the following processors:

• PutTCP

Changes:

• Added TLS support for ListenSyslog and ListenTCP.
• Added agent ID to exported metrics to improve observability when multiple agents are monitored.
• Windows binaries now have version metadata.
• AWS and Azure processors can now run on multiple threads.
• Various improvements to FilesystemRepository (content repository implementation).
• Improved throughput in certain high IO load scenarios, for example, when ConsumeWindowsEventLog and MergeContent are used together in a flow.
• Added 'Batch Size' property to TailFile to limit resource usage spikes in high volume log collection scenarios.

Fixed issues:

• Python scripting no longer requires the patchelf hack.
• Fixed memory leaks in AppendHostInfo and ConsumeWindowsEventLog.
• Fixed various issues with multi-gigabyte flow file handling.
• Fixed an issue in MergeContent where bins were often not flushed until Max Bin Age expired, even if they were full.

September 1, 2022

CEM MiNiFi C++ Agent - 1.22.08 release includes:

Added support for the following processors:

• CollectKubernetesPodMetrics (Linux only)
• ExecuteScript (Tech Preview)
• ExecutePythonProcessor (Tech Preview)

Changes:

• Flow files waiting in connections now have their metadata swapped out to disk.
• Multiple instances of the same processor type now have separate metrics.
• Added Python scripting tech preview, disabled by default.
• Improved InvokeHTTP performance by using HTTP Keep-Alive.
• Added push alert capability on certain errors.
• The following issues are fixed:

Fixed issues:

• Fixed date handling issues (cron scheduler, EL functions) in the Docker image release.
• Fixed ListenHTTP causing high CPU usage when used with timer driven scheduling, and a 0 ms scheduling period.
• Fixed an issue where duplicate processor names in a flow caused EFM to get stuck in a loop.
• The system certificate store is now correctly detected on Debian and Ubuntu when using SSLContextService.

August 25, 2022

CEM MiNiFi C++ Agent - 1.22.06-h1 release includes:

Added support for the following processors:

• FetchGCSObject, DeleteGCSObject, and ListGCSBucket for Google Cloud
• ListenTCP
• PostElasticsearch

Changes:

• Added a new InvokeHTTP attribute "Invalid HTTP Header Field Handling Strategy", which can prevent errors that occur when trying to send attributes with keys that would not be valid HTTP header names.
• Added support for pushing assets from EFM to MiNiFi C++ Agents. Requires EFM 1.4.1+ REST API, no UI yet.
• Improved SSL/TLS error logging, including logging a warning message 2 weeks before a certificate expires.
• Added optional C2 request compression, to reduce the network load of heartbeats. Activate with nifi.c2.rest.request.encoding=gzip in minifi.properties. Requires EFM 1.4.1+.
• Added the capability to expose agent metrics through a Prometheus-compatible endpoint.

Breaking changes:

• LogAttribute Maximum Payload Line Length is now 0 by default. The value was 80 earlier.
• Changed ConsumeWindowsEventLog output attributes to lowercase.separated.by.dots, to be consistent with other processors.
  • Users can use UpdateAttribute to emulate the old behavior.
• Fixed a bug in ExtractText where "Include Capture Group 0" was interpreted as the reverse of the boolean value. Switched to the correct interpretation, for example true became false and false became true.
• Fixed a bug in InvokeHTTP which made it try to send all flow file attributes as headers when "Attributes to Send" was empty or not set (introduced in 1.21.10).

Fixed issues:

• An issue which made ListenSyslog always fail validation in EFM (introduced in 1.22.04).
• Various issues with CRON_DRIVEN scheduling, making it ready for production use.
• Fixed an issue where EFM got stuck in a loop when a flow with more than one processor with the same name was published (affects 1.22.04 and 1.22.06).

May 4, 2022

CEM MiNiFi C++ Agent - 1.22.04 release includes:

Added support for the following processors:

• ListFile (both Linux and Windows)
• FetchFile (both Linux and Windows)
• ProcFsMonitor (Linux only)
• PutGCSObject (both Linux and Windows)

Changes:

• Improved the stability of ListenSyslog, and added support for running it on Windows.
• When using the docker image, MiNiFi logs are now available from docker logs directly.
• Added support for updating properties through EFM (CEM 1.4+).

Fixed issues:

• MINIFICPP-1788: MergeContent now flushes old bins on expiration even without additional inputs.
• MINIFICPP-1766: Fixed the bug that caused rollbacks when working with empty flowfiles.
• MINIFICPP-1802: Removed unnecessary AWS calls on startup.
• MINIFICPP-1773: Fixed the crash that could occur when using Provenance Repository in properties.
• MINIFICPP-1675: Fixed the possible crash that could occur with large regex inputs.
• Other small improvements and bug fixes.

March 22, 2022

CEM MiNiFi C++ Agent - 1.22.03 release includes:

February 4, 2022

CEM MiNiFi C++ Agent - 1.22.01 release includes:

Added support for the following processors:

• DeleteAzureBlobStorage

• DeleteAzureDataLakeStorage

• FetchAzureDataLakeStorage

• ListAzureDataLakeStorage

• PutSplunkHTTP

• QuerySplunkIndexingStatus

• PutUDP

Changes:

• Added SASL/PLAIN mechanism support to Kafka processors
• Fixed MINIFICPP-1692 - TLSSocket hangs when no more data is available

November 12, 2021

CEM MiNiFi C++ Agent - 1.21.10 release includes:

• Dynamic loading of extensions: Extensions are loaded at startup. The MSI installer also provides options to select the ones you want to use.
• Added ReplaceText processor
• Added RouteText processor
• Added DefragementText processor
• Improved Azure credential handling
• The release is available in docker image format as well.

August 30, 2021

CEM MiNiFi C++ Agent - 1.21.08 release includes:

• AppendHostInfo is now supported on Windows
• Added AttributesToJson processor
• Added Funnels
• Common binary for all unix systems

July 23, 2021

CEM MiNiFi C++ Agent - 1.21.06 release includes:

• Agent's configuration fingerprinting. You can enable a property in the agent's configuration so that a fingerprint of its configuration and flow definition is added to the heartbeat. It can be used to detect local changes made to a single agent compared to other agents belonging to the same class.
• Repository encryption. You can enable encryption of the repositories used to persist the data being processed by the agents.
• Support dropped for Ubuntu 16 and Debian 9.
• Support added for Ubuntu 20 and Debian 10.

June 2, 2021

CEM MiNiFi C++ Agent - 1.21.04 release includes:

• Added PerformanceDataMonitor processor
• Added ConsumeJournald processor
• Added resource consumption data to heartbeat
• Added 32 bit version of Windows installer
• Fixed CVE-2021-33191

April 9, 2021

CEM MiNiFi C++ Agent - 1.21.03 release includes:

• Fix for PublishKafka processor to properly support expression language
• Rewrite and performance improvements for the SQL processors

March 8, 2021

CEM MiNiFi C++ Agent - 1.21.02 release includes:

• Fix for PublishKafka processor
• Outputting JSON data on one line in ConsumeWindowsEventLog processor

February 15, 2021

CEM MiNiFi C++ Agent - 1.21.01 release includes:

• Support for JSON output in the Consume Windows Even Log processor
• Full Expression Language support on Windows
• Full S3 support (List, Fetch, Get, Put)

December 16, 2020

CEM MiNiFi C++ Agent - 1.20.11 release includes:

• Optional integration with the Windows Certificate store to retrieve certificates to be used for the agent to EFM communication over HTTPS

  For more information, see Integrating with the Windows certificate store.

• Optional capability to encrypt the flow definition when persisted on disk on the agent's host

  For more information, see Encrypting sensitive data.

November 16, 2020

CEM MiNiFi C++ Agent - 1.20.10 release includes:

• PutS3 processor to support sending data into AWS S3
• Improvements on the ListenHTTP processor
• Adding SNI information into raw TCP information during TLS handshake
• Adding support for FlowFile v3 format in MergeContent processor to keep payload and flow file attributes when sending data using InvokeHTTP
• Adding support for :format() expression language function on Windows hosts

October 14, 2020

CEM MiNiFi C++ Agent - 1.20.09 release includes:

• The content repository now uses an implementation based on RocksDB
• Fix on the security events caused by the agent when running on Windows
• Improvement on the memory footprint of the agent
• You can encrypt the sensitive properties of the MiNiFi C++ Agent's configuration