Fixed Issues in Flow Management

Review the list of Flow Management issues that are resolved in Cloudera DataFlow for Data Hub 7.2.15.

7.2.15.2 Technical Service Bulletins

TSB 2022-580: NiFi Processors cannot write to content repository: For the latest update on this issue see the corresponding Knowledge article: TSB 2022-580: NiFi Processors cannot write to content repository

TSB 2022-589: CVE-2022-33140 Apache NiFi ShellUserGroupProvider Vulnerability: For the latest update on this issue see the corresponding Knowledge article: TSB 2022-589: CVE-2022-33140 Apache NiFi ShellUserGroupProvider Vulnerability

7.2.15

NIFI-9943: Added Transform Provider to nifi-xml-processing.
NIFI-9901: Removed provided scope from nifi-xml-processing registry-ranger.

Added nifi-xml-processing to nifi-commons.
NIFI-9835: Fixed threading bug in which NioAsyncLoadBalanceClient calls LoadBalanceSession.isComplete() followed by LoadBalanceSession.isCanceled() but it's possible for the complete flag to change before the canceled flag (they are not updated atomically). So changed to use a single LoadBalanceSessionState enum that represents the state. Also made the private StandardProcessSession.commit(boolean) method synchronized. When a processor is terminated (as is the case in Offload), we roll back sessions and both the commit() and rollback() need to be synchronized. Only the public commit() method was synchronized, and now with commitAsync() happening, we had the ability to commit without any synchronization. This addresses that concern. Also fixed a typo in docs for MergeRecord.
NIFI-9834: When calling ByteArrayContentRepository.read() on a null Content Claim, return an empty ByteArrayInputStream instead of throwing NullPointerException.
NIFI-9827: Upgrading AWS Java SDK to 1.12.182 to pick up new AWS Regions.
NIFI-9818: Fix flaky tests.
NIFI-9815: Corrected log message formatting in multiple classes.
NIFI-9807: Added Refresh Window Property to OAuth2 Token Provider.
NIFI-9806: Introduce ConfigurableExtensionDefinition and VersionedConfigurableExtension (#5875).
NIFI-9801: Fixed error in previous correction of AccessToken.isExpired() margin calculation NIFI-9801 Stabilized shaky AccessTokenTest.
NIFI-9800: Unwrap SQLException in PutDatabaseRecord when table does not exist.
NIFI-9799: Enabled style checking for nifi-system-tests in ci-workflow.
NIFI-9797: Corrected AccessToken.isExpired() margin calculation.
NIFI-9796: Updated Registry Security Configuration to avoid warnings.
NIFI-9795: Checkstyle, rat issues in nifi-system-test-suite module.
NIFI-9794: If a node is OFFLOADING, do not allow connections to be deleted. Also if we fail to notify the node that it needs to offload its data, change its state back to DISCONNECTED.
NIFI-9791: Use maven.build.timestamp during manifest generation instead of buildhelper timestamp.
NIFI-9790: Fixed race condition in which SwappablePriorityQueue could attempt to access the 0th element of an empty collection.
NIFI-9789: Upgraded Logback from 1.2.10 to 1.2.11.
NIFI-9788: Updated commons-codec to 1.15 across all modules.
NIFI-9786: Added debug to KeyStoreUtils.isStoreValid.
NIFI-9785: Improved Login Credentials Writer File Handling.
NIFI-9783: When migrating FlowFiles from one ProcessSession to another, if any FlowFile had already been transferred, and the Relationship to which it was transferred was auto-terminated, we were updating the wrong member variable, which threw off our stats for the processor. Fixed that.
NIFI-9782: Excluded H2 DB from nifi-druid-bundle.
NIFI-9781: Fix handling when selecting array element via QueryRecord.
NIFI-9778: Fixing additional details for ScriptedPartitionRecord.
NIFI-9777: Adding support to remove attributes from verification requests.
NIFI-9775: Create RuntimeManifestService.
NIFI-9774: Upgraded Netty from 4.1.73 to 4.1.74.
NIFI-9771: When a Kafka record is obtained during config verification, we should produce an invalid response if the Record Reader is not able to produce any records from it.
NIFI-9766: Avoid intermittent SearchElasticsearchTest failures in CI pipeline.
NIFI-9765: Added documentation that covers how to build a custom binar…
NIFI-9764: Atlas reporting task sends 'unknown' hive_table when table is name not available.
NIFI-9763: Additional escaped VALUE column due to H2 changes.

Escaped VALUE column for Configure Details Auditing.
NIFI-9762: Adding DBCPConnectionPool config verification.
NIFI-9761: Correct PeerChannel processing for TLS 1.3.
NIFI-9759: Upgraded Spring Framework from 5.3.15 to 5.3.16.
NIFI-9757: Upgraded SLF4J from 1.7.35 to 1.7.36.
NIFI-9756: Add documentation for framework-level retry in Processors and update processor tab images.
NIFI-9754: Introduced VersionedExternalFlow - Updated stateless and StandardProcessGroup, etc. to make use of VersionedExternalFlow - Updated StatelessDataflowDefinition to use ExternalVersionedFlow instead of generic type - Updated Stateless Bootstrap to avoid loading stateless engine libs from root classpath but instead use a NarClassLoader to load the stateless nar.
NIFI-9751: Poll as needed during system-tests to ensure expected state.
NIFI-9750: Logging Improvements to support LoadBalanceProtocol troubleshooting.
NIFI-9749: Capture additional logging for system-test workflow runs.
NIFI-9748: Added new property for Output Format to LogAttribute. Also made the FlowFile Properties (file size, entry date, lineage start date) optional and renamed from 'Standard FlowFile Attributes' to 'FlowFile Properties' because this has led to confusion many times in the past, around users wanting to reference these things as attributes via EL but they are not actually attributes.
NIFI-9747: Track PID in nifi-bootstrap logging on shutdown.
NIFI-9745: Prevent insertion of revisions in NiFi registry when revision feature is disabled.
NIFI-9743: Upgraded Jetty from 9.4.44 to 9.4.45.
NIFI-9741: Make the close() method of WriteAvroResultWithExternalSchema idempotent.
NIFI-9738: VersionedComponent data members should derive from Object.
NIFI-9736: Improved TestRouteText to avoid intermittent failures.
NIFI-9735: Corrected Jetty Duplicate Mapping Warning.
NIFI-9734: Standardized exception cause message formatting.
NIFI-9733: Fixing StandardConnection.verifySourceStoppedOrFunnel infinite recursion.
NIFI-9732: Upgraded Zip4j from 2.8.0 to 2.9.1.
NIFI-9731: Updated to use a shorter, simpler output format for FlowFiles when creating bulletins.
NIFI-9730: Consider a change in value for retry-related fields from 'null' to the default value as an environmental change so that it's not flagged as a Local Modification, which would prevent users from updating the version of the Process Group that they are using.
NIFI-9729: When restarting components in the VersionedFlowSynchronizer, first filter out any components that are intended to be stopped.
NIFI-9728: Added support for User Assigned Managed Identity authentication for Azure ADLS and Blob_v12 processors.
NIFI-9727: IndexOutOfBoundsException in CorrelationAttributePartitioner.
NIFI-9726: Removed duplicate nifi-utils dependencies from graph modules.
NIFI-9725: On shutdown, instead of spawning a background thread to shutdown Cluster Coordinator, do so in the calling thread. This avoids a race condition whereby the cluster coordinator cannot be determined because the other thread has shutdown the FlowController.
NIFI-9724: Added set-sensitive-properties-algorithm command.
NIFI-9723: When we add controller-level Controller Services on restart of NiFi, ensure that all Controller Services are updated to include their property values, etc. Also ensure that for these services and reporting tasks we decrypt the property values.
NIFI-9722: Do not throw an Exception from verifyCanUpdateProperties when property descriptor & parameter descriptor's sensitivities don't match - instead allow the set to happen and let processor become invalid. Also, allow values such as abc#{param} for ghost processors.
NIFI-9721: Support enum types in AvroTypeUtil.buildAvroSchema().
NIFI-9716 and NIFI-9577: Addressed issue in the PathFilter for GetFile / ListFile. For any file that is found in the Input Directory directly, it was previously being listed/fetched even if it didn't match the PathFilter. Additionally, updated the code to create a new File FIlter for every invocation of onTrigger. This was necessary for NIFI-9577 because the directory to monitor supports Expression Language and as a result may change from invocation to invocation, if using a function such as now() but the PathFilter would always relativize the path based on the value that was obtained when the processor was scheduled.
NIFI-9715: Add option to output empty FlowFile from Elasticsearch REST API Json Query processors when there are no hits from query.
NIFI-9714: Added overloaded toMap to MapRecord that can convert sub-records into maps.
NIFI-9713: TagS3Object do not have provenance data.
NIFI-9711: Added support for flow.json.gz in SetSensitivePropertiesKey.
NIFI-9707: Resolved duplicate JLine dependency in nifi-toolkit.
NIFI-9704: Updated the ContentRepositoryScanTask to show details of how much content in the content repo is retained by each queue in the dataflow. Changed default for nifi.content.claim.max.appendable.size property from 1 MB to 50 KB. Updated docs to reflect the new default value and explain what the property does and how it's used.
NIFI-9699: Updated oidcCallback method to handle error cases. Added some unit tests.
NIFI-9698: When creating an Avro schema, ensure that any default value is converted from what is returned by RecordField.getDefaultValue() to what Avro requires.
NIFI-9696: DeleteS3Object don't have provenance.
NIFI-9692: Upgraded Apache Commons Lang3 to 3.12.0.
NIFI-9691: Added ForkEnrichment, JoinEnrichment processors.
NIFI-9689: When checking FlowFile Availability, consider swap queue and trigger data to be swapped in, since calling poll() will no longer happen if no data is available.

When all FlowFiles in a FlowFile Queue are penalized, do not schedule the destination to run. Also expose this fact via the ConnectionStatusSnapshotDTO, as this allows the front-end to render this information to the user in order to avoid confusion when it appears that the Processor has data but does nothing.
NIFI-9688: Improve Logback shutdown handling.
NIFI-9687: Add additional documentation for nifi.cluster.node.protocol.max.threads property to Admin Guide.
NIFI-9686: Renamed SNMP integration tests correctly.
NIFI-9685: Upgraded JNA to 5.10.0.
NIFI-9684: Fix: When starting/stopping a selected process group, it sends the parent process group id to the REST interface that is responsible to enable/disable transmission for all remote process groups within a process group. Need to send the id of the select process group instead.
NIFI-9681: Upgraded Apache Commons DBCP to 2.9.0.
NIFI-9679: Added access-environment-credentials permission.
NIFI-9678: Update Elasticsearch REST API processor integration-tests for Elasticsearch 8.x.
NIFI-9673: Improved DBCP and HikariCP test reliability.
NIFI-9672: Fix flaky tests caused by the use of HashMap.
NIFI-9669: Adding PutDynamoDBRecord processor.
NIFI-9668: Adding informative error when setting same value in 'set-param' CLI command.
NIFI-9663: Setting the "csv-escape" property has no effect in SelectHive3QL.
NIFI-9662: Remove unused mail-1.4.7 dependency from nifi-framework-bundle.
NIFI-9660: Upgraded Apache Tika to 2.3.0.
NIFI-9657: Create MoveADLS processor.
NIFI-9655: Add Queue Logging to ListenUDP.
NIFI-9652: Upgraded jetty-schemas from 3.1 to 5.2.
NIFI-9650: Upgraded OkHttp from 4.9.2 to 4.9.3.
NIFI-9649: Upgraded SLF4J from 1.7.32 to 1.7.35.
NIFI-9647: Added ExtractDocumentText Processor.
NIFI-9645: Updated PutSplunk to allow idle connection timeouts.
NIFI-9644: Improved TestExecuteStateless increasing TestRunner.run() wait allowed.
NIFI-9642: Update Admin Guide and User Guide with correct nifi.properties default values.
NIFI-9641: Adjusted the extraction of the chroot suffix for solr client connections.
NIFI-9639: Determine how long it takes to find cluster coordinator and perform DNS lookup when sending heartbeats and include in the logs.
NIFI-9638: Refactored Google Guava usage in extensions.

Refactored Google Guava references.
NIFI-9635: Upgraded Netty from 4.1.72 to 4.1.73
NIFI-9634: Upgraded Spring Framework to 5.3.15.
NIFI-9632: Removed nifi-lumberjack-bundle.
NIFI-9631: Enable cli.sh to be used with a symbolic link.
NIFI-9630: Migrated Registry REST API docs to swagger-codegen.
NIFI-9629: Ensure that when we are setting default values on Avro GenericRecord objects that we convert from the schema's default value to the proper type.
NIFI-9628: Added a uiOnly flag when requesting Controller Service.
NIFI-9626: Allowing Stateless NiFi to parse flow snapshots with unrecognized fields.
NIFI-9625: Refactored Distributed Cache Server and Client Tests - Replaced TestServerAndClient with separate classes for Set Server and Map Server - Implemented before and after annotations for starting and stopping server instances.

Added check for cache directory existence before clean NIFI-9625 Updated Map and Set Cache Server Tests to use random port.
NIFI-9624: Removed JCenter Repository.
NIFI-9621: Added Ignore Reserved Characters to FlattenJson.
NIFI-9620: Adding isStateful to StatelessDataflow.
NIFI-9619: Removed GPG key from Security Mailing List reporting.
NIFI-9618: Upgraded Checkstyle to 9.2.1.
NIFI-9617: Removed unused screenshots from documentation.
NIFI-9616: Included SLF4J bridge libraries in NiFi Stateless Kafka Connect assembly.
NIFI-9611: Restore commons-io to minifi-assembly.

Removed unnecessary references to nifi-processor-utils.

Removed duplicate nifi-utils dependency.
NIFI-9610: Refactored nifi-processor-utils to separate modules.
NIFI-9609: Added nifi-snowflake-bundle with SnowflakeComputingConnectionPool.
NIFI-9608: Disabled system-tests workflow for pull requests.
NIFI-9607: Honor Update Keys when Quoting Identifiers in PutDatabaseRecord.
NIFI-9606: Removed nifi-security-utils from nifi-framework-api.
NIFI-9601: Upgraded nifi-bootstrap to Jakarta Mail 2.
NIFI-9600: Removed Elasticsearch 2 Processors.
NIFI-9599: Updating explicit plugin/build version references.
NIFI-9597: Fixing all explicit version refs to main latest.

Fix Dockerfile URLs.
NIFI-9596: Fix newline bug in JythonScriptRunner NIFI-9596: Added comment to indicate why the Apache header is missing.
NIFI-9595: Removed nifi-kafka-0.x modules.
NIFI-9594: When converting Record to Avro GenericRecord, ensure that any default values that are defined in the GenericRecord's schema get applied, regardless of whether or not the field exists in the associated RecordSchema.
NIFI-9593: Missing catch clauses in Confluent Schema Registry client.
NIFI-9591: Removed nifi-kite-bundle.
NIFI-9590: Added support for sensitive properties in Azure authorizers to encrypt-config.
NIFI-9589: Support initial loading from the current max values in QueryDatabaseTable* processors.
NIFI-9588: Update doc for 'nifi.content.repository.archive.max.retention.period'.
NIFI-9587: Added JSON format for Prometheus Flow Metrics.
NIFI-9586: Removed Surefire ForkNodeFactory configuration.

Excluded additional assembly modules from ci-workflow.
NIFI-9585: Upgraded H2 from 1.4 to 2.1.210.
NIFI-9581: Add PutElasticsearchRecord relationship for output of successful Records sent to Elasticsearch.
NIFI-9580: UI work for framework-level retry in Processors.
NIFI-9576: Introduced a BlockListClassLoader that can be used by stateless in order to isolate both the Stateless Engine and the NiFi extensions from extraneous classes that exist in the System ClassLoader.

Allowed Stateless' BlockListClassLoader to load java11 jars/classes.
NIFI-9571: Corrected Session commit handling in PutTCP.
NIFI-9570: Separate classpath for NiFi Registry sensitive property providers.
NIFI-9569: SNMP manager UDP transportmapping changed to 0.0.0.0.
NIFI-9568: Updated nifi-jolt-transform-json-ui pom.xml to only include CSS and JS assets in WAR.
NIFI-9564: Removed unnecessary logback-classic test dependencies.
NIFI-9563: Enabled ListenTCP Pool Receive Buffers property.
NIFI-9552: Fix NoClassDefFound error in case of nifi-registry-ranger-assembly.
NIFI-9548: When disabling RPG transmission, wait for the ports to complete in a background thread instead of blocking the web thread. Also moved the RPG initialization logic into flow controller instead of flow service and added a delay in order to reduce likelihood of ConnectException happening when pointing to nodes in the same cluster.
NIFI-9545: Fix in-place replacement for LookupRecord processor.
NIFI-9544: LookupRecord - fixed behavior when no matching value in the LRS.
NIFI-9543: Add bring-to-front functionality to labels.
NIFI-9527: Upgraded snappy-java to 1.1.8.4.
NIFI-9525: Modify lib packaging to use files from build directory.
NIFI-9501: Added REST end-point to retrieve a RuntimeManifest.
NIFI-9481: Excluded Data Transfer REST methods from DoSFilter.
NIFI-9475: Provide Framework-Level Retries for NiFi Relationships.
NIFI-9455: Added aggregated predictions to Prometheus Flow Metrics.
NIFI-9453: Refactored ListenBeats using Netty.
NIFI-9438: Refactored sensitive-property-provider to multiple modules.
NIFI-9435: Added registries and names include parameters to Flow Metrics.
NIFI-9425: Added auto-load NAR capability to MiNiFi.
NIFI-9400: Ensure that we always use the CollectionUsage metrics in the mbeans instead of the Usage metrics.
NIFI-9390: Updates to MergeContent / MergeRecord so that they play nicely within Stateless.
NIFI-9348 and NIFI-7863: Added temporary suffix and fixed NIFI-7863 creation of the directories.
NIFI-9341: Corrected annotation syntax problem.

Added CEF RecordReader.
NIFI-9327: Added timewindow query to QueryNiFiReportingTask and MetricsEventReportingTask.
NIFI-9316: Registry Sort by label should be 'Last Updated (newest)' not 'Newest (update)'.
NIFI-9293: Ensure that we properly set the scheduled flag in the LifecycleState when stopping processors. Added system test to verify that @OnScheduled, onTrigger, @OnUnscheduled, @OnStopped are all called and in the expected order.
NIFI-9286: JOLT Expression Language Fixes NIFI-6213 and adds in functionality to use expression language in class and module specification.

Adding JOLT unit tests.

Addressing PR feedback Fixes a problem with the scope of the EL for module directory.

Alignment of JOLT processors.

Fix checkstyle.
NIFI-9281: Enabled building on Java 17.
NIFI-9233: Improve reliability of system integration tests.
NIFI-9227: Run Once not working when scheduling strategy is CRON or Event driven.
NIFI-9166: Refactored nifi-standard-services to use JUnit 5.
NIFI-9134: nifi-metrics-reporting-bundle to use JUnit 5.
NIFI-9133: Refactored nifi-media-bundle to use JUnit 5.
NIFI-9124: Refactored nifi-jms-bundle to use JUnit 5.
NIFI-9113: Refactored nifi-grpc-bundle to use JUnit 5.
NIFI-9103: Refactored nifi-datadog-bundle to use JUnit 5.
NIFI-9085: Refactored the Elasticsearch bundle to use JUnit 5.
NIFI-9072: Improvements to ValidateXML.
NIFI-9065: Add support for OAuth2AccessTokenProvider in InvokeHTTP.
NIFI-9064: Support Oracle timestamp when 'Use Avro Logical Types' is true for ExecuteSQLRecord and QueryDatabaseTableRecord.
NIFI-9058: Corrected AttributesToJSON Core Attributes filtering.
NIFI-8927: Add option to start/stop all controllers.
NIFI-8899: Add NiFi Registry version information to the Registry under an "about" button.
NIFI-8676: Added 'Tracking Entities' listing strategy to 'ListS3' and 'ListGCSBucket'.
NIFI-8549: Upgraded MiNiFi sensitive properties algorithm.
NIFI-8521: Removed nifi-influxdb-nar package from nifi-assembly.
NIFI-8492: Addressed issue in DatabaseReader class of IPLookupService that was attempting to set values on the JSON returned by MaxMind. Instead of modifying the object directly, we should use an Injectable in the Reader so that the value read will have the appropriate values but we don't need to modify those objects returned by MaxMind. Similar solution of NIFI-5814.
NIFI-8209: Added Neo4J 4.X and 3.X clients by splitting the current controller service along release lines. This was necessary because Neo4J broke compatibility in their client drivers for Java between 3.X and 4.X at the Java API level.

Updated module name.

Updated parent module.

Renamed a few misnamed modules.

Updated 1.15.0-SNAPSHOT references in cypher v3 package.

Updated neo4j 4.x driver.
NIFI-8040: When changing version of a flow, stop processors that have a state of Starting in addition to those with a state of Running.
NIFI-7865: amqp$header is splitted in the wrong way for "," and "}".
NIFI-7840: Upgrade to Groovy 3.0.8 and Spock 2.1.
NIFI-7835: Added authenticated SOCKS proxy support for SFTP.
NIFI-7333: Added OIDC trust store strategy property.
NIFI-7192: Added systemd reload to nifi.sh install on systemd servers.
NIFI-6871: Added HikariCPConnectionPool controller service.
NIFI-6740: Add configuration options to specify NiFi/Bootstrap communication ports.
NIFI-6699: Corrected SFTP symbolic link handling (#5744).
NIFI-6390 and NIFI-1825: When we write to a FlowFile and that results in a 0-byte FlowFile, remove the content claim altogether. This is more efficient to process, but far more importantly it prevents a 0-byte FlowFile from holding content in the Content Repository. Also fixed issue in which a Provenance Event cannot be replayed if there is no ContentClaim.
NIFI-6266: Corrected proxy FTP connect handling.
NIFI-6047: Cleaned up code to allow tests to run against 1.13.0-snapshot Removed DMC.

Started integrating changes from NIFI-6014.

Added DMC tests.

Added cache identifier recordpath test.

Added additional details.

Removed old additional details.

Made some changes requested in a follow up review.

Finished updates First round of code review cleanup Latest Removed EL from the dynamic properties. Finished code review requested refactoring. Checkstyle fix. Removed a Java 11 API.

Renamed processor to DeduplicateRecord.