Fixed Issues in Flow Management

Review the list of Flow Management issues that are resolved in Cloudera DataFlow for Data Hub 7.2.15.

NIFI-9943
Added Transform Provider to nifi-xml-processing.
NIFI-9901
  • Removed provided scope from nifi-xml-processing registry-ranger.
  • Added nifi-xml-processing to nifi-commons.
NIFI-9835
Fixed threading bug in which NioAsyncLoadBalanceClient calls LoadBalanceSession.isComplete() followed by LoadBalanceSession.isCanceled() but it's possible for the complete flag to change before the canceled flag (they are not updated atomically). So changed to use a single LoadBalanceSessionState enum that represents the state. Also made the private StandardProcessSession.commit(boolean) method synchronized. When a processor is terminated (as is the case in Offload), we roll back sessions and both the commit() and rollback() need to be synchronized. Only the public commit() method was synchronized, and now with commitAsync() happening, we had the ability to commit without any synchronization. This addresses that concern. Also fixed a typo in docs for MergeRecord.
NIFI-9834
When calling ByteArrayContentRepository.read() on a null Content Claim, return an empty ByteArrayInputStream instead of throwing NullPointerException.
NIFI-9827
Upgrading AWS Java SDK to 1.12.182 to pick up new AWS Regions.
NIFI-9818
Fix flaky tests.
NIFI-9815
Corrected log message formatting in multiple classes.
NIFI-9807
Added Refresh Window Property to OAuth2 Token Provider.
NIFI-9806
Introduce ConfigurableExtensionDefinition and VersionedConfigurableExtension (#5875).
NIFI-9801
Fixed error in previous correction of AccessToken.isExpired() margin calculation NIFI-9801 Stabilized shaky AccessTokenTest.
NIFI-9800
Unwrap SQLException in PutDatabaseRecord when table does not exist.
NIFI-9799
Enabled style checking for nifi-system-tests in ci-workflow.
NIFI-9797
Corrected AccessToken.isExpired() margin calculation.
NIFI-9796
Updated Registry Security Configuration to avoid warnings.
NIFI-9795
Checkstyle, rat issues in nifi-system-test-suite module.
NIFI-9794
If a node is OFFLOADING, do not allow connections to be deleted. Also if we fail to notify the node that it needs to offload its data, change its state back to DISCONNECTED.
NIFI-9791
Use maven.build.timestamp during manifest generation instead of buildhelper timestamp.
NIFI-9790
Fixed race condition in which SwappablePriorityQueue could attempt to access the 0th element of an empty collection.
NIFI-9789
Upgraded Logback from 1.2.10 to 1.2.11.
NIFI-9788
Updated commons-codec to 1.15 across all modules.
NIFI-9786
Added debug to KeyStoreUtils.isStoreValid.
NIFI-9785
Improved Login Credentials Writer File Handling.
NIFI-9783
When migrating FlowFiles from one ProcessSession to another, if any FlowFile had already been transferred, and the Relationship to which it was transferred was auto-terminated, we were updating the wrong member variable, which threw off our stats for the processor. Fixed that.
NIFI-9782
Excluded H2 DB from nifi-druid-bundle.
NIFI-9781
Fix handling when selecting array element via QueryRecord.
NIFI-9778
Fixing additional details for ScriptedPartitionRecord.
NIFI-9777
Adding support to remove attributes from verification requests.
NIFI-9775
Create RuntimeManifestService.
NIFI-9774
Upgraded Netty from 4.1.73 to 4.1.74.
NIFI-9771
When a Kafka record is obtained during config verification, we should produce an invalid response if the Record Reader is not able to produce any records from it.
NIFI-9766
Avoid intermittent SearchElasticsearchTest failures in CI pipeline.
NIFI-9765
Added documentation that covers how to build a custom binar…
NIFI-9764
Atlas reporting task sends 'unknown' hive_table when table is name not available.
NIFI-9763
  • Additional escaped VALUE column due to H2 changes.
  • Escaped VALUE column for Configure Details Auditing.
NIFI-9762
Adding DBCPConnectionPool config verification.
NIFI-9761
Correct PeerChannel processing for TLS 1.3.
NIFI-9759
Upgraded Spring Framework from 5.3.15 to 5.3.16.
NIFI-9757
Upgraded SLF4J from 1.7.35 to 1.7.36.
NIFI-9756
Add documentation for framework-level retry in Processors and update processor tab images.
NIFI-9754
Introduced VersionedExternalFlow - Updated stateless and StandardProcessGroup, etc. to make use of VersionedExternalFlow - Updated StatelessDataflowDefinition to use ExternalVersionedFlow instead of generic type - Updated Stateless Bootstrap to avoid loading stateless engine libs from root classpath but instead use a NarClassLoader to load the stateless nar.
NIFI-9751
Poll as needed during system-tests to ensure expected state.
NIFI-9750
Logging Improvements to support LoadBalanceProtocol troubleshooting.
NIFI-9749
Capture additional logging for system-test workflow runs.
NIFI-9748
Added new property for Output Format to LogAttribute. Also made the FlowFile Properties (file size, entry date, lineage start date) optional and renamed from 'Standard FlowFile Attributes' to 'FlowFile Properties' because this has led to confusion many times in the past, around users wanting to reference these things as attributes via EL but they are not actually attributes.
NIFI-9747
Track PID in nifi-bootstrap logging on shutdown.
NIFI-9745
Prevent insertion of revisions in NiFi registry when revision feature is disabled.
NIFI-9743
Upgraded Jetty from 9.4.44 to 9.4.45.
NIFI-9741
Make the close() method of WriteAvroResultWithExternalSchema idempotent.
NIFI-9738
VersionedComponent data members should derive from Object.
NIFI-9736
Improved TestRouteText to avoid intermittent failures.
NIFI-9735
Corrected Jetty Duplicate Mapping Warning.
NIFI-9734
Standardized exception cause message formatting.
NIFI-9733
Fixing StandardConnection.verifySourceStoppedOrFunnel infinite recursion.
NIFI-9732
Upgraded Zip4j from 2.8.0 to 2.9.1.
NIFI-9731
Updated to use a shorter, simpler output format for FlowFiles when creating bulletins.
NIFI-9730
Consider a change in value for retry-related fields from 'null' to the default value as an environmental change so that it's not flagged as a Local Modification, which would prevent users from updating the version of the Process Group that they are using.
NIFI-9729
When restarting components in the VersionedFlowSynchronizer, first filter out any components that are intended to be stopped.
NIFI-9728
Added support for User Assigned Managed Identity authentication for Azure ADLS and Blob_v12 processors.
NIFI-9727
IndexOutOfBoundsException in CorrelationAttributePartitioner.
NIFI-9726
Removed duplicate nifi-utils dependencies from graph modules.
NIFI-9725
On shutdown, instead of spawning a background thread to shutdown Cluster Coordinator, do so in the calling thread. This avoids a race condition whereby the cluster coordinator cannot be determined because the other thread has shutdown the FlowController.
NIFI-9724
Added set-sensitive-properties-algorithm command.
NIFI-9723
When we add controller-level Controller Services on restart of NiFi, ensure that all Controller Services are updated to include their property values, etc. Also ensure that for these services and reporting tasks we decrypt the property values.
NIFI-9722
Do not throw an Exception from verifyCanUpdateProperties when property descriptor & parameter descriptor's sensitivities don't match - instead allow the set to happen and let processor become invalid. Also, allow values such as abc#{param} for ghost processors.
NIFI-9721
Support enum types in AvroTypeUtil.buildAvroSchema().
NIFI-9716 and NIFI-9577
Addressed issue in the PathFilter for GetFile / ListFile. For any file that is found in the Input Directory directly, it was previously being listed/fetched even if it didn't match the PathFilter. Additionally, updated the code to create a new File FIlter for every invocation of onTrigger. This was necessary for NIFI-9577 because the directory to monitor supports Expression Language and as a result may change from invocation to invocation, if using a function such as now() but the PathFilter would always relativize the path based on the value that was obtained when the processor was scheduled.
NIFI-9715
Add option to output empty FlowFile from Elasticsearch REST API Json Query processors when there are no hits from query.
NIFI-9714
Added overloaded toMap to MapRecord that can convert sub-records into maps.
NIFI-9713
TagS3Object do not have provenance data.
NIFI-9711
Added support for flow.json.gz in SetSensitivePropertiesKey.
NIFI-9707
Resolved duplicate JLine dependency in nifi-toolkit.
NIFI-9704
Updated the ContentRepositoryScanTask to show details of how much content in the content repo is retained by each queue in the dataflow. Changed default for nifi.content.claim.max.appendable.size property from 1 MB to 50 KB. Updated docs to reflect the new default value and explain what the property does and how it's used.
NIFI-9699
Updated oidcCallback method to handle error cases. Added some unit tests.
NIFI-9698
When creating an Avro schema, ensure that any default value is converted from what is returned by RecordField.getDefaultValue() to what Avro requires.
NIFI-9696
DeleteS3Object don't have provenance.
NIFI-9692
Upgraded Apache Commons Lang3 to 3.12.0.
NIFI-9691
Added ForkEnrichment, JoinEnrichment processors.
NIFI-9689
  • When checking FlowFile Availability, consider swap queue and trigger data to be swapped in, since calling poll() will no longer happen if no data is available.
  • When all FlowFiles in a FlowFile Queue are penalized, do not schedule the destination to run. Also expose this fact via the ConnectionStatusSnapshotDTO, as this allows the front-end to render this information to the user in order to avoid confusion when it appears that the Processor has data but does nothing.
NIFI-9688
Improve Logback shutdown handling.
NIFI-9687
Add additional documentation for nifi.cluster.node.protocol.max.threads property to Admin Guide.
NIFI-9686
Renamed SNMP integration tests correctly.
NIFI-9685
Upgraded JNA to 5.10.0.
NIFI-9684
Fix: When starting/stopping a selected process group, it sends the parent process group id to the REST interface that is responsible to enable/disable transmission for all remote process groups within a process group. Need to send the id of the select process group instead.
NIFI-9681
Upgraded Apache Commons DBCP to 2.9.0.
NIFI-9679
Added access-environment-credentials permission.
NIFI-9678
Update Elasticsearch REST API processor integration-tests for Elasticsearch 8.x.
NIFI-9673
Improved DBCP and HikariCP test reliability.
NIFI-9672
Fix flaky tests caused by the use of HashMap.
NIFI-9669
Adding PutDynamoDBRecord processor.
NIFI-9668
Adding informative error when setting same value in 'set-param' CLI command.
NIFI-9663
Setting the "csv-escape" property has no effect in SelectHive3QL.
NIFI-9662
Remove unused mail-1.4.7 dependency from nifi-framework-bundle.
NIFI-9660
Upgraded Apache Tika to 2.3.0.
NIFI-9657
Create MoveADLS processor.
NIFI-9655
Add Queue Logging to ListenUDP.
NIFI-9652
Upgraded jetty-schemas from 3.1 to 5.2.
NIFI-9650
Upgraded OkHttp from 4.9.2 to 4.9.3.
NIFI-9649
Upgraded SLF4J from 1.7.32 to 1.7.35.
NIFI-9647
Added ExtractDocumentText Processor.
NIFI-9645
Updated PutSplunk to allow idle connection timeouts.
NIFI-9644
Improved TestExecuteStateless increasing TestRunner.run() wait allowed.
NIFI-9642
Update Admin Guide and User Guide with correct nifi.properties default values.
NIFI-9641
Adjusted the extraction of the chroot suffix for solr client connections.
NIFI-9639
Determine how long it takes to find cluster coordinator and perform DNS lookup when sending heartbeats and include in the logs.
NIFI-9638
  • Refactored Google Guava usage in extensions.
  • Refactored Google Guava references.
NIFI-9635
Upgraded Netty from 4.1.72 to 4.1.73
NIFI-9634
Upgraded Spring Framework to 5.3.15.
NIFI-9632
Removed nifi-lumberjack-bundle.
NIFI-9631
Enable cli.sh to be used with a symbolic link.
NIFI-9630
Migrated Registry REST API docs to swagger-codegen.
NIFI-9629
Ensure that when we are setting default values on Avro GenericRecord objects that we convert from the schema's default value to the proper type.
NIFI-9628
Added a uiOnly flag when requesting Controller Service.
NIFI-9626
Allowing Stateless NiFi to parse flow snapshots with unrecognized fields.
NIFI-9625
  • Refactored Distributed Cache Server and Client Tests - Replaced TestServerAndClient with separate classes for Set Server and Map Server - Implemented before and after annotations for starting and stopping server instances.
  • Added check for cache directory existence before clean NIFI-9625 Updated Map and Set Cache Server Tests to use random port.
NIFI-9624
Removed JCenter Repository.
NIFI-9621
Added Ignore Reserved Characters to FlattenJson.
NIFI-9620
Adding isStateful to StatelessDataflow.
NIFI-9619
Removed GPG key from Security Mailing List reporting.
NIFI-9618
Upgraded Checkstyle to 9.2.1.
NIFI-9617
Removed unused screenshots from documentation.
NIFI-9616
Included SLF4J bridge libraries in NiFi Stateless Kafka Connect assembly.
NIFI-9611
  • Restore commons-io to minifi-assembly.
  • Removed unnecessary references to nifi-processor-utils.
  • Removed duplicate nifi-utils dependency.
NIFI-9610
Refactored nifi-processor-utils to separate modules.
NIFI-9609
Added nifi-snowflake-bundle with SnowflakeComputingConnectionPool.
NIFI-9608
Disabled system-tests workflow for pull requests.
NIFI-9607
Honor Update Keys when Quoting Identifiers in PutDatabaseRecord.
NIFI-9606
Removed nifi-security-utils from nifi-framework-api.
NIFI-9601
Upgraded nifi-bootstrap to Jakarta Mail 2.
NIFI-9600
Removed Elasticsearch 2 Processors.
NIFI-9599
Updating explicit plugin/build version references.
NIFI-9597
  • Fixing all explicit version refs to main latest.
  • Fix Dockerfile URLs.
NIFI-9596
Fix newline bug in JythonScriptRunner NIFI-9596: Added comment to indicate why the Apache header is missing.
NIFI-9595
Removed nifi-kafka-0.x modules.
NIFI-9594
When converting Record to Avro GenericRecord, ensure that any default values that are defined in the GenericRecord's schema get applied, regardless of whether or not the field exists in the associated RecordSchema.
NIFI-9593
Missing catch clauses in Confluent Schema Registry client.
NIFI-9591
Removed nifi-kite-bundle.
NIFI-9590
Added support for sensitive properties in Azure authorizers to encrypt-config.
NIFI-9589
Support initial loading from the current max values in QueryDatabaseTable* processors.
NIFI-9588
Update doc for 'nifi.content.repository.archive.max.retention.period'.
NIFI-9587
Added JSON format for Prometheus Flow Metrics.
NIFI-9586
  • Removed Surefire ForkNodeFactory configuration.
  • Excluded additional assembly modules from ci-workflow.
NIFI-9585
Upgraded H2 from 1.4 to 2.1.210.
NIFI-9581
Add PutElasticsearchRecord relationship for output of successful Records sent to Elasticsearch.
NIFI-9580
UI work for framework-level retry in Processors.
NIFI-9576
  • Introduced a BlockListClassLoader that can be used by stateless in order to isolate both the Stateless Engine and the NiFi extensions from extraneous classes that exist in the System ClassLoader.
  • Allowed Stateless' BlockListClassLoader to load java11 jars/classes.
NIFI-9575
Updating copyright year to 2022.
NIFI-9571
Corrected Session commit handling in PutTCP.
NIFI-9570
Separate classpath for NiFi Registry sensitive property providers.
NIFI-9569
SNMP manager UDP transportmapping changed to 0.0.0.0.
NIFI-9568
Updated nifi-jolt-transform-json-ui pom.xml to only include CSS and JS assets in WAR.
NIFI-9564
Removed unnecessary logback-classic test dependencies.
NIFI-9563
Enabled ListenTCP Pool Receive Buffers property.
NIFI-9552
Fix NoClassDefFound error in case of nifi-registry-ranger-assembly.
NIFI-9548
When disabling RPG transmission, wait for the ports to complete in a background thread instead of blocking the web thread. Also moved the RPG initialization logic into flow controller instead of flow service and added a delay in order to reduce likelihood of ConnectException happening when pointing to nodes in the same cluster.
NIFI-9545
Fix in-place replacement for LookupRecord processor.
NIFI-9544
LookupRecord - fixed behavior when no matching value in the LRS.
NIFI-9543
Add bring-to-front functionality to labels.
NIFI-9527
Upgraded snappy-java to 1.1.8.4.
NIFI-9525
Modify lib packaging to use files from build directory.
NIFI-9501
Added REST end-point to retrieve a RuntimeManifest.
NIFI-9481
Excluded Data Transfer REST methods from DoSFilter.
NIFI-9475
Provide Framework-Level Retries for NiFi Relationships.
NIFI-9455
Added aggregated predictions to Prometheus Flow Metrics.
NIFI-9453
Refactored ListenBeats using Netty.
NIFI-9438
Refactored sensitive-property-provider to multiple modules.
NIFI-9435
Added registries and names include parameters to Flow Metrics.
NIFI-9425
Added auto-load NAR capability to MiNiFi.
NIFI-9400
Ensure that we always use the CollectionUsage metrics in the mbeans instead of the Usage metrics.
NIFI-9390
Updates to MergeContent / MergeRecord so that they play nicely within Stateless.
NIFI-9348 and NIFI-7863
Added temporary suffix and fixed NIFI-7863 creation of the directories.
NIFI-9341
  • Corrected annotation syntax problem.
  • Added CEF RecordReader.
NIFI-9327
Added timewindow query to QueryNiFiReportingTask and MetricsEventReportingTask.
NIFI-9316
Registry Sort by label should be 'Last Updated (newest)' not 'Newest (update)'.
NIFI-9293
Ensure that we properly set the scheduled flag in the LifecycleState when stopping processors. Added system test to verify that @OnScheduled, onTrigger, @OnUnscheduled, @OnStopped are all called and in the expected order.
NIFI-9286
  • JOLT Expression Language Fixes NIFI-6213 and adds in functionality to use expression language in class and module specification.
  • Adding JOLT unit tests.
  • Addressing PR feedback Fixes a problem with the scope of the EL for module directory.
  • Alignment of JOLT processors.
  • Fix checkstyle.
NIFI-9281
Enabled building on Java 17.
NIFI-9233
Improve reliability of system integration tests.
NIFI-9227
Run Once not working when scheduling strategy is CRON or Event driven.
NIFI-9166
Refactored nifi-standard-services to use JUnit 5.
NIFI-9134
nifi-metrics-reporting-bundle to use JUnit 5.
NIFI-9133
Refactored nifi-media-bundle to use JUnit 5.
NIFI-9124
Refactored nifi-jms-bundle to use JUnit 5.
NIFI-9113
Refactored nifi-grpc-bundle to use JUnit 5.
NIFI-9103
Refactored nifi-datadog-bundle to use JUnit 5.
NIFI-9085
Refactored the Elasticsearch bundle to use JUnit 5.
NIFI-9072
Improvements to ValidateXML.
NIFI-9065
Add support for OAuth2AccessTokenProvider in InvokeHTTP.
NIFI-9064
Support Oracle timestamp when 'Use Avro Logical Types' is true for ExecuteSQLRecord and QueryDatabaseTableRecord.
NIFI-9058
Corrected AttributesToJSON Core Attributes filtering.
NIFI-8927
Add option to start/stop all controllers.
NIFI-8899
Add NiFi Registry version information to the Registry under an "about" button.
NIFI-8676
Added 'Tracking Entities' listing strategy to 'ListS3' and 'ListGCSBucket'.
NIFI-8549
Upgraded MiNiFi sensitive properties algorithm.
NIFI-8521
Removed nifi-influxdb-nar package from nifi-assembly.
NIFI-8492
Addressed issue in DatabaseReader class of IPLookupService that was attempting to set values on the JSON returned by MaxMind. Instead of modifying the object directly, we should use an Injectable in the Reader so that the value read will have the appropriate values but we don't need to modify those objects returned by MaxMind. Similar solution of NIFI-5814.
NIFI-8209
  • Added Neo4J 4.X and 3.X clients by splitting the current controller service along release lines. This was necessary because Neo4J broke compatibility in their client drivers for Java between 3.X and 4.X at the Java API level.
  • Updated module name.
  • Updated parent module.
  • Renamed a few misnamed modules.
  • Updated 1.15.0-SNAPSHOT references in cypher v3 package.
  • Updated neo4j 4.x driver.
NIFI-8040
When changing version of a flow, stop processors that have a state of Starting in addition to those with a state of Running.
NIFI-7865
amqp$header is splitted in the wrong way for "," and "}".
NIFI-7840
Upgrade to Groovy 3.0.8 and Spock 2.1.
NIFI-7835
Added authenticated SOCKS proxy support for SFTP.
NIFI-7333
Added OIDC trust store strategy property.
NIFI-7192
Added systemd reload to nifi.sh install on systemd servers.
NIFI-6871
Added HikariCPConnectionPool controller service.
NIFI-6740
Add configuration options to specify NiFi/Bootstrap communication ports.
NIFI-6699
Corrected SFTP symbolic link handling (#5744).
NIFI-6390 and NIFI-1825
When we write to a FlowFile and that results in a 0-byte FlowFile, remove the content claim altogether. This is more efficient to process, but far more importantly it prevents a 0-byte FlowFile from holding content in the Content Repository. Also fixed issue in which a Provenance Event cannot be replayed if there is no ContentClaim.
NIFI-6266
Corrected proxy FTP connect handling.
NIFI-6047
  • Cleaned up code to allow tests to run against 1.13.0-snapshot Removed DMC.
  • Started integrating changes from NIFI-6014.
  • Added DMC tests.
  • Added cache identifier recordpath test.
  • Added additional details.
  • Removed old additional details.
  • Made some changes requested in a follow up review.
  • Finished updates First round of code review cleanup Latest Removed EL from the dynamic properties. Finished code review requested refactoring. Checkstyle fix. Removed a Java 11 API.
  • Renamed processor to DeduplicateRecord.