What's New in HDF 3.5.1
HDF 3.5.1 is a major release that includes new features, bug fixes, and improvements.
Apache NiFi Updates
NiFi has been upgraded from 1.9.0 to 1.11.4.
 | Note |
|---|---|
The NiFi version in HDF
3.5.1.0 is NiFi 1.11.4.3.5.1.0, which includes fixes that is available
in Apache NiFi 1.11.4 as well as additional improvements. |
New features and improvements in Apache NiFi include:
- Parameters. The parameters are a new concept in NiFi that allows users to define Parameter Contexts attached to process groups so that any property (including sensitive properties and properties not supporting expression language) can be parameterized. This is particularly useful in CI/CD pipelines when deploying workflows across multiple environments. Parameters should be used as a replacement of the variables.
- Predictive monitoring. NiFi has now an internal analytics
framework which can be enabled to predict back pressure
occurrence, given the configured settings for threshold on a
queue. It uses recent observations from a queue (either number
of objects or content size over time) and calculates predictions
based on a model to anticipate backpressure.
For more details on analytics properties, see Analytics Properties.
- Rules engine (Technical Preview). It is now possible to define a Rules Engine controller service to support execution of a centralized set of rules (stored as files or provided within the service configuration) against a provided set of data called facts. Upon execution, the rules engine will determine what rules have been met and return a list of actions that should be executed based on the conditions defined within the rules. This is particularly useful for workflow monitoring.
- Content and Flow File repositories encryption (Technical Preview).
In addition to the Provenance repository it is now possible to
encrypt both Content and Flow File repositories on disks to
comply with strict security requirements. This feature is in
technical preview.
For more details on flow file repository properties, see Encrypted Write Ahead FlowFile Repository Properties.
For more details on content repository properties, see Encrypted File System Content Repository Properties.
- SQL reporting task. The QueryNiFiReportingTask allows users to execute SQL queries against tables containing information on Connection Status, Processor Status, Bulletins, Process Group Status, JVM Metrics, Provenance and Connection Status Predictions. In combination with Site to Site, it is particularly useful to define fine-grained monitoring capabilities on top of the running workflows.
Apache Kafka Updates
Kafka has been upgraded from 2.1.0 to 2.3.1.
For more information on fixed Apache Kafka patches, see Release Notes - Kafka - Version 2.3.1.
Schema Registry Updates
Schema Registry has been upgraded from 0.7.0 to 0.8.1.
New features, improvements, and fixes in Schema Registry include:
- ISSUE 548: Make compatible with Avro 1.9.0.
- ISSUE 555 : Expose an option to disable canonical check while adding a new schema version.
- ISSUE 560 : Close unclosed ResultSet in storage.
- ISSUE 568 : Synchronize Kerberos ticket renewal for non-renewable ticket.
- ISSUE 569: Support Kerberos credentials based Basic Authentication.
- ISSUE 574 : Fix the issue while re-creating a deleted schema.
- ISSUE 575: Add migration to enable editing of schema post upgrade to 0.6.0.
- ISSUE 578: Find schema version by fingerprint of schema text.
- ISSUE 582 : Using OpenJDK intstead of OracleJDK for Travis builds.
- ISSUE 586: KerberosBasicAuthenticationHandler should validate Authorization before processing.
- ISSUE-588: Support for Oracle TLS configs in Storage layer.
- ISSUE-590 : Update release-sign-artifacts maven profile to generate javadocs and sources.
- ISSUE-592: Security changes to enable Trusted Proxy in SPNEGO authentication.
- ISSUE-596: Fixed the Bootstrap script error on passing "drop-create" option.
- ISSUE-598: Fix SR to run on Java 11.
- ISSUE-603: Registry is failing to parse the doAs user.
- ISSUE-605: Bump up the Avro version to 1.9.1.
- ISSUE-607: Schema Registry UI hangs occasionally on first use.
- ISSUE-608: Support one way SSL in schema registry client.
- ISSUE-610: Provide JAAS config dynamically via config.
- MINOR : Handle NPE when logging is enabled to debug mode
- MINOR : Update nexus url
- MINOR : Updated README.md
- MINOR : Doc for running schema registry behind a load balancer
- Hortonworks licenses changed to Cloudera licenses.
| Note |
|---|---|
In this release, license headers are updated for all the files.
In case you upgrade schema registry to 0.8.1 "bootstrap-storage.sh
migrate" would fail. Please run "bootstrap-storage.sh repair" before
the migrate command. |