What's New in HDF 3.4.0
HDF 3.4.0 is a minor release that includes the following new features and bug fixes:
Apache Kafka Updates
We strongly discourage downgrading from Kafka 2.1.0 to earlier versions. Though Ambari allows you to downgrade to earlier versions of Kafka after you upgrade to 2.1.0, consider the following before you do so:
- KAFKA-4514 Add Codec for ZStandard Compression (KIP-110).
- KAFKA-4682 Committed offsets should not be deleted if a consumer is still active (KIP-211).
- KAFKA-5886 Introduce delivery.timeout.ms producer config (KIP-91).
- KAFKA-7264 Initial Kafka support for Java 11.
- KAFKA-7333 Protocol changes to allow fetchers to detect and handle log truncation (KIP-320).
- KAFKA-7394 Allow OffsetsForLeaderEpoch requests with topic describe ACL (KIP-320).
- KAFKA-7395 Add fencing to replication protocol (KIP-320).
- Streams API improvements (KIP-319, KIP-321, KIP-330, KIP-353, KIP-356).
- Admin script and admin client API improvements to simplify admin operation (KIP-231, KIP-308, KIP-322, KIP-324, KIP-338, KIP-340).
- DNS handling improvements (KIP-235, KIP-302).
For a full list of features and bug fixes, see the following: https://dist.apache.org/repos/dist/release/kafka/2.1.0/RELEASE_NOTES.html.
Apache NiFi Updates
NiFi has been upgraded from 1.8.0 to 1.9.0.
- New processors and controller services can be loaded live while the nodes are running.
- Database connection pooling now supports Kerberos secured Databases.
- Penalty duration remaining now visible in the UINodes can be offloaded to prepare for decommissioning.
- Several standard security headers now supported (HSTS, X-XSS-Protection, Content-Security-Policy).
- Record Readers and Writers now easier to use with automatic schema inference.
- New or Improved Processors and Controller Services
- Kudu Processors updated, validated, and now support Kerberos.
- Various updates to Amazon Web Services, Big Query, and Google Cloud Storage Processors.
- MongoDB GridFS processors now included.
- HiveQL for Hive 1.1 now supported.
For a full list of features and bug fixes, see the following: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344357.
Apache NiFi Registry Updates
NiFi Registry has been upgraded from 0.2.0 to 0.3.0.
- New Apache Ranger Authorizer allows authorization rules to be defined in Apache Ranger.
- Expansion and refinement of the Versioned Flow data model to facilitate easier use by consumers.
- Security enhancements: upgrades dependencies that have been patched for security issues.
- Event Hook API improvements
- New REGISTRY_START event type
- Event Type Whitelisting feature for Event Hook Providers
- Docker image improvements.
For a full list of features and bug fixes, see the following: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343483&styleName=Html&projectId=12320920
Schema Registry Updates
Schema Registry has been upgraded from 0.5.4 to 0.7.0.
- ISSUE-509: Provide a delete API for schema metadata (#510).
- ISSUE-496: Kafka Streams and Schema Registry Integration Support.
- ISSUE-500: Changes related to enabling JWT based SSO along with SPNEGO/Kerberos.
- EAR-9794: SchemaRegistry UI is not getting rendered properly.
- ISSUE-513: Fix storage manager to correctly handle the types in queries.
- ISSUE-517 Fix concurrent schema version adds.
- ISSUE-502: Field props stripped in deserialization.
- Update documentation: schema-registry.rst.
- ISSUE-543 Fix broken json format in Schema.Field when it's used as nested - introduced in Jackson 2.9added test which doesn't make sense for Streamline to have.
- ISSUE-538 Move transaction package from common to storage-core.
This helps us to isolate
registry-common, which brings huge number of transitive dependencies especially from
- ISSUE-540 Exclude Avro dependency from registry-common moduleThis
would save lots of modules which has
registry-commonas a transitive dependency.
- Upgrade jackson-databind to v2.9.8 to fix the following CVEs: CVE-2018-14719, CVE-2018-14720, CVE-2018-14721, CVE-2018-1000873, CVE-2018-7489, CVE-2018-19362, CVE-2017-15095, CVE-2018-19361, CVE-2017-7525, CVE-2018-19360, CVE-2017-17485, CVE-2018-5968.
- Remove deprecated API usages.
logback.xmlfile to adjust logger levels.
- fixed the typographical errors and broken links in documents.
- ISSUE-506: Resolve reported CVEs (CVE-2014-0114, CVE-2015-6420, CVE-2015-7501, CVE-2017-15708, CVE-2012-5783, CVE-2014-3577, CVE-2012-6153, CVE-2015-5262, CVE-2018-1288, CVE-2009-2625, CVE-2012-0881, CVE-2009-1523, CVE-2011-4461, CVE-2017-9735, CVE-2017-7656, CVE-2017-7657, CVE-2017-7658, CVE-2018-12536, CVE-2014-0193, CVE-2015-2156, CVE-2016-4970).
Apache Ranger Updates
Ranger plugin properties in Ambari and scripting requirements for NiFi Registry are enabled.