What's New in HDF 3.2.0

Apache Kafka Updates

Kafka has been upgraded from 1.0.0 to 1.1.0 with some critical bug fixes including:

KAFKA-6172 - Cache lastEntry in TimeIndex to avoid unnecessary disk access
KAFKA-6175 - AbstractIndex should cache index file to avoid unnecessary disk access during resize()
KAFKA-6258 - SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data

In addition to these bug fixes, an important new feature was added to capture producer and topic partition level metrics without instrumenting or configuring interceptors on the clients.

This provides an non-invasive approach to capture important metrics for producers without refactoring/modifying your existing Kafka clients.

Apache Storm Updates

Storm has been upgraded from 1.1.0 to 1.2.1. Storm 1.2.1 now supports all HDP 3.0 components including Hadoop/HDFS 3.0, HBase 2.0 and Hive 3.

Key Bug Fixes include:

STORM-2993 - Storm HDFS bolt throws ClosedChannelException when Time rotation policy is used
BUG-104149 - Prevent supervisor crash when topology conf file doesn't exist in updateBlobsForTopology
BUG-94588 - Skip the user while checking isProcessAlive

For a full list of bug fixes, see the following:

http://rsync.mirror.anlx.net/apache/storm/apache-storm-1.2.2/RELEASE_NOTES.html

SAM Updates

New Features and Improvements in SAM include:

Rule, Projection and Aggregate processor supports writing complex conditions
Atlas integration – Ability to import ‘Atlas’ as a service and push SAM app information to Atlas during deployment
Compatibility with HDP 3.0 (Hive 3.0, Hbase 2.0, Hadoop/HDFS 3.0)
Test mode enhancement to support test mode without a cluster
Support for updating a SAM app environment

Key Bug Fixes include:

SAM view mode metrics related fixes
Custom processor handling null values

Schema Registry Updates

Schema Registry has been upgraded from 0.5.0 to 0.5.2 with some important improvements including:

ISSUE-441 Let addOrUpdate be compatible with PostgreSQL under 9.5
ISSUE-418 Add JWT based SSO as alternative authentication to common-auth module.
ISSUE-420 Add migration to add extents for created tables to fix serializable as an isolation level for Oracle database
ISSUE-415 Client side schema validation - Disable or Customize

Apache NiFi Updates

NiFi has been upgraded from 1.5.0 to 1.7.0. NiFi 1.7.0 now supports all HDP 3.0 components including Hadoop/HDFS 3.0, HBase 2.0 and Hive 3.

New Features:

NIFI-5214 - Add a REST lookup service
NIFI-5133 - Create a processor to Publish and Subscribe to Google Pub/Sub
NIFI-4982 - Add a DistributedMapCacheLookupService
NIFI-4963 - Add support for Hive 3.0 processors
NIFI-4917 - Create a new Controller Service for specifying Keytabs
NIFI-4906 - Add GetHdfsFileInfo Processor
NIFI-4872 - NIFI component high resource usage annotation
NIFI-4833 - Add ScanHBase processor
NIFI-4809 - Implement a SiteToSiteMetricsReportingTask
NIFI-4227 - Create a ForkRecord processor
NIFI-4185 - Add XML record reader
NIFI-5186 - Update UI to account for asynchronous validation

Key Bugs Fixed:

NIFI-5200 - Nested ProcessSession.read resulting in outer stream being closed.
NIFI-5033 - Cannot update variable referenced in restricted components
NIFI-4837 - Thread leak on HandleHTTPRequest processor

Apache NiFi Registry Updates

NiFi Registry has been upgrade from 0.1.0 to 0.2.0.

New Features:

NIFIREG-162 Add Git backed persistence provider
NIFIREG-173 Allow metadata DB to use other DBs besides H2
NIFIREG-172 Add self-hosted Swagger UI to nifi-registry-web-api WAR
NIFIREG-160 Implement a hook provider allowing users to create their own hook implementations that execute when a new flow is committed to the registry
NIFIREG-134 Enable Spring Boot Actuator REST API endpoints
NIFIREG-120 Docker image for NiFi Registry
NIFIREG-109 LdapUserGroupProvider: Allow admin to configure group membership attribute
NIFIREG-39 Leverage Apache NiFi Flow Design System

Key Bugs Fixed:

NIFIREG-174 - Parameter conf.dir in bootstrap.conf not working
NIFIREG-153 - The angular router module not properly injected.
NIFIREG-144 - NPE when accessing for metadata of 'latest' version and no versions exists
NIFIREG-140 - Nifi Registry not able to start - NoClassDefFoundError org/apache/nifi/registry/util/FileUtils
NIFIREG-126 - Entering an invalid bucket id in a deep link causes JS error

Apache MiNiFi Java Updates

MiNiFi Java has been upgraded from 0.4.0 to 0.5.0. Improvements include integrations with the MiNiFi C2 server.

New Features:

MINIFI-449 - Transform NiFi Registry's VersionedFlowSnapshot to YAML
MINIFI-441 - Improved Docker image
MINIFI-434 - Preserve security properties when pulling a new config

Key Bugs Fixed:

MINIFI-460 - SiteToSiteProvenanceReporting task fails to start
MINIFI-457 - Getting flow status for a processor returns "no processor"
MINIFI-456 - Exception when hitting MiNiFi C2
MINIFI-450 - PullHttpChangeIngestor needs to properly close OkHttp response
MINIFI-435 - MiNiFi - Cannot perform secured S2S with NiFi 1.5.0
MINIFI-431 - C2 - Exception while agent tries to pull the config

Apache MiNiFi C++ Updates

MiNiFi C++ has been upgraded from 0.4.0 to 0.5.0. Improvements around stability, extendible processors, and interactive ease of use have been made.

New Features:

MINIFICPP-508 - Support EL for ApplyTemplate Template property
MINIFICPP-476 - Support true/false literals as function args in EL
MINIFICPP-475 - Allow preferred image dimensions/mode to be set in GetUSBCamera properties
MINIFICPP-474 - Implement getDelimitedField EL function
MINIFICPP-472 - Implement date manipulation EL functions
MINIFICPP-471 - Implement string manipulation EL functions
MINIFICPP-469 - Implement base64 encode/decode EL functions
MINIFICPP-463 - Implement escape/unescape URL EL functions
MINIFICPP-445 - Implement escape/unescape CSV functions in expression language
MINIFICPP-436 - Implement encode/decode XML EL functions
MINIFICPP-434 - Create SQL (lite) processors
MINIFICPP-410 - MQTT should use controller service instead of a manual SSL configuration
MINIFICPP-401 - Implement Dynamic Relationships
MINIFICPP-397 - Implement RouteOnAttribute
MINIFICPP-394 - Implement MQTT C2 protocol
MINIFICPP-382 - Add SUSE support to bootstrap process.
MINIFICPP-31 - Support UpdateAttribute

Key Bugs Fixed:

MINIFICPP-495 - bootstrap.sh will exit with error in default install Debian systems
MINIFICPP-430 - GetFile incorrectly reports 0 file size for non-regular files
MINIFICPP-396 - RPG does not pass SSLContextService to site to site client in a consistent way
MINIFICPP-389 - FlexLexer.h missing, causing broken Alpine docker build
MINIFICPP-385 - RPG destruction can lead to EOFException in NiFi when sockets are not closed.