Chapter 1. Hortonworks DataFlow 3.0.0 Release Notes

This document provides you with the latest information about the HDF 3.0.0 release and its product documentation.

Component Support

HDF 3.0.0 includes the following components:

Apache Ambari 2.5.1
Apache Kafka 0.10.2.1
Apache NiFi 1.2.0
Apache Ranger 0.7.0
Apache Storm 1.1.0
Apache ZooKeeper 3.4.6
Apache MiNiFi Java Agent 0.2.0
Apache MiNiFi C++ Technical Preview
Hortonworks Schema Registry 0.3.0
Hortonworks Streaming Analytics Manager 0.5.0

Component Availability in HDF

Previous HDF releases shipped with the following components versions.

	NiFi	Storm	Kafka	ZooKeeper	Ambari	Ranger	MiNiFi Java Agent	MiNiFi C++	Streaming Analytics Manager	Schema Registry
HDF 3.0.0	1.2.0	1.1.0	0.10.2.1	3.4.6	2.5.1	0.7.0	0.2.0	TP	0.5.0	0.3.0
HDF 2.1.4	1.1.0	1.0.2	1.10.1	3.4.6	2.4.2.0	0.6.2	0.1.0	TP	EA	EA
HDF 2.1.2	1.1.0	1.0.2	0.10.1	3.4.6	2.4.2.0	0.6.2	0.1.0	TP	EA	EA
HDF 2.1.1	1.1.0	1.0.2	0.10.1	3.4.6	2.4.2.0	0.6.2	0.1.0	TP	EA	EA
HDF 2.1.0	1.1.0	1.0.2	0.10.1	3.4.6	2.4.2.0	0.6.2	0.1.0	TP	EA	EA
HDF 2.0.2	1.0.0	1.0.1	0.10.0.1	3.4.6	2.4.1.0	0.6.0	0.0.1	TP	N/A	N/A
HDF 2.0.1	1.0.0	1.0.1	0.10.0.1	3.4.6	2.4.1.0	0.6.0	0.0.1	TP	N/A	N/A
HDF 2.0.0	1.0.0	1.0.1	0.10.0.1	3.4.6	2.4.0.1	0.6.0	0.0.1	TP	N/A	N/A
HDF 1.2.1	0.6.1	0.10.0	0.9.0.1	3.4.6	N/A	N/A	N/A	N/A	N/A	N/A
HDF 1.2.0.1	0.6.1	0.10.0	0.9.0.1	3.4.6	N/A	N/A	N/A	N/A	N/A	N/A
HDF 1.2.0	0.6.0	0.10.0	0.9.0.1	3.4.6	N/A	N/A	N/A	N/A	N/A	N/A
HDF 1.1.0	0.4.0	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
HDF 1.0	0.3.0	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A

What's New in HDF 3.0.0

Component	New features
Streaming Analytics Manager	Stream Analytics Manager is a new HDF component. Use Streaming Analytics Manager to design, develop, deploy and manage streaming analytics apps with a drag-and-drop visualization paradigm. Streaming Analytics Manager allows you to build streaming analytics apps for event correlation, context enrichment, complex pattern matching, and analytic aggregations. You can create alerts and notifications when insights are discovered. Streaming Analytics Manager is agnostic to the underlying streaming engine, and it can support multiple streaming substrates such as Storm, Spark Streaming, Flink, etc. The first streaming engine fully supported is Apache Storm.
Schema Registry	Schema Registry is a new HDF component. Schema Registry improves end-to-end data governance and operational efficiency by providing a centralized registry, supporting version management and enabling schema validation. There are three major benefits for this new module: Centralized registry – A shared repository of schemas eliminates the need to attach schema to every piece of data. Applications can flexibly interact with each other in order to save or retrieve schemas for the data they need to access. Fully integrated with the flow management component of HDF, including NiFi, Schema Registry allows schemas created using NiFi to be easily managed and reused by the entire platform. Version management – Defines relationships between schemas and enables schemas to be shared between HDF components and applications. Schema Registry supports schema evolution so that a consumer and producer can understand different schema versions but still read all the information shared between them. Schema validation – Schema Registry supports schema validation by enabling generic format conversion and generic routing to ensure data quality.
HDF Flow Management	Powered by Apache NiFi 1.2.0 now supports: Running multiple versions of the same component Real-time data provenance indexing New change data capture options such as reading directly from MySQL binlogs Record oriented capability to acquire and route event data with easy schema aware transformations between formats such as Avro, JSON, CSV

Component

New features

Streaming Analytics Manager

Stream Analytics Manager is a new HDF component.

Use Streaming Analytics Manager to design, develop, deploy and manage streaming analytics apps with a drag-and-drop visualization paradigm. Streaming Analytics Manager allows you to build streaming analytics apps for event correlation, context enrichment, complex pattern matching, and analytic aggregations. You can create alerts and notifications when insights are discovered.

Streaming Analytics Manager is agnostic to the underlying streaming engine, and it can support multiple streaming substrates such as Storm, Spark Streaming, Flink, etc. The first streaming engine fully supported is Apache Storm.

Schema Registry

Schema Registry is a new HDF component.

Schema Registry improves end-to-end data governance and operational efficiency by providing a centralized registry, supporting version management and enabling schema validation.

There are three major benefits for this new module:

Centralized registry – A shared repository of schemas eliminates the need to attach schema to every piece of data. Applications can flexibly interact with each other in order to save or retrieve schemas for the data they need to access. Fully integrated with the flow management component of HDF, including NiFi, Schema Registry allows schemas created using NiFi to be easily managed and reused by the entire platform.
Version management – Defines relationships between schemas and enables schemas to be shared between HDF components and applications. Schema Registry supports schema evolution so that a consumer and producer can understand different schema versions but still read all the information shared between them.
Schema validation – Schema Registry supports schema validation by enabling generic format conversion and generic routing to ensure data quality.

HDF Flow Management

Running multiple versions of the same component
Real-time data provenance indexing
New change data capture options such as reading directly from MySQL binlogs
Record oriented capability to acquire and route event data with easy schema aware transformations between formats such as Avro, JSON, CSV

Unsupported Features

Some features exist within HDF 3.0.0, but Hortonworks does not currently support these capabilities.

Technical Preview Features

The following features are available within HDF 3.0.0 but are not ready for production deployment. Hortonworks encourages you to explore these technical preview features in non-production environments and provide feedback on your experiences through the Hortonworks Community Forums.

Table 1.1. Technical Previews

Component	Feature
MiNiFi	MiNiFi C++
Streaming Analytics Manager	SAM Stream Insights module (Druid and Apache Superset) Sinks Cassandra OpenTSDB Solr Sources HDFS

Community Driven Features

The following features are developed and tested by the Hortonworks community but are not officially supported by Hortonworks. These features are excluded for a variety of reasons, including insufficient reliability or incomplete test case coverage, declaration of non-production readiness by the community at large, and feature deviation from Hortonworks best practices. Do not use these features in your production environments.

Community Driven Kafka features

Kafka Connect
Kafka Streams

Community Driven NiFi Tools and Services

Embedded ZooKeeper
Sensitive key migration toolkit
Docker image for Apache NiFi

Community Driven NiFi Processors

AttributeRollingWindow
AWSCredentialsProviderControllerService
CompareFuzzyHash
ConsumeEWS
ConsumeIMAP
ConsumePOP3
ConvertExcelToCSVProcessor
DebugFlow
DeleteDynamoDB
DeleteGCSObject
DeleteHDFS
ExtractCCDAAttributes
ExtractEmailAttachments
ExtractEmailHeaders
ExtractMediaMetadata
ExtractTNEFAttachments
FetchAzureBlobStorage
FetchGCSObject
FuzzyHashContent
GetDynamoDB
GetHDFSEvents
GetSNMP
ISPEnrichIP
InferAvroSchema
ListenBeats
ListenLumberjack
ListenSMTP
ListAzureBlobStorage
ListGCSBucket
ListS3
ModifyBytes
OrcFormatConversion
PutAzureBlobStorage
PutDynamoDB
PutGCSObject
PutIgniteCache
PutKinesisFirehose
PutKinesisStream
PutLambda
PutSlack
PutTCP
PutUDP
QueryDNS
SetSNMP
SpringContextProcessor
StoreInKiteDataset

Community Driven NiFi Controller Services

AWSCredentialsProviderControllerService
GCPCredentialsControllerService

Community Driven NiFi Reporting Tasks

SiteToSiteBulletinReportingTask
SiteToSiteStatusReportingTask
DataDogReportingTask

Unsupported Customizations

Hortonworks cannot guarantee that default NiFi processors are compatible with proprietary protocol implementations or proprietary interface extensions. For example, we support interfaces like JMS and JDBC that are built around standards, specifications, or open protocols. But we do not support customizations of those interfaces, or proprietary extensions built on top of those interfaces.

Deprecated Technologies

This section points out any technology from previous releases that has been deprecated or removed from this release (operating systems, Java versions, databases, product features). Use this section as a guide for your implementation plans.

Deprecated: Technology that Hortonworks is removing in a future release. Deprecated items are supported until they are removed; deprecation gives you time to plan for removal.
Removed: Technology that Hortonworks has removed from production and is no longer supported.

Table 1.2. Deprecated Operating Systems

Operating System	Release Deprecated	Release Removed
Debian 6	HDF 2.1.2	HDF 3.0.0
Ubuntu 12	HDF 3.0.0	HDF 3.0.0

Table 1.3. Deprecated NiFi Processors

Processor	Release Deprecated
ConvertCSVToAvro	HDF 3.0.0
ConvertJSONToAvro	HDF 3.0.0
GetKafka	HDF 2.0.0
PutKafka	HDF 2.0.0
EvaluateRegularExpression	HDF 1.0.0

HDF Repository Locations

Use the following table to identify the HDF 3.0.0 repository location for your operating system and operational objectives. HDF 3.0.0 supports the following operating systems:

Table 1.4. RHEL/Oracle Linux/CentOS 6 HDF repository & additional download locations

OS	Format	Download location
RHEL/Oracle Linux/CentOS 6 (64-bit):	HDF Base URL	http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.0.0
	HDF Repo	http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.0.0/hdf.repo
	RPM tarball	http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.0.0/HDF-3.0.0.0-centos6-rpm.tar.gz
	Tars tarball	http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.0.0/HDF-3.0.0.0-centos6-tars-tarball.tar.gz
	HDF Management Pack	http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.0.0-453.tar.gz
	HDP and Ambari Repositories
	Ambari	http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.1.0/ambari.repo
	HDP	http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.1.0/hdp.repo
	OS Agnostic Downloads
	NiFi only	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.tar.gz
	NiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Java Agent	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.tar.gz

Table 1.5. RHEL/Oracle Linux/CentOS 7 HDF repository & additional download locations

OS	Format	Download location
RHEL/Oracle Linux/CentOS 7 (64-bit):	HDF Base URL	http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.0.0
	HDF Repo	http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.0.0/hdf.repo
	RPM tarball	http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.0.0/HDF-3.0.0.0-centos7-rpm.tar.gz
	Tars tarball	http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.0.0/HDF-3.0.0.0-centos7-tars-tarball.tar.gz
	HDF Management Pack	http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.0.0-453.tar.gz
	MiNiFi C++ Agent	http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.0.0/tars/nifi-minifi-cpp/nifi-minifi-cpp-0.2.0-bin.tar.gz
	HDP and Ambari Repositories
	Ambari	http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.1.0/ambari.repo
	HDP	http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.1.0/hdp.repo
	OS Agnostic Downloads
	NiFi only	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.tar.gz
	NiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Java Agent	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.tar.gz

Table 1.6. SLES 11 SP3/SP4 HDF repository & additional download locations

OS	Format	Download location
SUSE Enterprise Linux 11 SP3, SP4	HDF Base URL	http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.0.0
	Repo	http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.0.0/hdf.repo
	RPM tarball	http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.0.0/HDF-3.0.0.0-suse11sp3-rpm.tar.gz
	Tars tarball	http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.0.0/HDF-3.0.0.0-suse11sp3-tars-tarball.tar.gz
	HDF Management Pack	http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.0.0-453.tar.gz
	HDP and Ambari Repositories
	Ambari	http://public-repo-1.hortonworks.com/ambari/suse11/2.x/updates/2.5.1.0/ambari.repo
	HDP	http://public-repo-1.hortonworks.com/HDP/suse11sp3/2.x/updates/2.6.1.0/hdp.repo
	OS Agnostic Downloads
	NiFi only	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.tar.gz
	NiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Java Agent	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.tar.gz

Table 1.7. SLES 12 HDF repository & additional download locations

OS	Format	Download location
SUSE Linux Enterprise Server (SLES) v12 SP1	HDF Base URL	http://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.0.0
	Repo	http://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.0.0/hdf.repo
	RPM tarball	http://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.0.0/HDF-3.0.0.0-sles12-rpm.tar.gz
	Tars tarball	http://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.0.0/HDF-3.0.0.0-sles12-tars-tarball.tar.gz
	HDF Management Pack	http://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.0.0-453.tar.gz
	HDP and Ambari Repositories
	Ambari	http://public-repo-1.hortonworks.com/ambari/sles12/2.x/updates/2.5.1.0/ambari.repo
	HDP	http://public-repo-1.hortonworks.com/HDP/sles12/2.x/updates/2.6.1.0/hdp.repo
	OS Agnostic Downloads
	NiFi only	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.tar.gz
	NiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Java Agent	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.tar.gz

Table 1.8. Ubuntu 14 HDF repository & additional download locations

OS	Format	Download location
Ubuntu Trusty (14.04) (64-bit)	HDF Base URL	http://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.0.0
	Repo	http://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.0.0/hdf.list
	Deb tarball	http://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.0.0/HDF-3.0.0.0-ubuntu14-deb.tar.gz
	Tars tarball	http://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.0.0/HDF-3.0.0.0-ubuntu14-tars-tarball.tar.gz
	HDF Management Pack	http://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.0.0-453.tar.gz
	MiNiFi C++ Agent	http://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.0.0/tars/nifi-minifi-cpp/nifi-minifi-cpp-0.2.0-bin.tar.gz
	HDP and Ambari Repositories
	Ambari	http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.5.1.0/ambari.list
	HDP	http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.6.1.0/hdp.list
	OS Agnostic Downloads
	NiFi only	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.tar.gz
	NiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Java Agent	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.tar.gz

Table 1.9. Debian 7 HDF repository & additional download locations

OS	Format	Download location
Debian 7	HDF Base URL	http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.0.0
	Repo	http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.0.0/hdf.list
	Deb tarball	http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.0.0/HDF-3.0.0.0-debian7-deb.tar.gz
	Tars tarball	http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.0.0/HDF-3.0.0.0-debian7-tars-tarball.tar.gz
	HDF Management Pack	http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.0.0-453.tar.gz
	HDP and Ambari Respositories
	Ambari	http://public-repo-1.hortonworks.com/ambari/debian7/2.x/updates/2.5.1.0/ambari.list
	HDP	http://public-repo-1.hortonworks.com/HDP/debian7/2.x/updates/2.6.1.0/hdp.list
	OS Agnostic Downloads
	NiFi only	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-1.2.0.3.0.0.0-453-bin.tar.gz
	NiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/nifi-toolkit-1.2.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Java Agent	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-1.0.3.0.0.0-453-bin.tar.gz
	MiNiFi Toolkit	Zip File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.zip Tar.gz File – http://public-repo-1.hortonworks.com/HDF/3.0.0.0/minifi-toolkit-1.0.3.0.0.0-453-bin.tar.gz

Behavioral Changes

NiFi 1.2 supports TLS version 1.2 and later. If you are using custom code that accesses the NiFi API, ensure that you have upgraded to TLS version 1.2 or later.

Apache Patch Information

The following sections list patches in each HDF 3.0.0 component beyond what was fixed in the base version of the Apache component.

NiFi

HDF 3.0.0 provides NiFi 1.2.0 and the following Apache patches:

NIFI-391: Need ability to deprecate components.
NIFI-1452: Yield Duration can short circuit long Timer Driven Run Schedule.
NIFI-1709: The nifi.sh install script assumes RHEL directories.
NIFI-1963: Cluster - Flow inheritability
NIFI-2199: NiFi cannot be restarted through SSH (nifi.sh hangs).
NIFI-2624: JDBC-to-Avro processors handle BigDecimals as Strings.
NIFI-2702: GetJMS vs ConsumeJMS
NIFI-3112: When running "nifi.sh start" errors that occur in the bootstrap are routed to dev/null.
NIFI-3191: HDFS Processors Should Allow Choosing LZO Compression.
NIFI-3404: Add lookup processor for enrichments/joins to reference data.
NIFI-3568: Default thread pool that is created for cluster request replication is not sufficient.
NIFI-3640: GetEventHub should not assume domain names.
NIFI-3644: Add DetectDuplicateUsingHBase processor.
NIFI-3671: GenerateFlowFile throws NPE in FileSystemRepository.
NIFI-3707: UI - sort attributes on List Queue display.
NIFI-3719: Timezone error in 5-minute statistics processing time.
NIFI-3732: Distributed cache clients should use a timeout when opening a connection.
NIFI-3761: testFullyDocumentedProcessor assertion never evals to false.
NIFI-3763: Add new processor to log user defined messages built with NiFi Expression Language.
NIFI-3771: Move HL7 test docs into class members to avoid git rewriting CRs.
NIFI-3791:SiteToSiteStatusReportingTask - include back pressure data for connections.
NIFI-3802: Update site/evangelize how to access ASF NiFi Hipchat room.
NIFI-3809: S2S Reporting tasks should provide the option to choose transport protocol.
NIFI-3816: Add JSON mime.type attribute for S2S Provenance and Bulletins reporting tasks.
NIFI-3830: Clicking on any component on Linux causes %3F to continuously be added to beginning of query string.
NIFI-3832: AvroRecordSetWriter initial validation fails with NullPointerException.
NIFI-3837: ReplaceText - Backreference escape issue in Regex Replace mode when using EL.
NIFI-3838: Create UpdateRecord processor and provide a RecordPath DSL to specify fields.
NIFI-3848: Refresh client's component revision data when node reconnects to cluster.
NIFI-3850: Error in Admin Guide web property nifi.web.jetty.working.directory.
NIFI-3851: Add expression language support to AWS Endpoint Override URL.
NIFI-3852: Add expression language support to Cassandra processors.
NIFI-3853: Escape configurations prior to writing flow.xml.
NIFI-3854: Expand expression language support for Kafka processors.
NIFI-3855: Add attachment support to PutSlack.
NIFI-3856: PutCloudWatchMetric support for StatisticSets and Dimensions.
NIFI-3857: Create PartitionRecord processor.
NIFI-3859: Provide filtering options in S2SProvenanceReportingTask.
NIFI-3860: Consider relaxing the constraint that ProcessSession enforces we give it the most recent version of a FlowFile.
NIFI-3861: AvroReader logical date and time conversion does not work with nullable type.
NIFI-3863: Create a LookupRecord Processor.
NIFI-3864: Update RM guide to handle Docker build preconditions.
NIFI-3865: Unable to view content when clustered.
NIFI-3867: Add Expression Language support to HiveConnectionPool.
NIFI-3868: NullPointerException when trying to retrieve bulletins if insufficient permissions.
NIFI-3871: Converting Avro Maps
NIFI-3873: Content viewer to display Avro logical types more human readable way.
NIFI-3876: UI hanging if using a blank name for a processor.
NIFI-3878: Update AmbariReportingTask to set 1 minute as default run schedule.
NIFI-3879: Null Avro Values.
NIFI-3881: PutHiveStreaming properties not properly evaluating EL.
NIFI-3885: Add expression language support to remaining Dynamo properties.
NIFI-3894: NiFi Crashes with OOM if compression is enabled on an RPG with high number of FlowFiles.
NIFI-3895: AvroReader 'Schema Access Strategy' defaults to 'Use Embedded Avro Schema' but it doesn't use embedded schema by default.
NIFI-3896: DeprecationNotice usage is counter-intuitive.
NIFI-3900: Inconsistent schedule state across the cluster.
NIFI-3901: Web Api - Address minor issues with the Provenance API.
NIFI-3903: Basic Unit Test Coverage for AWS Processors.
NIFI-3904: Updating component can unnecessarily reload component.
NIFI-3906: UI - Additional user input validation.
NIFI-3907: Web Api - Response headers.
NIFI-3908: Changing version on processors is not filtering by type.
NIFI-3909: PublishKafka transfer error on empty record input.
NIFI-3910: ConvertRecord or CSVReader not handling input data which has more columns than the schema properly.
NIFI-3912: Schema write strategy validation throws NPE in FreeFormTextRecordSetWriter.
NIFI-3914: Remote Process Group will not Enable if any Input Port or Output Port is not connected.
NIFI-3917: CSV Reader and Writer services should assume there is no header by default.
NIFI-3918: JsonTreeReader doesn't read Choice data type.
NIFI-3919: AvroTypeUtil throws Exception before trying every possible type.
NIFI-3922: Inconsistency in Resume, Stop, Start of Processors/Controller Services.
NIFI-3923: When clustered, nodes can start processors before repositories have finished initializing.
NIFI-3925: Add Template list with long template name.
NIFI-3932: Symbols in API comments cause problems with Swagger Codegen.
NIFI-3933: Cluster - Cluster Coordinator removal due to lack of heartbeats.
NIFI-3935: TestPersistentProvenanceRepository frequently fails in travis / likely slow machine timing bug in test.
NIFI-3938: Add a ScriptedLookupService.
NIFI-3940: Oder jruby-complete artifacts contained LGPL code.
NIFI-3942: IPLookupService should auto reload geo dictionary file.
NIFI-3943: Help/hover text is misaligned in the Scope drop-down in the Enable Controller Service dialog.
NIFI-3945: Improve documentation around security for 0.10 Kafka processors.
NIFI-3946: Should update LookupRecord to allow multi criteria lookups.
NIFI-3948: PublishKafkaRecord_0_10 appears to no longer write messages on correct boundaries.
NIFI-3949: NullPointerException from GrokReader.
NIFI-3951: RecordPath throws ArrayIndexOutOfBoundsException if referencing element 0 of an empty array.
NIFI-3952: UpdateRecord should provide field.value, field.name, etc variables to Expression Language.
NIFI-3954: javax.net.SSL.PeerUnverifiedException Hostname www.googleapis.com does the match the certificate subject provided by the peer.
NIFI-3955: Confusing provenance searchable fields behavior.
NIFI-3958: Avro decimal logical type fail if database returns Zero precision.
NIFI-3960: ExtractGrok - Include exception message when compile fails.
NIFI-3961: HiveConnectionPool with yarn queue settings.
NIFI-3962: Update ConsumeKafkaRecord_0_10 to better batch together messages into fewer flowfiles.
NIFI-3963: Remote Process Group does not honor yield duration if transmission is disabled then reenabled.
NIFI-3969: MergeContent creates bins too early when a lot of data is queued up.
NIFI-3971: UpdateAttribute metric for output 'size' is larger than it should be when cloning.
NIFI-3972: Controller Service failing to enabled because the service it depends on is not fully enabled on NiFi restart.
NIFI-3976: ConsumePOP3 and ConsumeIMAP do NOT delete messages.
NIFI-3979: ListHDFS always skips files with latest timestamp.
NIFI-3981: When clustered, Ports sometimes are not running when NiFi restarts.
NIFI-3984: Support secure access to HWX Schema Registry.
NIFI-3985: SiteToSiteProvenanceReportingTask should support starting at end of Provenance Stream or Beginning of Stream.
NIFI-3986: Site-to-Site Provenance Reporting Task constantly emitting EOFException.
NIFI-3989: DataTypeUtils handles isLongTypeCompatible, isDoubleTypeCompatible, etc. poorly.
NIFI-3990: Reduce excessive garbage collection caused by record readers.
NIFI-3995: Update 'Schema Access Strategy' for Hwx Content Encoded Schema to use updated header info.
NIFI-4001: Bump ParCEFone version on ParseCEF.
NIFI-4002: Add PutElasticsearchHttpRecord processor.
NIFI-4003: Controller Service for HortonworksSchemaRegistry should expose Cache Duration as a property.
NIFI-4009: Support functions in RecordPath.
NIFI-4011: Bump Dockerfile version to 1.3.0

Kafka

This release provides Kafka 0.10.1.2 with no additional Apache patches.

HDP 2.6.0 provided Kafka 0.10.1.2 with no additional Apache patches.

Storm

This release provides Storm 1.1.0 and the following Apache patches:

STORM-1114: Handle race condition in Storm/Trident transactional state when ZK nodes have already been created/deleted.
STORM-2194: ReportErrorAndDie doesn't always die.
STORM-2429: non-string values in supervisor.scheduler.meta cause crash.
STORM-2431: the default blobstore.dir is storm.local.dir/blobs which is different from distcache-blobstore.md.
STORM-2432: Storm-Kafka-Client Trident Spout Seeks Incorrect Offset With UNCOMMITTED_LATEST Strategy.
STORM-2440: Kill process if executor catches `java.net.SocketTimeoutException`.
STORM-2450: Write resources into correct local directory.
STORM-2451: windows storm.cmd does not set log4j2 config file correctly by default.
STORM-2482: Refactor the Storm auto credential plugins to be more usable.
STORM-2482: Refactor the Storm auto credential plugins to be more usable.
STORM-2496: Dependency artifacts should be uploaded to blobstore with READ permission for all.
STORM-2498: Fix Download Full File link in 1.x branch.
STORM-2501: Auto populate Hive credentials using Hive MetaStore delegation tokens.
STORM-2511: Submitting a topology with name containing unicode getting failed..
STORM-2518: NPE during uploading dependency artifacts with secured cluster.
STORM-2520: AutoHDFS should prefer cluster-wise hdfs kerberos principal to global hdfs kerberos principal.
STORM-2528: Bump log4j version to 2.8.2.

HDP 2.6.0 provided Storm 1.1.0 with no additional Apache patches.

Ranger

This release provides Ranger 0.7.0 and the following Apache patches:

RANGER-1436: Disable, by default, deny policies with ranger.servicedef.enableDenyAndExceptionsInPolicies config parameter.
RANGER-1436: Turn Ranger Deny Policy & Except Conditions block to On by default .
RANGER-1475: reducing the highest time stamp value to pick all the users syncd during sync cycle.
RANGER-1490: Increase size of sort_order column of x_policy_resource_map.
RANGER-1531: Good coding practice while parsing XML documents in Ranger.
RANGER-1546: Code Improvement To Follow Best Practices.
RANGER-1548: Ranger needs better error messages when Ambari Infra is off.
RANGER-1550: HDFS test connection and resource lookup failing.
RANGER-1612: When servicedef is accessed, def_options property "enableDenyAndExceptionsInPolicies" is returned as "false" if there is no value set for it.

HDP 2.6.0 provided Ranger 0.7.0 and the following Apache patches:

RANGER-1378: Update MySQL Schema to fix issues related to only_full_group_by restriction of MySQL 5.7 version..
RANGER-1383: Use resource matchers for filtering service policies.
RANGER-1392: Hive test connection is failing even if jdbc.url configured is correct to 2.6-maint.
RANGER-1392: Revert "RANGER-1392: Hive test connection is failing even if jdbc.url configured is correct to 2.6-maint".
RANGER-1401: Add consolidated db schema script for SQLServer DB flavor.
RANGER-1405: groups are not shown if exact user name is passed in search filter.
RANGER-1406: Audit spoolfile not getting created when ranger service user didn't have permission to log into Solr.
RANGER-1407: Service update transaction log is not generated in some cases.
RANGER-1409: User role get deleted from table when he tries to update his role to a restricted role.
RANGER-1413: Fix issues uncovered by static code analysis.
RANGER-1413: Good coding practice in Ranger recommended by static code analysis.
RANGER-1417: Ranger Upgrade is failing for Oracle DB flavor.
RANGER-1422: Ranger Knox Plugin audit doesn't have the access type populated.
RANGER-1428: In certain scenario user data contains junk email-id.
RANGER-1434: Enable Group Search First causes issues when Enable Group Sync is disabled - 2.6-maint branch.
RANGER-1435: Allow different files to be specified for unix based usersync - 2.6-maint.
RANGER-1435: fixed minor issue of resource filenames from previous commit.
RANGER-1440: Improve install script to retry failing statements.
RANGER-1448: Change of import / export icons on Ranger UI.
RANGER-1453: Ranger KMS failed to start with Exception] : More than one Master Key exists.
RANGER-1459: Ranger update policy API is failing on Postgres / Oracle for case sensitive ACLs.
RANGER-1477: 'show databases' fails with access-denied when user doesn't have access to some of the databases.

ZooKeeper

This release provides ZooKeeper 3.4.6 and the following Apache patches:

ZOOKEEPER-2141: ACL cache in DataTree never removes entries.

HDP 2.6.0 provided ZooKeeper 3.4.6 and the following Apache patches:

ZOOKEEPER-2227: stmk four-letter word fails execution at server while reading trace mask argument.

Common Vulnerabilities and Exposures

The following CVEs have been fixed in HDF 3.0.0.

CVE-2017-7665

Summary: Apache NiFi XSS issue on certain user input components.

Severity: Important

Versions Fixed:

HDF 3.0.0 and HDF 2.1.4

Apache NiFi 1.3.0 and Apache NiFi 0.7.4

Versions Affected:

Apache NiFi 0.0.1 – 0.7.3

Apache NiFi 1.0.0 – 1.2.0

Description: There are certain user input components in the Apache NiFi UI which had been guarding for some forms of XSS issues but were insufficient.

Mitigation: The fix for more complete user input sanitization is applied on Apache NiFi 0.7.4 and Apache NiFi 1.3.0 releases. If you are running a prior 0.x or 1.x release, upgrade to the appropriate release.

Apache NiFi Bug ID: NIFI-3906

NiFi-3906 has been fixed in Apache NiFi 1.3, as well in the version of NiFi available in HDF 3.0.0. See NiFi Apache Patches for a full list of patches.

CVE-2017-7667

Summary: Apache NiFi XFS issue due to insufficient response headers.

Severity: Important

Versions Fixed:

HDF 3.0.0 and HDF 2.1.4

Apache NiFi 1.3.0 and Apache NiFi 0.7.4

Versions Affected:

Apache NiFi 0.0.1 – 0.7.3

Apache NiFi 1.0.0 – 1.2.0

Description: Apache NiFi needs to establish the response header telling browsers to only allow framing with the same origin.

Mitigation: The fix to set this response header is applied on Apache NiFi 0.7.4 and Apache NiFi 1.3.0 releases. If you are running a prior 0.x or 1.x release, upgrade to the appropriate release.

Apache NiFi Bug ID: NIFI-3907

NiFi-3907 has been fixed in Apache NiFi 1.3, as well in the version of NiFi available in HDF 3.0.0. See NiFi Apache Patches for a full list of patches.

CVE-2016-8746

Summary: Apache Ranger path matching issue in policy evaluation

Severity: Normal

Vendor: Hortonworks

Versions Affected: All HDF versions 2.0.x and 2.1.x using Apache Ranger versions 0.6.0/0.6.1/0.6.2.

Users affected: All users of the ranger policy admin tool.

Impact: Ranger policy engine incorrectly matches paths in certain conditions when a policy contains wildcards and recursive flags.

Fix detail: Fixed policy evaluation logic

Recommended Action: Users should upgrade to HDF 3.0.0 (with Apache Ranger 0.7.0+)

CVE-2016-8751

Summary: Apache Ranger stored cross site scripting issue

Severity: Normal

Vendor: Hortonworks

Versions Affected: HDF versions 2.0.x, 2.1.x using Apache Ranger versions 0.6.0/0.6.1/0.6.2.

Users affected: All users of the ranger policy admin tool.

Impact: Apache Ranger is vulnerable to a Stored Cross-Site Scripting when entering custom policy conditions. Admin users can store some arbitrary javascript code execute when normal users login and access policies.

Fix detail: Added logic to sanitize the user input.

Recommended Action: Users should upgrade to HDF 3.0.0 (with Apache Ranger 0.7.0+)

CVE-2017-7676

Summary: Apache Ranger policy evaluation ignores characters after ‘*’ wildcard character

Severity: Critical

Vendor: Hortonworks

Versions Affected: HDF versions 2.0.x, 2.1.x.

Users affected: Environments that use Ranger policies with characters after ‘*’ wildcard character – like my*test, test*.txt

Impact: Policy resource matcher ignores characters after ‘*’ wildcard character, which can result in unintended behavior.

Fix detail: Ranger policy resource matcher was updated to correctly handle wildcard matches.

Recommended Action: Upgrade to HDF 3.0.0.

CVE-2017-7677

Summary: Apache Ranger Hive Authorizer should check for RWX permission when external location is specified

Severity: Critical

Vendor: Hortonworks

Versions Affected: HDF versions 2.0.x, 2.1.x.

Users affected: Environments that use external location for hive tables

Impact: In environments that use external location for hive tables, Apache Ranger Hive Authorizer should check for RWX permission for the external location specified for create table.

Fix detail: Ranger Hive Authorizer was updated to correctly handle permission check with external location.

Recommended Action: Upgrade to HDF 3.0.0.

Known Issues

Hortonworks Bug ID

Apache JIRA

Component

Summary

BUG-82327

Streaming Analytics Manager

Issue: In HA installations of SAM, you may run into a situation where bootstrap has not been executed.

Result: When bootstrap has not been started, the UI displays the following message:

1. No component definitions found.
2. Please run ./bin/streamline bootstrap to 
initialize the component definitions.

Workaround for a secured cluster:

kinit -kt /etc/security/keytabs/streamline.service.keytab 
streamline-{cluster_name_in_lower-case}@domain

Workaround for an unsecured cluster:

cd /usr/hdf/current/streamline
./bootstrap/bootstrap.sh

BUG-82398

NiFi

Issue: For some VMs and operating systems, insufficient entropy may cause NiFi startup and runtime to be extremely slow.

Workaround: To work aeound this issue, update your bootstrap template.

From the Ambari UI, select NiFi, and click the Configs tab.
Scroll to Advanced nifi-bootstrap-env.
Add the following line the java arguement section of your bootstrap template:
```
java.arg.17=-Djava.security.egd=file:/dev/urandom
```

If java.arg.17is already in use, replace it with the next available number. For example:

java.arg.18=-Djava.security.egd=file:/dev/urandom

BUG-82681

AMBARI-21261

Ambari/ NiFi

Issue: When NiFi is installed on an Ambari-managed HDP cluster, NiFi information is not available in Ambari Metrics.

Workaround: There is no workaround for this issue.

BUG-82311

AMBARI-21263

Ambari/Solr/NiFi

Issue: The Ambari Server upgrade fails when the HDP Search (Solr) or HDF management pack is installed.

Error Message:

File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 896, in install_mpack
    (mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode)
  File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 792, in _install_mpack
    process_service_definitions_artifact(artifact, artifact_source_dir, options)
  File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 515, in process_service_definitions_artifact
    create_symlink(src_service_definitions_dir, dest_service_definitions_dir, file, options.force)
  File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 235, in create_symlink
    create_symlink_using_path(src_path, dest_link, force)
  File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 247, in create_symlink_using_path
    sudo.symlink(src_path, dest_link)
  File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 123, in symlink
    os.symlink(source, link_name)
OSError: [Errno 17] File exists

Workaround for HDP Search mpack: To work around this issue, enter the following commands:

rm -rf /var/lib/ambari-server/resources/common-services_05_06_17_19_49.old/SOLR
rm -rf /var/lib/ambari-server/resources/common-services/SOLR

Workaround for HDF mpack: To work around this issue, enter the following commands:

rm -rf /var/lib/ambari-server/resources/common-services/NIFI/ 
rm -rf /var/lib/ambari-server/resources/common-services_17_02_17_18_20.old/NIFI/ 
rm -rf /var/lib/ambari-server/resources/common-services_20_06_17_15_47.old/NIFI/

	Note
	In each case, the date-time pattern is a timestamp of the management pack file and differs in every case.

BUG-82561

AMBARI-21291

Ambari/HDF

Issue: You may experience a schema error when upgrading to Ambari 2.5.1.

Error Message:

WARN [main] DBAccessorImpl:814 - Error executing query: UPDATE request AS a SET cluster_host_info = b.cluster_host_info FROM stage AS b WHERE a.request_id = b.request_id, errorCode = 0, message = ERROR: column b.cluster_host_info does not exist 
Position: 45 
20 Jun 2017 19:45:23,466 ERROR [main] SchemaUpgradeHelper:209 - Upgrade failed. 
org.postgresql.util.PSQLException: ERROR: column b.cluster_host_info does not exist 
Position: 45

Workaround: To work around this issue, re-add the column to the stage table before issuing the 'ambari-server upgrade' command.

Example for updating the Ambari Postgres database:

psql -U ambari (default password is 'bigdata')
alter table stage add column cluster_host_info bytea not null default '{}';

Third-Party Licenses

HDF 3.0.0 deploys numerous third-party licenses and dependencies, all of which are compatible with the Apache software license. For complete third-party license information, see the licenses and notice files contained within the distribution.