Chapter 1. Hortonworks DataFlow 3.0.1 Release Notes
This document provides you with the latest information about the HDF 3.0.1 release and its product documentation.
Component Support
HDF 3.0.1 includes the following components:
Apache Ambari 2.5.1
Apache Kafka 0.10.2.1
Apache NiFi 1.2.0
Apache Ranger 0.7.0
Apache Storm 1.1.0
Apache ZooKeeper 3.4.6
Apache MiNiFi Java Agent 0.2.0
Apache MiNiFi C++ Technical Preview
Hortonworks Schema Registry 0.3.0
Hortonworks Streaming Analytics Manager 0.5.0
Component Availability in HDF
Previous HDF releases shipped with the following components versions.
| NiFi | Storm | Kafka | ZooKeeper | Ambari | Ranger | MiNiFi Java Agent | MiNiFi C++ | Streaming Analytics Manager | Schema Registry | |
|---|---|---|---|---|---|---|---|---|---|---|
| HDF 3.0.1 | 1.2.0 | 1.1.0 | 0.10.2.1 | 3.4.6 | 2.5.1 | 0.7.0 | 0.2.0 | TP | 0.5.0 | 0.3.0 |
| HDF 3.0.0 | 1.2.0 | 1.1.0 | 0.10.2.1 | 3.4.6 | 2.5.1 | 0.7.0 | 0.2.0 | TP | 0.5.0 | 0.3.0 |
| HDF 2.1.4 | 1.1.0 | 1.0.2 | 1.10.1 | 3.4.6 | 2.4.2.0 | 0.6.2 | 0.1.0 | TP | N/A | N/A |
| HDF 2.1.2 | 1.1.0 | 1.0.2 | 0.10.1 | 3.4.6 | 2.4.2.0 | 0.6.2 | 0.1.0 | TP | N/A | N/A |
| HDF 2.1.1 | 1.1.0 | 1.0.2 | 0.10.1 | 3.4.6 | 2.4.2.0 | 0.6.2 | 0.1.0 | TP | N/A | N/A |
| HDF 2.1.0 | 1.1.0 | 1.0.2 | 0.10.1 | 3.4.6 | 2.4.2.0 | 0.6.2 | 0.1.0 | TP | N/A | N/A |
| HDF 2.0.2 | 1.0.0 | 1.0.1 | 0.10.0.1 | 3.4.6 | 2.4.1.0 | 0.6.0 | 0.0.1 | TP | N/A | N/A |
| HDF 2.0.1 | 1.0.0 | 1.0.1 | 0.10.0.1 | 3.4.6 | 2.4.1.0 | 0.6.0 | 0.0.1 | TP | N/A | N/A |
| HDF 2.0.0 | 1.0.0 | 1.0.1 | 0.10.0.1 | 3.4.6 | 2.4.0.1 | 0.6.0 | 0.0.1 | TP | N/A | N/A |
| HDF 1.2.1 | 0.6.1 | 0.10.0 | 0.9.0.1 | 3.4.6 | N/A | N/A | N/A | N/A | N/A | N/A |
| HDF 1.2.0.1 | 0.6.1 | 0.10.0 | 0.9.0.1 | 3.4.6 | N/A | N/A | N/A | N/A | N/A | N/A |
| HDF 1.2.0 | 0.6.0 | 0.10.0 | 0.9.0.1 | 3.4.6 | N/A | N/A | N/A | N/A | N/A | N/A |
| HDF 1.1.0 | 0.4.0 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
| HDF 1.0 | 0.3.0 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
What's New in HDF 3.0.1
HDF 3.0.1 is a maintenance release that includes the following improvements and bug fixes:
NiFi bug fixes and improvements
BUG-78728 – POST lineage query does not contain eventID in the request.
BUG-83921 – Occasional deadlock when trying to clean up old Content Claims.
BUG-83845 – Processors failing to run with validation errors when fields are populated.
BUG-83704 – Make configurable maximum number of concurrent requests.
BUG-83689 – Slow response times when requesting Process Group Status.
BUG-83542 – NiFi reports too many pending requests.
BUG-83538 – GenerateTableFetch - Add a warning when not correctly setting the connections.
BUG-83516 – ConsumeKafkaRecord can throw NPE if using an Avro Reader and data on topic is missing a required field.
BUG-83515 – Queue with flow files thinks it is empty.
BUG-83514 – Average Lineage Duration stat is sometimes incorrect for cluster aggregate.
BUG-83511 – ConsumeEWS May Generate Unusable Emails.
BUG-83510 – Error in SiteToSiteProvenanceReportingTask event type description.
BUG-83509 – StandardFlowSynchronizerSpec relies on other test.
BUG-83508 – Avro Schema with default values can cause NPE.
BUG-83507 – ListSFTP fails to retrieve user home directory for certain user accounts.
BUG-83506 – ListFTP/ListSFTP does not provide file.size attribute.
BUG-83505 – QueryRecord hits a run time exception when using double quotes.
BUG-83504 – NPE at AbstractConfiguredComponent.validate if controller service bundle is not found.
BUG-83503 – Funnel with no outgoing connections and a queued Flow File uses high CPU.
BUG-83502 – Inconsistent State for RPGs.
BUG-83501 – GrokRecordReader returns empty record for lines that do not match even when "No Match Behavior" set to "Skip Line".
BUG-83500 – IPLookupService returns a Record with all null fields if the given IP cannot be found.
BUG-83499 – Connection config window allows you to drag prioritizers in weird ways
BUG-83498 – GetSQS Does Not Request Message Attributes.
BUG-83322 – Add UTF8 support for GetFTP
BUG-82984 – ZooKeeper snapshots are not been auotpurged, despite having autopurge options configured.
BUG-82398 – HDF/NiFi Ambari deployment needs to align with bootstrap.conf/nifi.properties.
BUG-82138 – Enable expression language on Fetch/List/Put SFTP and List/Fetch/Put/Delete HDFS processor properties.
Streaming Analytics bug fixes and improvements
BUG-81163 – HBase Client retries must be configurable.
BUG-81109 – Add Select All to most to processor where applicable.
BUG-84191 – Fix issues observed during Hive Sink testing.
BUG-84164 – The response code for invalid ambari username password while adding service pool is 200.
BUG-83427 – Avro schema with nullable fields throws runtime exception in SAM application.
BUG-83426 – Reader and writer SR schema versions for sources and sinks.
BUG-83180 – Add proxy settings for SAM in Mpack.
BUG-83179 – Enable Hive Sink with token based auth in SAM tests.
BUG-83176 – UDF Table text needs to be cleaned up.
BUG-83174 – Make SSL trust store password config field in kafka source and sink mandatory for SSL.
BUG-83173 – SAM UI Improvements.
BUG-82882 – Add service text is too close to the other service name.
BUG-82877 – Testing email notifier enabled.
BUG-82876 – Errors in sam logs.
BUG-82400 – Connecting Edge did not go away when upstream component was removed.
BUG-82337 – Streamline occasionally runs into problem with bootstrap.
BUG-82112 – Putting a special character like question mark in any processor name causes exception in App Deployment.
BUG-82082 – Developer Role should not see edit actions on Service Pool entities.
BUG-81282 – Delete should not go through if the topology is running.
Storm bug fixes and improvements
BUG-83178 – Storm token based auth needs to use the Users identity rather than impersonation user identity.
For complete information about the HDF 3.0.0 release, see the HDF 3.0.0 Release Notes.
Unsupported Features
Some features exist within HDF 3.0.1, but Hortonworks does not currently support these capabilities.
Technical Preview Features
The following features are available within HDF 3.0.1 but are not ready for production deployment. Hortonworks encourages you to explore these technical preview features in non-production environments and provide feedback on your experiences through the Hortonworks Community Forums.
Table 1.1. Technical Previews
| Component | Feature |
|---|---|
|
MiNiFi | MiNiFi C++ |
| Streaming Analytics Manager |
|
Community Driven Features
The following features are developed and tested by the Hortonworks community but are not officially supported by Hortonworks. These features are excluded for a variety of reasons, including insufficient reliability or incomplete test case coverage, declaration of non-production readiness by the community at large, and feature deviation from Hortonworks best practices. Do not use these features in your production environments.
Community Driven Kafka features
Kafka Connect
Kafka Streams
Community Driven NiFi Tools and Services
Embedded ZooKeeper
Sensitive key migration toolkit
Docker image for Apache NiFi
Community Driven NiFi Processors
AttributeRollingWindow
AWSCredentialsProviderControllerService
CompareFuzzyHash
ConsumeEWS
ConsumeIMAP
ConsumePOP3
ConvertExcelToCSVProcessor
DebugFlow
DeleteDynamoDB
DeleteGCSObject
DeleteHDFS
ExtractCCDAAttributes
ExtractEmailAttachments
ExtractEmailHeaders
ExtractMediaMetadata
ExtractTNEFAttachments
FetchAzureBlobStorage
FetchGCSObject
FuzzyHashContent
GetDynamoDB
GetHDFSEvents
GetSNMP
ISPEnrichIP
InferAvroSchema
ListenBeats
ListenLumberjack
ListenSMTP
ListAzureBlobStorage
ListGCSBucket
ListS3
ModifyBytes
OrcFormatConversion
PutAzureBlobStorage
PutDynamoDB
PutGCSObject
PutIgniteCache
PutKinesisFirehose
PutKinesisStream
PutLambda
PutSlack
PutTCP
PutUDP
QueryDNS
SetSNMP
SpringContextProcessor
StoreInKiteDataset
Community Driven NiFi Controller Services
AWSCredentialsProviderControllerService
GCPCredentialsControllerService
Community Driven NiFi Reporting Tasks
SiteToSiteBulletinReportingTask
SiteToSiteStatusReportingTask
DataDogReportingTask
Unsupported Customizations
Hortonworks cannot guarantee that default NiFi processors are compatible with proprietary protocol implementations or proprietary interface extensions. For example, we support interfaces like JMS and JDBC that are built around standards, specifications, or open protocols. But we do not support customizations of those interfaces, or proprietary extensions built on top of those interfaces.
Deprecated Technologies
This section points out any technology from previous releases that has been deprecated or removed from this release (operating systems, Java versions, databases, product features). Use this section as a guide for your implementation plans.
- Deprecated
Technology that Hortonworks is removing in a future release. Deprecated items are supported until they are removed; deprecation gives you time to plan for removal.
- Removed
Technology that Hortonworks has removed from production and is no longer supported.
Table 1.2. Deprecated Operating Systems
|
Operating System | Release Deprecated | Release Removed |
|---|---|---|
| Debian 6 | HDF 2.1.2 | HDF 3.0.0 |
| Ubuntu 12 | HDF 3.0.0 | HDF 3.0.0 |
Table 1.3. Deprecated NiFi Processors
|
Processor | Release Deprecated |
|---|---|
| ConvertCSVToAvro | HDF 3.0.0 |
| ConvertJSONToAvro | HDF 3.0.0 |
| GetKafka | HDF 2.0.0 |
| PutKafka | HDF 2.0.0 |
| EvaluateRegularExpression | HDF 1.0.0 |
HDF Repository Locations
Use the following table to identify the HDF 3.0.0 repository location for your operating system and operational objectives. HDF 3.0.0 supports the following operating systems:
Table 1.4. RHEL/Oracle Linux/CentOS 6 HDF repository & additional download locations
| OS | Format | Download location |
|---|---|---|
| RHEL/Oracle Linux/CentOS 6 (64-bit): | HDF Base URL | http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.0 |
| HDF Repo | http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.0/hdf.repo | |
| RPM tarball | http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.0/HDF-3.0.1.0-centos6-rpm.tar.gz | |
| Tars tarball | http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.0/HDF-3.0.1.0-centos6-tars-tarball.tar.gz | |
| HDF Management Pack | http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.0-43.tar.gz | |
| HDP and Ambari Repositories | ||
| Ambari | http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.1.0/ambari.repo | |
| HDP | http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.1.0/hdp.repo | |
| OS Agnostic Downloads | ||
| NiFi only | ||
| NiFi Toolkit | ||
| MiNiFi Java Agent | ||
| MiNiFi Toolkit | ||
Table 1.5. RHEL/Oracle Linux/CentOS 7 HDF repository & additional download locations
Table 1.6. SLES 11 SP3/SP4 HDF repository & additional download locations
| OS | Format | Download location |
|---|---|---|
| SUSE Enterprise Linux 11 SP3, SP4 | HDF Base URL | http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.1.0 |
| Repo | http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.1.0/hdf.repo | |
| RPM tarball | http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.1.0/HDF-3.0.1.0-suse11sp3-rpm.tar.gz | |
| Tars tarball | http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.1.0/HDF-3.0.1.0-suse11sp3-tars-tarball.tar.gz | |
| HDF Management Pack | http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.1.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.0-43.tar.gz | |
| HDP and Ambari Repositories | ||
| Ambari | http://public-repo-1.hortonworks.com/ambari/suse11/2.x/updates/2.5.1.0/ambari.repo | |
| HDP | http://public-repo-1.hortonworks.com/HDP/suse11sp3/2.x/updates/2.6.1.0/hdp.repo | |
| OS Agnostic Downloads | ||
| NiFi only | ||
| NiFi Toolkit | ||
| MiNiFi Java Agent | ||
| MiNiFi Toolkit | ||
Table 1.7. SLES 12 HDF repository & additional download locations
| OS | Format | Download location |
|---|---|---|
| SUSE Linux Enterprise Server (SLES) v12 SP1 | HDF Base URL | http://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.1.0 |
| Repo | http://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.1.0/hdf.repo | |
| RPM tarball | http://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.1.0/HDF-3.0.1.0-sles12-rpm.tar.gz | |
| Tars tarball | http://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.1.0/HDF-3.0.1.0-sles12-tars-tarball.tar.gz | |
| HDF Management Pack | http://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.1.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.0-43.tar.gz | |
| HDP and Ambari Repositories | ||
| Ambari | http://public-repo-1.hortonworks.com/ambari/sles12/2.x/updates/2.5.1.0/ambari.repo | |
| HDP | http://public-repo-1.hortonworks.com/HDP/sles12/2.x/updates/2.6.1.0/hdp.repo | |
| OS Agnostic Downloads | ||
| NiFi only | ||
| NiFi Toolkit | ||
| MiNiFi Java Agent | ||
| MiNiFi Toolkit | ||
Table 1.8. Ubuntu 14 HDF repository & additional download locations
Table 1.9. Debian 7 HDF repository & additional download locations
| OS | Format | Download location |
|---|---|---|
| Debian 7 | HDF Base URL | http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.1.0 |
| Repo | http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.1.0/hdf.list | |
| Deb tarball | http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.1.0/HDF-3.0.1.0-debian7-deb.tar.gz | |
| Tars tarball | http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.1.0/HDF-3.0.1.0-debian7-tars-tarball.tar.gz | |
| HDF Management Pack | http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.1.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.0-43.tar.gz | |
| HDP and Ambari Respositories | ||
| Ambari | http://public-repo-1.hortonworks.com/ambari/debian7/2.x/updates/2.5.1.0/ambari.list | |
| HDP | http://public-repo-1.hortonworks.com/HDP/debian7/2.x/updates/2.6.1.0/hdp.list | |
| OS Agnostic Downloads | ||
| NiFi only | ||
| NiFi Toolkit | ||
| MiNiFi Java Agent | ||
| MiNiFi Toolkit | ||
Common Vulnerabilities and Exposures
The following CVEs have been fixed in HDF 3.0.1.
CVE-2016-8746
| Summary: Apache Ranger path matching issue in policy evaluation |
| Severity: Normal |
| Vendor: Hortonworks |
| Versions Affected: All HDF versions 2.0.x and 2.1.x using Apache Ranger versions 0.6.0/0.6.1/0.6.2. |
| Users affected: All users of the ranger policy admin tool. |
| Impact: Ranger policy engine incorrectly matches paths in certain conditions when a policy contains wildcards and recursive flags. |
| Fix detail: Fixed policy evaluation logic |
| Recommended Action: Users should upgrade to HDF 3.0.0 (with Apache Ranger 0.7.0+) |
CVE-2016-8751
| Summary: Apache Ranger stored cross site scripting issue |
| Severity: Normal |
| Vendor: Hortonworks |
| Versions Affected: HDF versions 2.0.x, 2.1.x using Apache Ranger versions 0.6.0/0.6.1/0.6.2. |
| Users affected: All users of the ranger policy admin tool. |
| Impact: Apache Ranger is vulnerable to a Stored Cross-Site Scripting when entering custom policy conditions. Admin users can store some arbitrary javascript code execute when normal users login and access policies. |
| Fix detail: Added logic to sanitize the user input. |
| Recommended Action: Users should upgrade to HDF 3.0.0 (with Apache Ranger 0.7.0+) |
CVE-2017-7676
| Summary: Apache Ranger policy evaluation ignores characters after ‘*’ wildcard character |
| Severity: Critical |
| Vendor: Hortonworks |
| Versions Affected: HDF versions 2.0.x, 2.1.x. |
| Users affected: Environments that use Ranger policies with characters after ‘*’ wildcard character – like my*test, test*.txt |
| Impact: Policy resource matcher ignores characters after ‘*’ wildcard character, which can result in unintended behavior. |
| Fix detail: Ranger policy resource matcher was updated to correctly handle wildcard matches. |
| Recommended Action: Upgrade to HDF 3.0.0. |
CVE-2017-7677
| Summary: Apache Ranger Hive Authorizer should check for RWX permission when external location is specified |
| Severity: Critical |
| Vendor: Hortonworks |
| Versions Affected: HDF versions 2.0.x, 2.1.x. |
| Users affected: Environments that use external location for hive tables |
| Impact: In environments that use external location for hive tables, Apache Ranger Hive Authorizer should check for RWX permission for the external location specified for create table. |
| Fix detail: Ranger Hive Authorizer was updated to correctly handle permission check with external location. |
| Recommended Action: Upgrade to HDF 3.0.0. |
Known Issues
|
Hortonworks Bug ID |
Apache JIRA |
Component |
Summary | |||
|---|---|---|---|---|---|---|
| BUG-82327 | Streaming Analytics Manager |
Issue: In HA installations of SAM, you may run into a situation where bootstrap has not been executed. Result: When bootstrap has not been started, the UI displays the following message: 1. No component definitions found. 2. Please run ./bin/streamline bootstrap to initialize the component definitions. Workaround for a secured cluster: kinit -kt /etc/security/keytabs/streamline.service.keytab
streamline-{cluster_name_in_lower-case}@domain
Workaround for an unsecured cluster: cd /usr/hdf/current/streamline ./bootstrap/bootstrap.sh | ||||
| BUG-82681 | AMBARI-21261 | Ambari/ NiFi |
Issue: When NiFi is installed on an Ambari-managed HDP cluster, NiFi information is not available in Ambari Metrics. Workaround: There is no workaround for this issue. | |||
| BUG-82311 | AMBARI-21263 | Ambari/Solr/NiFi |
Issue: The Ambari Server upgrade fails when the HDP Search (Solr) or HDF management pack is installed. Error Message: File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 896, in install_mpack
(mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 792, in _install_mpack
process_service_definitions_artifact(artifact, artifact_source_dir, options)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 515, in process_service_definitions_artifact
create_symlink(src_service_definitions_dir, dest_service_definitions_dir, file, options.force)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 235, in create_symlink
create_symlink_using_path(src_path, dest_link, force)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 247, in create_symlink_using_path
sudo.symlink(src_path, dest_link)
File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 123, in symlink
os.symlink(source, link_name)
OSError: [Errno 17] File exists
Workaround for HDP Search mpack: To work around this issue, enter the following commands: rm -rf /var/lib/ambari-server/resources/common-services_05_06_17_19_49.old/SOLR rm -rf /var/lib/ambari-server/resources/common-services/SOLR Workaround for HDF mpack: To work around this issue, enter the following commands: rm -rf /var/lib/ambari-server/resources/common-services/NIFI/ rm -rf /var/lib/ambari-server/resources/common-services_17_02_17_18_20.old/NIFI/ rm -rf /var/lib/ambari-server/resources/common-services_20_06_17_15_47.old/NIFI/
| |||
| BUG-82561 | AMBARI-21291 | Ambari/HDF |
Issue: You may experience a schema error when upgrading to Ambari 2.5.1. Error Message: WARN [main] DBAccessorImpl:814 - Error executing query: UPDATE request AS a SET cluster_host_info = b.cluster_host_info FROM stage AS b WHERE a.request_id = b.request_id, errorCode = 0, message = ERROR: column b.cluster_host_info does not exist Position: 45 20 Jun 2017 19:45:23,466 ERROR [main] SchemaUpgradeHelper:209 - Upgrade failed. org.postgresql.util.PSQLException: ERROR: column b.cluster_host_info does not exist Position: 45 Workaround: To work around this issue, re-add the column to the stage table before issuing the 'ambari-server upgrade' command. Example for updating the Ambari Postgres database:
|
![[Note]](../common/images/admon/note.png)
Third-Party Licenses
HDF 3.0.1 deploys numerous third-party licenses and dependencies, all of which are compatible with the Apache software license. For complete third-party license information, see the licenses and notice files contained within the distribution.