New Features - Hortonworks Data Platform

Release Notes

Also available as:

PDF

New Features

This section highlights new features in HDP 2.5.0.

Table 1.1. New Features

Apache Component	Feature
Atlas	Core Capabilities: Object Versioning & Audit (ATLAS-493) Message Durability & Replay (ATLAS-912) Ambari Service Integration (AMBARI-14853) Embedded Solr for Search (ATLAS-490, AMBARI-15990) Embedded HBase (ATLAS-498) Soft Deletes (ATLAS-622) HDFS Path Model (ATLAS-751) Enterprise Readiness: Ranger Plugin for Authorization (ATLAS-495) Kerberos support (ATLAS-785) HA improvements (ATLAS-513) Ease of Use: Dynamic Tag Based Access Policy (RANGER-274) UI Authentication (ATLAS-494) Rolling upgrade support (AMBARI-15830) Expanded Sqoop Support (ATLAS-379, ATLAS-837) Expanded Hive Support (ATLAS-492) Storm / Kafka Support (ATLAS-183) Falcon Support (ATLAS-835)
Falcon	Core Features: Apache Spark Integration (FALCON-1763) Apache Atlas Integration (FALCON-1570) Provider integration with Microsoft Azure Data Factory pipelines (FALCON-1627) Security Features: Falcon REST API trusted proxy support (FALCON-1027) Certify HDFS encryption support with Falcon Manageability: Capture data transfer details for various available replications (FALCON-1479) Support for multiple lib paths in Falcon process (FALCON-887) Supply user-defined properties to Oozie workflows during schedule (FALCON-1573) Add ability to rerun process by ID (FALCON-1525) Improved Metrics related to pipelines and instances (FALCON-1766) Usability: Falcon UI to enable entity/process/feed edits and management (FALCON-790) Pluggable filesystem support (FALCON-1012)
Flume	Developer Productivity: Flume-Kafka Channel (FLUME-2500) TailDir (FLUME-2498)
HBase	Developer Productivity: HBase C+ Client Upgrade jamon-runtime to version 2.4.1 (HBASE-14493) Core Features: HBase Backup/Restore Based on HBaseSnapshot (HBASE-7912) MOB (Moderate Object Storage) Support (HBASE-11339) Date-Tiered Compaction Strategy (HBASE-15181) FIFO Compaction Strategy (HBASE-14468) Streamlined Operations: Revamped HBase Dashboard in Ambari (HBASE-12133, HBASE-13965, HBASE-14082, HBASE-14166, HBASE-14205, HBASE-14274, HBASE-14274, HBASE-14278, HBASE-14459, HBASE-14778, HBASE-14862, HBASE-14869, HBASE-14891, HBASE-14976, HBASE-14983, HBASE-15135, HBASE-15163, HBASE-15222, HBASE-15376, HBASE-15377, HBASE-15412, HBASE-15435, HBASE-15464, HBASE-15518, HBASE-15519, HBASE-15586, HBASE-15671)
Hive	EDW Offload: Dynamically partitioned hash join : Query Execution (HIVE-4924) Composable Stats from partition stats/ HLL (HIVE-13418) Vectorization: NPE for added columns in ORC non-partitioned tables (aka Schema Evolution) (HIVE-9985) Microstrategy Superapp: SQL Gaps: order by UDF in over clause (HIVE-13475) Cognos OLAP asks (HIVE-4662) General ORC schema evolution (HIVE-13178) Don't create empty bucket files (HIVE-13040) More scalable partition loads (HIVE-12988) Interactive SQL: Sub-5-Second: Compile simple query in less than 300 milliseconds (includes metastore) (HIVE-7926) Bucketed Table: prune buckets for filters (HIVE-10924) Sub-5-Second: Parallelize Query compilation in HS2 (HIVE-11525) Statement.setQueryTimeout() implemented in the Hive JDBC driver (HIVE-4239) Plan generation : Improve performance of PartitionPruner and PredicatePushDown transformation optimizations (HIVE-12878) Support rely/no-validate PK/FK relationship in Hive (HIVE-13076) Optimize metadata-only HAVING query (HIVE-12640, HIVE-12645) Enterprise Readiness: Hiveserver2 ThriftHttpServlet does not process X-Forwarded-For (HIVE-10673) Make Beeline usage easier (HIVE-11141) Log additional information in HiveServer2 (HIVE-11684) Support lazy initiation for TEZ session when start hive CLI (HIVE-11969) Extend Hive JDBC functionality to support Hive View use cases (HIVE-6535) Hive - support Row level filtering and column masking (HIVE-13125)
Kafka	Streamlined Operations: Kafka Rack Awareness (KIP-36) Enterprise Readiness: New Consumer API Improved Kafka SASL Support (KIP-43)
Knox	Core Security: Knox Support for Hadoop/Yarn/HBase/Hive High Availability (KNOX-567) Knox should support authentication using SPNEGO from browser (KNOX-25) Enterprise Readiness: Knox SSO for Ambari and Ranger UI (RANGER-685, AMBARI-15479, AMBARI-15778, AMBARI-16638) Knox support for SAML v2.0 based SSO system integration (KNOX-641)
Oozie	Core Features: Spark action (OOZIE-1983)
Phoenix	Developer Productivity: "OFFSET" feature of Phoenix for traversing the data in pages and retrieving only those many records from HBase. (PHOENIX-2722) Phoenix .NET Driver Phoenix ODBC Driver Phoenix Query Server (PHOENIX-971) and HardeningWork for General Availability (CALCITE-1128, CALCITE-1159, CALCITE-1180, CALCITE-1190, PHOENIX-2792) Support Parallel Order-By (PHOENIX-2126) Improve efficiency of the Phoenix bulkloader (PHOENIX-1973) Query Phoenix via Hive using PhoenixStorageHandler (PHOENIX-331, PHOENIX-2743) Core Features: Multi-tenancy Improvements: Namespace (PHOENIX-1311) Multi-tenancy Improvements: RegionServer Group based Assignment (HBASE-6721) Multi-tenancy Improvements: Local Index in Apache Phoenix goes into default HBase namespace (PHOENIX-1311) Stabilize Phoenix Local Secondary Indexes (PHOENIX-1734) Option to Make Secondary Indexes Always Readable (PHOENIX-2221)
Ranger	Advanced Security: Ranger policy model to support data masking (RANGER-873) Ranger policy model to support row-filtering (RANGER-908) Dynamic Tag Based Access Policy (RANGER-274) Time and Prohibition-based Access Policy (RANGER-606) Location-based Access Policy (RANGER-596) Enterprise Readiness: Ranger KMS – Safenet Luna HSM Integration (RANGER-868) Improvements on Reports page in Ranger Admin : Enhanced search by resource access, tags, etc. download report to a Excel, csv (RANGER-913) Remove support for DB based auditing (RANGER-900) Script and process to migrate existing audit from RDBMS to Solr (RANGER-271) Support multiple OU in LDAP search for Ranger usersync (RANGER-803) UserSync : limit the groups associated with users based on the group-search results (RANGER-869) Provide support to delete Users and Groups from Ranger Admin UI (RANGER-888) Add Ranger related rules to SmartSense
Spark	Performance: HBase connector (HBASE-14789) Streamlined Operations: YARN ATS v1.5 integration (SPARK-11373) Developer Productivity: SparkR (SPARK-5654) Advanced Security: Security Enhancements: Certification with HDFS TDE & wire encryption
Storm	Developer Productivity: Storm Windowing and State Management(STORM-1167) New Storm Connectors (STORM-1075, STORM-845, STORM-851) Kafka Spout using new Client APIs (STORM-822) Storm Distributed Log Search (STORM-902) Storm Dynamic Log Levels (STORM-412) Performance: Storm Performance Improvements (STORM-1151, STORM-1526, STORM-1537, STORM-1539) Streamlined Operations: Storm Management View via Ambari (AMBARI-16095) Storm Topology Event Inspector (STORM-954) Storm Dynamic Worker Profiling (STORM-1157) Storm Resource Aware Scheduling (STORM-894) Enterprise Readiness: Automatic Back Pressure (STORM-886) Distributed Cache (STORM-876) Improved Nimbus High Availability (STORM-166) Pacemaker Storm Daemon (STORM-885)
Zeppelin	Core Capability: Zeppelin general availability Advanced Security: Authentication – notes run as user (ZEPPELIN-548, ZEPPELIN-797, ZEPPELIN-769) Authorization of Notebook (ZEPPELIN-549, ZEPPELIN-838, ZEPPELIN-946) Kerberos Support (ZEPPELIN-446) Ambari managed LDAP configuration (AMBARI-15758) Zeppelin support for SSL (one way) Streamlined Operations: Multi User support (ZEPPELIN-773) Ambari managed install (AMBARI-15265, AMBARI-16779) Manageability: Zeppelin Ambari View (AMBARI-16957)