What's new in this release: Apache Hive
HDP 3.0 includes Apache Hive 3 enhancements that can help you improve query performance and comply with regulations.
The following list briefly describes a few key enhancements and covers unsupported interfaces.
Workload management
Managing resources is critical to Hive LLAP (low-latency analytical processing), especially in a multitenant environment. Using workload management, you can create resource pools and allocate resources to match needs and prevent contention for those resources. Workload management improves parallel query execution and cluster sharing for queries running on Hive LLAP, and also improves performance of non-LLAP queries. You implement workload management on the command line using Hive.
Transaction processing improvements
Mature versions of ACID (Atomicity, Consistency, Isolation, and Durability) transaction processing and low latency analytical processing (LLAP) evolve in Hive and HDP 3.0. ACID tables are enhanced to serve as the default table type in HDP 3.0, without performance or operational overload. Using ACID table operations facilitates compliance with the right to be forgotten requirement of the GDPR (General Data Protection Regulation). Application development and operations are simplified with stronger transactional guarantees and simpler semantics for SQL commands. You do not need to bucket ACID tables, so maintenance is easier. You no longer need to perform ACID delete operations in a Hive table.
Materialized views
With improvements in transactional semantics comes advanced optimizations, such as materialized view rewrites and automatic query cache. With these optimizations, you can deploy new Hive application types. Because multiple queries frequently need the same intermediate roll up or joined table, you can avoid costly, repetitious query portion sharing, by precomputing and caching intermediate tables into views. The query optimizer automatically leverages the precomputed cache, improving performance. Materialized views increase the speed of join and aggregation queries in business intelligence (BI) and dashboard applications, for example.
Cost-based optimizer enhancements
Hive can push down the filtering, sorting, and joining of columns in a query. For example, MySQL tables joins can be pushed down to underlying database.
Direct, low latency Hive query of Kafka topics
You can ingest Kafka into ACID tables, or query the data in the Kafka message from Hive. With HDP 3.0, you can create a Druid table within Hive from a Kafka topic in a single command. This feature simplifies queries of Kafka data by eliminating the data processing step between delivery by Kafka and querying in Druid.
Superset
HDP 3 introduces a technical preview of Apache Superset, the data exploration and visualization UI platform. Superset is a way to create HDP dashboards. Using Superset, installed by default as a service in Ambari, you can connect to Hive, create visualizations of Hive data, and create custom dashboards on Hive datasets. Superset is an alternative to Hive View, which is not available in HDP 3.0.
Spark integration with Hive
You can use Hive 3 to query data from Apache Spark and Apache Kafka applications, without workarounds. The Hive Warehouse Connector supports reading and writing Hive tables from Spark.
Hive security improvements
Apache Ranger secures Hive data by default. To meet customer demands for concurrency improvements, ACID support for GDPR (General Data Protection Regulation), render security, and other features, Hive now tightly controls the file system and computer memory resources. With the additional control, Hive better optimizes workloads in shared files and YARN containers. The more Hive controls the file system, the better Hive can secure data.
Deprecated, unavailable, and unsupported interfaces
In HDP 3.0 and later, Hive does not support the following features:
- Apache Hadoop Distributed Copy (DistCp)
- WebHCat
- Hcat CLI
- Hive CLI (replaced by Beeline)
- SQL Standard Authorization
- MapReduce execution engine (replaced by Tez)