OpenTelemetry support for Hive
Overview
OpenTelemetry (OTel) provides an open-source solution for collecting, processing, and exporting telemetry data, including metrics from applications. OTel helps users gain visibility into query performance and troubleshoot query failures.
As part of this offering, Hive in Cloudera Data Warehouse includes an OTel exporter that helps to collect, filter, and publish telemetry information, such as infrastructure and workload metrics, live and historical query data.
HiveServer2 and LLAP integration with OTel
OTel threads function independently in both HiveServer2 and LLAP daemons. When query execution begins, these threads capture essential telemetry data, which is transmitted to an OTel collector based on a configurable recurring schedule. The collected data is then processed and forwarded to backend systems for visualization.
HiveServer2 integrates with the OTel agent to expose both query-related data and JVM metrics. A dedicated thread or service runs within HiveServer2 to handle this integration. This thread collects query details and metrics, which are then transmitted through the OTel agent. These transmitted metrics can be collected by OTel collectors for analysis and visualization.
Metrics specific to each LLAP daemon, such as JVM and memory-related statistics, are also transmitted for detailed observability.
Availability
OTel support for Hive is made effective as of the Cloudera Data Warehouse on cloud1.10.1 version. After upgrading the Cloudera Data Warehouse version, you must also upgrade existing Hive Virtual Warehouses to 2025.0.19.0 or higher versions to enable and configure the OTel integration
