Cloudera Docs

Debug Web UI for Impala Daemon

The Impala Daemon (impalad) Web UI includes information about configuration settings, running and completed queries, and associated performance and resource usage for queries.

To debug and troubleshoot an impalad using a web-based interface, open the URL http://impala‑server‑hostname:25000/ in a browser. (For secure clusters, use the prefix https:// instead of http://.)

Because each Impala node produces its own set of debug information, you should choose a specific node that you want to investigate an issue on.

Main Page

The main impalad Web UI page at / lists the following information about the impalad:

  • The version of the impalad daemon

    The Version section also contains other information, such as when Impala was built and what build flags were used.

  • Process start time
  • Hardware information
  • OS information
  • Process information
  • CGroup information
Admission Controller Page

The Admission Controller impalad debug Web UI is at /admission page under the main impalad Web UI.

Use the /admission page to troubleshoot queued queries and the admission control.

The admission page provides the following information about each resource pool to which queries have been submitted at least once:

  • Time since the statestored received the last update
  • A warning if this impalad is considered disconnected from the statestored and thus the information on this page could be stale.
  • Pool configuration
  • Queued queries submitted to this coordinator, in the order of submission
  • Running queries submitted to this coordinator
  • Pool stats
    • Average of time in queue: An exponential moving average which represents the average time in queue over the last 10 to 12 queries. If a query is admitted immediately, the wait time of 0 is used in calculating this average wait time.
  • Histogram of the distribution of peak memory used by queries admitted to the pool

    Use the histogram to figure out settings for the minimum and maximum query MEM_LIMIT ranges for this pool.

    The histogram displays data for all queries admitted to the pool, including the queries that finished, got canceled, or hit an error.

Click on the pool name to only display information relevant to that pool. You can then refresh the debug page to see only the information for that specific pool.

Click Reset informational stats for all pools to reset the stats that keep track of historical data, such as Totals stats, Time in queue (exponential moving average), and the histogram.

The above information is also available as a JSON object from the following HTTP endpoint: 
http://impala-server-hostname:port/admission?json
Known Backends Page

The Known backends page of the impalad debug Web UI is at /backends under the main impalad Web UI.

This page lists the following info for each of the impalad nodes in the cluster. Because each impalad daemon knows about every other impalad daemon through the StateStore, this information should be the same regardless of which node you select.

  • Address of the node: Host name and port
  • KRPC address: The KRPC address of the impalad. Use this address when you issue the SHUTDOWN command for this impalad.
  • Whether acting as a coordinator
  • Whether acting as an executor
  • Quiescing status: Specify whether the graceful shutdown process has been initiated on this impalad.
  • Memory limit for admission: The amount of memory that can be admitted to this backend by the admission controller.
  • Memory reserved: The amount of memory reserved by queries that are active, either currently executing or finished but not yet closed, on this backend.

    The memory reserved for a query that is currently executing is its memory limit, if set. Otherwise, if the query has no limit or if the query finished executing, the current consumption is used.

  • Memory of the queries admitted to this coordinator: The memory submitted to this particular host by the queries admitted by this coordinator.
Catalog Page

The Catalog page of the impalad debug Web UI is at /catalog under the main impalad Web UI.

This page displays a list of databases and associated tables recognized by this instance of impalad. You can use this page to locate which database a table is in, check the exact spelling of a database or table name, look for identical table names in multiple databases. The primary debugging use case would be to check if an impalad instance has knowledge of a particular table that someone expects to be in a particular database.

Hadoop Configuration

The Hadoop Configuration page of the impalad debug Web UI is at /hadoop-varz under the main impalad Web UI.

This page displays the Hadoop common configurations that Impala is running with.

JMX

The JMX page of the impalad debug Web UI is at /jmx under the main impalad Web UI.

This page displays monitoring information about various JVM subsystems, such as memory pools, thread management, runtime. etc.

Java Log Level

The Change log level page of the impalad debug Web UI is at /log_level under the main impalad Web UI.

This page displays the current Java and backend log levels, and it allows you to change the log levels dynamically without having to restart the impalad.

Logs Page

The INFO logs page of the impalad debug Web UI is at /logs under the main impalad Web UI.

This page shows the last portion of the impalad.INFO log file, including the info, warning, and error logs for the impalad. You can see the details of the most recent operations, whether the operations succeeded or encountered errors. This page provides one central place for the log files and saves you from looking around the filesystem for the log files, which could be in different locations on clusters that use cluster management software.

Memz Page

The Memory Usage page of the impalad debug Web UI is at /memz under the main impalad Web UI.

This page displays the summary and detailed information about memory usage by the impalad.

Metrics Page

The Metrics page of the impalad debug Web UI is at /metrics under the main impalad Web UI.

This page displays the current set of metrics, counters and flags representing various aspects of impalad internal operations.

Queries Page

The Queries page of the impalad debug Web UI is at /queries under the main impalad Web UI.

This page lists:

  • Currently running queries
  • Queries that have completed their execution, but have not been closed yet
  • Completed queries whose details still reside in memory

The queries are listed in reverse chronological order, with the most recent at the top. You can control the amount of memory devoted to completed queries by specifying the -‑-‑query_log_size startup option for impalad.

This page provides:
  • How many SQL statements are failing (State value of EXCEPTION)
  • How large the result sets are (# rows fetched)
  • How long each statement took (Start Time and End Time)

Click the Details link for a query to display the detailed performance characteristics of that query, such as the profile output.

On the query detail page, in the Profile tab, you have options to export the query profile output to the Thrift, text, or Json format.

The Queries page also includes the Query Locations section that lists the number of running queries with fragments on this host.

RPC Services Page

The RPC durations page of the impalad debug Web UI is at /rpcz under the main impalad Web UI.

This page displays information, such as the duration, about the RPC communications of this impalad with other Impala daemons.

Sessions Page

The Sessions page of the impalad debug Web UI is at /session under the main impalad Web UI.

This page displays information about the sessions currently connected to this impalad instance. For example, sessions could include connections from the impala-shell command, JDBC or ODBC applications, or the Impala Query UI in the Hue web interface.

Threadz Page

The Threads page of the impalad debug Web UI is at /threadz under the main impalad Web UI.

This page displays information about the threads used by this instance of impalad, and it shows which categories they are grouped into. Making use of this information requires substantial knowledge about Impala internals.

Varz Page
The Varz page of the impalad debug Web UI is at /varz under the main impalad Web UI.

This page shows the configuration settings in effect when this instance of impalad communicates with other Hadoop components such as HDFS and YARN. These settings are collected from a set of configuration files.

The bottom of this page also lists all the command-line settings in effect for this instance of impalad.

Prometheus Metrics Page

At /metrics_prometheus, under the main impalad Web UI, the metrics are generated in Prometheus exposition format that Prometheus can consume for event monitoring and alerting.

The /metrics_prometheus is not shown in the Web UI list of pages.