Metrics and queries

You can use metrics and queries to create a custom Cluster Utilization Report in Cloudera Manager.

Many of the metrics described below use a data granularity of hourly. This is not required, but is recommended because some of the YARN utilization metrics are only available hourly and using the hourly granularity allows for consistent reporting.

Cluster-Level CPU and Memory Metrics

Total cluster CPU usage

Data Granularity: hourly

Units: percentage

tsquery:

SELECT
   cpu_percent_across_hosts
WHERE
   category=CLUSTER
   AND clusterName=Cluster_Name

Total CPU Cores in the cluster

Data Granularity: hourly

Units: CPU cores

tsquery:

SELECT
   total_cores_across_hosts
WHERE
   category=CLUSTER
   AND clusterName=Cluster_Name

Total cluster memory usage

Data Granularity: hourly

Units: percentage

tsquery:

SELECT
   100 * total_physical_memory_used_across_hosts/total_physical_memory_total_across_hosts
WHERE
   category=CLUSTER
   AND clusterName=Cluster_Name

Total cluster memory usage

Time series of total cluster memory usage.

Data Granularity:hourly

Units: Byte seconds

tsquery:

SELECT
   total_physical_memory_total_across_hosts
WHERE
   category=CLUSTER
   AND clusterName=Cluster_Name

CPU used by Impala

Time series of total Impala CPU usage in milliseconds.

Data Granularity: hourly

Units: milliseconds

tsquery:

SELECT
   counter_delta(impala_query_thread_cpu_time_rate)
WHERE
   category=CLUSTER
   AND clusterName=Cluster_Name

Memory used by Impala

Time series of Impala memory usage

Data Granularity: hourly

Units: byte seconds

tsquery:

SELECT
   counter_delta(impala_query_memory_accrual_rate)
WHERE
   category=CLUSTER
   AND clusterName=Cluster_Name

CPU used by YARN

The yarn_reports_containers_used_cpu metric used in this tsquery is generated per hour, therefore the data granularity used for this query is the raw metric value.

Data Granularity: Raw

Units: percent seconds

tsquery:

SELECT
   yarn_reports_containers_used_cpu FROM REPORTS
WHERE
   category=SERVICE
   AND clusterName=Cluster_Name

Memory used by YARN

Yarn memory usage. The yarn_reports_containers_used_memory metric used in this tsquery is generated per hour, therefore the data granularity used for this query is the raw metric value.

Data Granularity: raw metric value

Units: megabyte seconds

tsquery:

SELECT
   yarn_reports_containers_used_memory
FROM
   REPORTS
WHERE
   category=SERVICE
   AND clusterName=Cluster_Name

Pool-Level CPU and Memory Metrics

CPU used by Impala pool

CPU usage for an Impala pool.

Data Granularity: hourly

Units: milliseconds

tsquery:

SELECT
   counter_delta(impala_query_thread_cpu_time_rate)
WHERE
   category=IMPALA_POOL
   AND poolName=Pool_Name

Memory used by Impala pool

Data Granularity: hourly

Units: byte seconds

tsquery:

SELECT
   counter_delta(impala_query_memory_accrual_rate)
WHERE
   category=IMPALA_POOL
   AND poolName=Pool_Name

CPU used by YARN pool

Provides CPU metrics per YARN pool and user. You can aggregate a pool-level metric from this query.

Data Granularity: Raw

Units: percent seconds

tsquery:

SELECT
   yarn_reports_containers_used_cpu FROM REPORTS
WHERE
   category=YARN_POOL_USER

Memory used by YARN pool

Provides memory metrics per YARN pool and user. You can aggregate a pool-level metric from this query.

Data Granularity: hourly

Units: megabyte seconds

tsquery:

SELECT
   yarn_reports_containers_used_memory
FROM
   REPORTS
WHERE
   category=YARN_POOL_USER

YARN Metrics

YARN VCore usage

Data Granularity: Raw

Units: VCore seconds

tsquery:

SELECT
   yarn_reports_containers_used_vcores
FROM
   REPORTS
WHERE
   category=SERVICE
   AND clusterName=Cluster_Name

Total VCores available to YARN

Data Granularity: hourly

Units: Number of VCores (Note that this value is not multiplied by the time unit.)

tsquery:

SELECT
   total_allocated_vcores_across_yarn_pools + total_available_vcores_across_yarn_pools
WHERE
   category=SERVICE
   AND clusterName=Cluster_Name

YARN Memory usage

Data Granularity: Raw

Units: MB seconds

tsquery:

SELECT
   yarn_reports_containers_used_memory FROM REPORTS
WHERE
   category=SERVICE
   AND clusterName=Cluster_Name

Total memory available to YARN

Data Granularity: hourly

Units: MB (Note that this value is not multiplied by the time unit.)

tsquery:

SELECT
   total_available_memory_mb_across_yarn_pools + total_allocated_memory_mb_across_yarn_pools
WHERE
   category=SERVICE
   AND clusterName=Cluster_Name

Pool-level VCore usage

The results of this query return the usage for each user in each pool. To see the total usage for a pool, sum all users of the pool.

Data Granularity: Raw

Units: VCore seconds

tsquery:

SELECT
   yarn_reports_containers_used_vcores FROM REPORTS
WHERE
   category=YARN_POOL_USER

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool-level memory usage

The results of this query return the usage for each user in each pool. To see the total usage for a pool, sum all users of the pool.

Data Granularity: Raw

Units: MB seconds

tsquery:

SELECT
   yarn_reports_containers_used_memory FROM REPORTS
WHERE
   category=YARN_POOL_USER

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool-level allocated VCores

The results of this query return the usage for each user in each pool. To see the total usage for a pool, sum all users of the pool.

Data Granularity: raw metric value

Units: VCore seconds

tsquery:

SELECT
   yarn_reports_containers_allocated_vcores FROM REPORTS
WHERE
   category=YARN_POOL_USER

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool-level allocated memory

The results of this query return the usage for each user in each pool. To see the total usage for a pool, sum all users of the pool.

Data Granularity: raw metric value

Units: megabyte seconds

tsquery:

SELECT
   yarn_reports_containers_allocated_memory
FROM
   REPORTS
WHERE
   category=YARN_POOL_USER

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool-level steady fair share VCore

Data Granularity: hourly

Units: VCores

tsquery:

SELECT
   steady_fair_share_vcores
WHERE
   category=YARN_POOL

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool-level fair share VCore

Data Granularity: hourly

Units: VCores

tsquery:

SELECT
   fair_share_vcores
WHERE
   category=YARN_POOL

Pool-level steady fair share memory

Data Granularity: hourly

Units: MB

tsquery:

SELECT
   steady_fair_share_mb
WHERE
   category=YARN_POOL

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool-level fair share memory

Data Granularity: hourly

Units: MB

tsquery:

SELECT
   fair_share_mb
WHERE
   category=YARN_POOL

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Metric indicating contention

Data Granularity: hourly

Units: percentage

tsquery:

SELECT
   container_wait_ratio
WHERE
   category=YARN_POOL

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

YARN Contention-Related Metrics

Use the following metrics to monitor resource contention.

Pool-level allocated VCores when contention occurs

Data Granularity: hourly

Units: VCores

tsquery:

SELECT
   allocated_vcores_with_pending_containers
WHERE
   category=YARN_POOL

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool level steady fair share VCores when contention occurs

Data Granularity: hourly

Units: VCores

tsquery:

SELECT
   steady_fair_share_vcores_with_pending_containers
WHERE
   category=YARN_POOL

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool level fair share VCores when contention occurs

Data Granularity: hourly

Units: VCores

tsquery:

SELECT
   fair_share_vcores_with_pending_containers
WHERE
   category=YARN_POOL

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool level allocated memory when contention occurs

Data Granularity: hourly

Units: MB

tsquery:

SELECT
   allocated_memory_mb_with_pending_containers
WHERE
   category=YARN_POOL

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool level steady fair share memory when contention occurs

Data Granularity: hourly

Units: MB

tsquery:

SELECT
   steady_fair_share_mb_with_pending_containers
WHERE
   category=YARN_POOL

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Pool level fair share memory when contention occurs

Data Granularity: hourly

Units: MB

tsquery:

SELECT
   fair_share_mb_with_pending_containers
WHERE
   category=YARN_POOL

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Impala-Specific Metrics

To view metrics for a specific pool, add poolName=Pool Name to the tsquery statement.

Total reserved memory

Data Granularity: hourly

Units: MB seconds

tsquery:

SELECT
   total_impala_admission_controller_local_backend_mem_reserved_across_impala_daemon_pools
WHERE
   category=CLUSTER
   AND clusterName=Cluster_Name

Total used memory

Data Granularity: hourly

Units: MB seconds

tsquery:

SELECT
   total_impala_admission_controller_local_backend_mem_usage_across_impala_daemon_pools
 WHERE
   category=CLUSTER
   AND clusterName=Cluster_Name

Total available memory

Data Granularity: hourly

Units: MB seconds

tsquery:

SELECT
   total_mem_tracker_process_limit_across_impalads
WHERE
   category=CLUSTER
   AND clusterName=Cluster_Name