Workload trigger counters

A workload management trigger executes an action when the condition defined in the trigger expression is met. You can use all counters exposed by Apache Tez, the file system, and Hive with workload management. The following types of counters are available: Directed acyclic graph (DAG), file system, , input split, Hive, job, shuffle error, and task counters.

Table 1. DAG counters
Counter Name Aggregation Level
AM_CPU_MILLISECONDS AM
AM_GC_TIME_MILLIS AM
DATA_LOCAL_TASKS DAG
FALLOW_SLOTS_MILLIS_TASKS DAG
NUM_FAILED_TASKS DAG
NUM_FAILED_UBERTASKS DAG
NUM_KILLED_TASKS DAG
NUM_SUCCEEDED_TASKS DAG
NUM_UBER_SUBTASKS DAG
OTHER_LOCAL_TASKS DAG
RACK_LOCAL_TASKS DAG
SLOTS_MILLIS_TASKS DAG
TOTAL_LAUNCHED_TASKS DAG
TOTAL_LAUNCHED_UBERTASKS DAG
Table 2. File system counters
Counter Name Aggregation Level Comment
BYTES_READ DAG Supports all file system schemes. If S3A then counter must be prefixed with "S3A_". Example: S3A_FILE_BYTES_READ
BYTES_WRITTEN DAG
FILE_BYTES_READ DAG Local file system
FILE_BYTES_WRITTEN DAG Local file system
HDFS_BYTES_READ DAG
HDFS_BYTES_WRITTEN DAG
LARGE_READ_OPS DAG
READ_OPS DAG
WRITE_OPS DAG
Table 3. Input split counters
Counter Name Aggregation Level
DAG_GROUPED_INPUT_SPLITS DAG
DAG_INPUT_DIRECTORIES DAG
DAG_INPUT_FILES DAG
DAG_RAW_INPUT_SPLITS DAG
GROUPED_INPUT_SPLITS DAG
INPUT_DIRECTORIES DAG
INPUT_FILES DAG
RAW_INPUT_SPLITS DAG
VERTEX_GROUPED_INPUT_SPLITS VERTEX
VERTEX_INPUT_DIRECTORIES VERTEX
VERTEX_INPUT_FILES VERTEX
VERTEX_RAW_INPUT_SPLITS VERTEX
Table 4. Hive counters
Counter Name Aggregation Level Comment
CREATED_DYNAMIC_PARTITIONS DAG
CREATED_FILES DAG
DAG_TOTAL_TASKS DAG Sum of all tasks launched for the DAG
DESERIALIZE_ERRORS DAG
ELAPSED_TIME_MS QUERY Time from start of query submission to end of query execution
EXECUTION_TIME_MS DAG Time from start of DAG to end of DAG
RECORDS_IN DAG
RECORDS_OUT DAG
RECORDS_OUT_INTERMEDIATE DAG
VERTEX_TOTAL_TASKS VERTEX Equivalent to task paraellelism of particular vertex
Table 5. Job counters
Counter Name Aggregation Level
DATA_LOCAL_MAPS DAG
FALLOW_SLOTS_MILLIS_MAPS DAG
FALLOW_SLOTS_MILLIS_REDUCES DAG
NUM_FAILED_MAPS DAG
NUM_FAILED_REDUCES DAG
NUM_FAILED_UBERTASKS DAG
NUM_KILLED_MAPS DAG
NUM_KILLED_REDUCES DAG
NUM_UBER_SUBMAPS DAG
NUM_UBER_SUBREDUCES DAG
OTHER_LOCAL_MAPS DAG
RACK_LOCAL_MAPS DAG
SLOTS_MILLIS_MAPS DAG
SLOTS_MILLIS_REDUCES DAG
TOTAL_LAUNCHED_MAPS DAG
TOTAL_LAUNCHED_REDUCES DAG
TOTAL_LAUNCHED_UBERTASKS DAG
Table 6. Shuffle error counters
Counter Name Aggregation Level
BAD_ID DAG
CONNECTION DAG
IO_ERROR DAG
WRONG_LENGTH DAG
WRONG_MAP DAG
WRONG_REDUCE DAG
Table 7. Task counters
Counter Name Aggregation Level
ADDITIONAL_SPILL_COUNT DAG
ADDITIONAL_SPILLS_BYTES_READ DAG
ADDITIONAL_SPILLS_BYTES_WRITTEN DAG
COMBINE_INPUT_RECORDS DAG
COMBINE_OUTPUT_RECORDS DAG
COMMITTED_HEAP_BYTES DAG
CPU_MILLISECONDS DAG
FIRST_EVENT_RECEIVED DAG
GC_TIME_MILLIS DAG
INPUT_GROUPS DAG
INPUT_RECORDS_PROCESSED DAG
INPUT_SPLIT_LENGTH_BYTES DAG
LAST_EVENT_RECEIVED DAG
MERGE_PHASE_TIME DAG
MERGED_MAP_OUTPUTS DAG
NUM_DISK_TO_DISK_MERGES DAG
NUM_FAILED_SHUFFLE_INPUTS DAG
NUM_MEM_TO_DISK_MERGES DAG
NUM_SHUFFLED_INPUTS DAG
NUM_SKIPPED_INPUTS DAG
NUM_SPECULATIONS DAG
OUTPUT_BYTES DAG
OUTPUT_BYTES_PHYSICAL DAG
OUTPUT_BYTES_WITH_OVERHEAD DAG
OUTPUT_LARGE_RECORDS DAG
OUTPUT_RECORDS DAG
PHYSICAL_MEMORY_BYTES DAG
REDUCE_INPUT_GROUPS DAG
REDUCE_INPUT_RECORDS DAG
REDUCE_OUTPUT_RECORDS DAG
REDUCE_SKIPPED_GROUPS DAG
REDUCE_SKIPPED_RECORDS DAG
SHUFFLE_BYTES DAG
SHUFFLE_BYTES_DECOMPRESSED DAG
SHUFFLE_BYTES_DISK_DIRECT DAG
SHUFFLE_BYTES_TO_DISK DAG
SHUFFLE_BYTES_TO_MEM DAG
SHUFFLE_CHUNK_COUNT DAG
SHUFFLE_PHASE_TIME DAG
SKIPPED_RECORDS DAG
SPILLED_RECORDS DAG
SPLIT_RAW_BYTES DAG
VIRTUAL_MEMORY_BYTES DAG