Metric Sources Sent to Workload Manager

Describes the resources from which Telemetry Publisher and Databus Producer collects diagnostic metrics.

Telemetry Publisher and Databus Producer collect and transmit metrics, as well as configuration and log files, from Impala, Oozie, Hive, YARN, and Spark services for jobs running on your clusters to Workload Manager.

The following example, describes how metrics are collected from a Data Hub public cloud environment:

  • Pull — Telemetry Publisher pulls diagnostic metrics from Oozie, YARN, and Spark periodically (by default, once per minute).
  • Push — An Agent pushes diagnostic data from Hive and Impala to Telemetry Publisher within 5 seconds after a job finishes.
The following diagram, shows an example of a Data Hub public cloud environment:


After the diagnostic data reaches Telemetry Publisher or Databus Producer, it is stored temporarily in its data directory and periodically (once per minute) exported to Workload Manager.