Workload Manager Architecture

Describes the components and architecture of a basic Workload Manager environment.

A Workload Manager environment comprises the following:
  • Cloudera Environment, which is a secure and governed cloud service platform. The Workload Manager service and all its main component services run in the Control Plane of the Cloudera Workload Manager framework. Users access the Workload Manager web user interface from the web host server in this framework.
  • Working Environment, which contains your Workload Clusters in your Workload environments, such as Production, Development, and Staging.
  • Workload Cluster, which is one or more CDP clusters managed by Cloudera. Depending on your environment's Cloudera data service, each cluster is associated with either Telemetry Publisher or Databus WXM Client (formally named Databus Producer) through the Cloudera Management Console that also manages your workload services and environments, users, and resources.

The below Workload Manager Architecture for Public Cloud diagram shows the communication between Workload Manager and your workload clusters through either Telemetry Publisher or Databus WXM Client. Where, the Workload Manager service, including its main component services, run in the Cloudera Control Plane and the area on the right is your Working Environment that contains the clusters and services required to run your workload processes.

Cloudera Management Console (not shown) manages the clusters and services in each of your working environments. Workload Analytics is enabled for Workload Manager from the Cloudera Management Console when you register your environment.

Telemetry Publisher and Databus WXM Client collect and send diagnostic information about job and query processes from your Workload Clusters to Workload Manager and its services. To ensure that all data transfer is secure between your Workload Clusters and Workload Manager, its services, and its S3 bucket, Telemetry Publisher and Databus WXM Client communicate with Network Load Balancer, which is inside the Cloudera Environment's demilitarized zone, through the secure Transport Layer Security (TLS) protocol, as follows:

  1. When a job is completed, Telemetry Publisher and Databus WXM Client request access to Workload Manager and its services through Network Load Balancer, which is inside the Cloudera Environment's demilitarized zone. When received and processed, a signed Workload Manager S3 URL is returned to Telemetry Publisher and Databus WXM Client.
  2. When the URL is received, Telemetry Publisher and Databus WXM Client perform a secure and direct protocol test using the Workload Manager S3 URL, before sending any diagnostic data.
Figure 1. Workload Manager Architecture for Public Cloud


The following diagram shows the services from which Telemetry Publisher collects diagnostic metrics in a Data Hub environment:
Figure 2. Data Hub Environment


The following diagrams show the services from which Databus WXM Client (formally named Databus Producer) collects diagnostic metrics in a Cloudera Data Warehouse (CDW) and a Cloudera Data Engineering (CDE) working environment. Where, the Databus WXM Client continually communicates and checks for recently completed jobs and queries to see if there is any diagnostic data to transfer, such as task and event logs and job and query history files, with the Hive DDL History, LLAP History, and Impala History pollers (HiveDDLHistoryPoller, LlapHistoryPoller, and ImpalaHistoryPoller):
  • Figure 3. CDW Environment