Monitoring Cloudera Data Engineering with Cloudera Observability

Integrating Cloudera Data Engineering on premises with Cloudera Observability allows for collecting Spark job telemetry, viewing resource usage through job-level metrics, optimizing Spark workloads, troubleshooting performance through dashboards, and accessing historical analysis from the Cloudera Data Engineering job runs page.

Integrating Cloudera Data Engineering with Cloudera Observability enables the collection of execution telemetry for Spark jobs. This integration provides administrators and users with visibility into resource usage and monitors Spark job-specific execution metrics. Using the generated dashboards, you can administer, optimize, and troubleshoot Spark job workloads grouped by users and pools.

For Cloudera Data Engineering users, Cloudera Observability provides a historical analysis tool for Spark batch jobs. After a job completes, you can navigate directly from the Cloudera Data Engineering job runs page to Cloudera Observability to understand why a specific run took a certain amount of time. This historical context allows you to compare current runs against previous performance trends and perform a deeper analysis of job stages, including CPU, memory, and I/O usage.

Feature highlights:

• Historical trends and baseline analysis: Establish performance benchmarks by analyzing long-term telemetry patterns for Spark job runs.
• Baseline scores and anomaly detection: Identify performance deviations by using baseline scores for Spark job telemetry payloads.
• Job comparison: Compare Spark job behavior and performance across different runs using telemetry data.
• Health checks: Use health check features to monitor Spark job status.
• Resource analysis widgets: Use dedicated widgets to analyze resource consumption for Spark job telemetry.
• Cost tracking: Calculate Spark job costs based on actual resource consumption recorded in telemetry payloads.