Cloudera Docs

Diagnostic logs with Cloudera AI workloads

Diagnostic logs provide a real-time view of output generated by workloads running in Cloudera AI. They enable you to monitor workload startup, track code execution, and troubleshoot errors without requiring direct access to the underlying Kubernetes containers.

Diagnostic logs are available for interactive sessions and other supported workloads, including long-running and distributed workloads such as Spark jobs.

Cloudera AI automatically collects generated logs and arranges the events in the precise chronological order. The ordered data is processed and continuously streams new messages as they are generated. The live data stream feeds directly into the Cloudera AI user interface, allowing users to monitor their workloads in real time. This stream can include the following information:
  • Application output generated by your workload code
  • Runtime startup and status messages
  • Warnings, exceptions, and error messages
  • Processing and execution status updates
  • Logs generated by supported distributed workload components, such as Spark executors

Live Log preserves the event sequence for each workload, making it easier to follow startup, execution, and failure events in the order they occur.

Access control for log data

Log data is isolated by workloads. You can view logs only for workloads you are authorized to access.

Access permissions can depend on factors such as workload ownership, project membership, application permissions, and administrative roles.

Log retention

Log data is periodically removed according to the retention policy configured for the Cloudera AI Workbench. Logs belonging to deleted or sufficiently old workloads might no longer be available.

Logs can be used primarily for the following purposes:

  • Monitoring workloads while they are running

  • Investigating startup and execution failures

  • Reviewing recent workload output

  • Identifying errors before downloading a complete workload log bundle

Log optimization

Workloads that generate large volumes of debug output can make important messages harder to identify and can consume logging resources unnecessarily. For long-running applications and Spark jobs, apply the following recommendations:

  • Use INFO, WARN, or ERROR log levels unless detailed debugging is required.
  • Avoid repeatedly logging large objects, records, or repetitive status messages.
  • Disable verbose debug logging after troubleshooting is complete.
  • Write clear, structured log messages that identify the operation or task being performed.

Log data is designed to stream workload output to the user interface in real time. It is not intended to replace a centralized log search or analytics system for querying large volumes of historical log data across extended time ranges.