Cloudera Docs

Auditing for Cloudera Lakehouse Optimizer

Cloudera Lakehouse Optimizer auditing provides an audit trail of user and system activities on the service, including policy changes, namespace operations, authentication events, and table maintenance actions. Audit events are written as NDJSON files to a configurable filesystem location.

Cloudera Lakehouse Optimizer emits an audit event for every state-changing REST API call and every authentication attempt. Events are buffered in memory and periodically written to audit log files on a Hadoop-compatible filesystem such as HDFS, S3, GCS, Azure Blob Storage, or local FS.

What is audited

Cloudera Lakehouse Optimizer audits the following categories of activity:

  • Policies: Create, update, delete, view, subscribe, unsubscribe, evaluate, dry-run, pause, and resume operations—including failed or denied requests.
  • Namespaces: Declare, reschedule, delete, pause, resume, list, and view operations.
  • Tables: Pause, unpause, list, statistics, and registry eviction operations.
  • Tasks: View, list, and event-ingestion trigger operations.
  • Configuration: Reconfigure, cache clear, health check, and backup or restore operations.
  • UI: Read-only browse operations performed through the Cloudera Lakehouse Optimizer UI.

Requests denied by validation or authorization checks are recorded with a FAILURE or DENY outcome before the service returns a response.

Audit event format

Each audit event is a single-line JSON object (NDJSON). The following table describes the audit event fields

Table 1. Audit event fields
Field Description
timestamp, hostName, serviceName Set when the audit event is published.
userName, clientIp, role Read from the authenticated user session. For login and authentication-failure events, these values are supplied manually.
action Identifies the REST operation that was attempted (for example, CREATE_POLICY_DEFINITION or LOGIN).
outcome SUCCESS, FAILURE, or DENY.
message Human-readable description of the event.
Resource fields policies, namespaces, tables, catalog, resourceUri, and taskId when applicable.

Audit log files

Audit files are written to <basePath>/<prefix>.<sequence>, for example hdfs:///user/dlm/audit-logs/clo-audit.log.42. On HDFS and local filesystems, the default file permission is 775. Files roll over based on configured time and event-count thresholds. On startup, Cloudera Lakehouse Optimizer scans existing audit files and resumes writing to the latest file when appropriate.

Cloudera Lakehouse Optimizer does not delete old audit files. Configure external retention such as HDFS quotas, object-store lifecycle policies, or a log-shipping pipeline.

Supported filesystems

Cloudera Lakehouse Optimizer supports local, HDFS, and object-store locations through the Hadoop FileSystem API. HDFS supports native append and POSIX file permissions. Object stores such as S3 do not support native append or POSIX permissions; Cloudera Lakehouse Optimizer falls back to read-and-rewrite and relies on bucket policies for access control. Configure bucket versioning, object lock, and restrictive IAM policies for tamper-evident audit storage on object stores.

Audit pipeline behavior

If audit initialization fails, Cloudera Lakehouse Optimizer continues to serve requests. Subsequent audit events are dropped, and the service logs a rate-limited warning with the drop count. Monitor the application log for audit pipeline errors.

For the complete list of auditing configuration properties, see Cloudera Lakehouse Optimizer auditing configuration properties.