Cloudera Docs

Log collection

Cloudera requires that the logs of Cloudera Streams Messaging - Kubernetes Operator components are stored long term for diagnostic and supportability purposes. Learn about the settings for platform level log collection recommended by Cloudera.

Logs can be collected using the log collector feature of the specific Kubernetes platform. Ensuring that log collection is correctly set up is your responsibility. Cloudera recommends at least one week of retention time for the collected logs.

Using kubectl logs is not sufficient in some cases. This is because pods are created and destroyed dynamically by operator applications. The logs of destroyed pods are deleted, which makes them inaccessible. Log collection can ensure that the logs of already deleted pods are retained.

The following collects the recommended and required logging practices for specific Kubernetes platforms.

Openshift

Latest OpenShift versions support the Vector log collector. Log collection and forwarding can be configured using a ClusterLogging resource.

Ensure the following if you are on Openshift:

  • The ClusterLogging resource includes all namespaces and pods used by the operators.

  • Use a log sink that supports time-based retention. The ClusterLogging resource supports a number of log sinks. Cloudera recommends using a sink that supports time-based retention to limit storage costs. Additionally, the selected sink should allow easy access to the collected logs when a diagnostic investigation requires them.

RKE2 with Rancher

Rancher relies on the Logging operator for log collection. Log collection can be configured using Flow, ClusterFlow, Output, and ClusterOutput resources.

Ensure the following if you are on RKE2 with Rancher:

  • When using a Flow resource, ensure that the Flow resource includes all namespaces and pods used by the operators.

  • Use a log sink that supports time-based retention. Output and ClusterOutput resources support a number of log sinks. Cloudera recommends using a sink that supports time-based retention to limit storage costs. Additionally, the selected sink should allow easy access to the collected logs when a diagnostic investigation requires them.