Forwarding logs to your observability system

You can forward logs from environments activated in Cloudera Data Warehouse (CDW) to observability and monitoring systems such as Datadog, New Relic, or Splunk. You learn how to configure a CDW environment for these systems.

After configuring log forwarding as described in this task, logs flow from CDW to your system automatically. You enjoy the convenience of sorting, searching, and viewing logs on your own system instead of grepping logs from diagnostic bundles on S3 or ABFS. In addition to configuring the log forwarding, you configure removal of debug logs and text strings from the logs. You can configure log forwarding to one of the following observability systems:

Datadog — https://github.com/DataDog/fluent-plugin-datadog
Honeycomb.io — https://docs.honeycomb.io/getting-data-in/logs/log-collectors/fluentd/
New Relic — https://github.com/newrelic/newrelic-fluentd-output
Splunk — https://github.com/splunk/fluent-plugin-splunk-hec (covers both Splunk-HEC and Splunk-SCS)

You create the log forwarding configuration in valid fluentd format. The configuration is inserted into a larger fluentd configuration. All fluentd events are copied and relabeled with the new label @cloudera_cdw. Your custom configuration is then inserted between <label> tags:

<label @cloudera_cdw>

customer config goes here

</label>

You can use any of the built-in fluentd filter, formatter, parser, or output plugins to build the custom config.

Before configuring log forwarding you must activate an AWS environment or activate an Azure environment in CDW.
You must be familiar with fluentd and accept the responsibility of configuring log forwarding to your observability systems.

In the Data Warehouse service, expand the Environments column by clicking More….
In Environments, locate your environment, and click Options > Edit > Observability tab.
Decide how you want to create the fluentd config.
- Write your own fluentd config from the ground up.
- Use a Cloudera-provided snippet as a template to write your fluentd config.
In Log Forwarding Configuration, click Options.
Select one of the systems, such as Datadog, to configure.
A fluentd snippet appears. For example, the Datadog snippet appears:
Replace the snippet with the fluentd config you wrote from the ground up, or customize the provided snippet.
For example, to customize the provided snippet replace the placeholder {{API Key}} with the actual key.
(Optional) If debug level log messages are not designed, add a fluentd filter to remove them: In the environment, click Options , and select Remove Debug Logs.
The fluentd snippet appears for removing debug logs. For example:

No user customization is necessary to remove debug logs.
(Optional) If certain log messages do not provide value for you, remove them with a fluentd grep exclude filter: In the environment, click Options , select Grep Exclude, and replace {{PATTERN}} with the grep expression that matches the phrase you want to exclude.

For more information about using Grep Exclude, see https://docs.fluentd.org/filter/grep.
If you use a proxy server for outbound traffic, provide the proxy server's CA certificates PEM bundle as described in the next task.
Click Apply.
CDW tests the log forwarding configuration and proxy CA certificates bundle, and saves the configuration if both are valid. An invalid log forwarding config error message appears in the event of a configuration problem. For example:
If your configuration is valid, CDW initiates a restart of fluentd to apply the updated config. You see the following indicators of success:
- The environment Running indicator changes, blinks Updating, and then once again says Running.
- You see logs appearing in your observability system.
  Many factors affect how long it takes for forwarding to begin, but generally, the bigger your CDW environment, the longer it takes.