Retrieving audit events

In Private Cloud, Control Plane audit data can be retrieved by configuring the OpenTelemetry (OTel) collector. The OTel collector can be configured to send data to external systems – such IBM Guardian – using the syslog OTel exporter.

OTel collector configuration

The OTel collector is used to receive the audit events. It supports the following three types of data:

Traces
Metrics
Logs

The audit events are treated as logs in the OTel collector. Currently configuration of an OpenTelemetry exporter is only possible by editing the Kubernetes configmap cdp-release-opentelemetry-collector in the <cdp-project> namespace.

The default config contains only the logging exporter. To collect audit events in an external system such as rsyslog, the appropriate exporter config needs to be added there. To edit the configmap, run the following command:

kubectl edit cm cdp-release-opentelemetry-collector -n <cdp-project>

The default structure of the configmap is as follows:

# Valid values are "daemonset", "deployment", and "statefulset".
mode: "deployment"

config:
  receivers:
    jaeger: null
    prometheus: null
    zipkin: null
  service:
    pipelines:
      logs:
        exporters:
          - logging
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
      metrics: null
      traces: null

ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

Forwarding to OTel

Forwarding of audit events to the OTel collector is disabled by default. You can enable OTel to receive audit events by configuring the following environment variable:

kubectl edit deploy cdp-release-thunderhead-audit-private -n <cdp-project>

# Add the following environment variable
        - name: FORWARDING_ENABLED
          value: "true"

Syslog OTel exporter configuration

This section provides an example of how to modify the OTel configmap to send audit events to a rsyslog endpoint using the syslog exporter. An example of adding a syslog exporter is described below. For additional information about the syslog exporter example, see: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/syslogexporter/examples/config_with_syslog_receiver.yaml

Sample syslog insecure configuration

The following snippet from the cdp-release-opentelemetry-collector configmap shows how to configure a syslog exporter without TLS:

apiVersion: v1
data:
  relay: |
    exporters:
      logging:
        verbosity: basic
      syslog:
       network: tcp
       port: 514
       endpoint: adt-demo-1.vpc.cloudera.com
       tls:
         insecure: true
       protocol: rfc3164
.
.
.
      pipelines:
        logs:
          exporters:
          - logging
          - syslog

Additionally syslog needs to be added under the services | logs | pipelines | exporters section.

In this example the rsyslog audit events are logged under: /var/log/messages:

Aug 30 22:45:35 ena-3.vpc.cloudera.com - {"action":"setEnvironmentSetting","actor_crn":"crn:altus:iam:us-west-1:8f5a8f29-7834-4b66-8946-ebd7d2cf8508:user:17aa0daf-4f92-45fa-a8c9-6ca0478eec31","agent":"environments","evtTime":1693435535994,"id":"c1080e42-b0ba-4bd4-b1dd-4bd0f7881f49","reqUser":"admin","request_id":"44c24ab0-34bb-456a-a945-f10a72ad49c7","response_parameters":"{ }","result":"SUCCESS","text":""}
Aug 30 22:46:45 ena-3.vpc.cloudera.com - {"action":"getUser","actor_crn":"crn:altus:iam:us-west-1:8f5a8f29-7834-4b66-8946-ebd7d2cf8508:user:17aa0daf-4f92-45fa-a8c9-6ca0478eec31","agent":"iam","api_version":"__API_VERSION__","cliIP":"10.42.1.7","evtTime":1693435605667,"id":"58724fb9-69d5-4a92-b1f5-5412809a9e8c","mutating":"false","reqData":"{ \"userId\": null }","reqUser":"admin","request_id":"ec66f71d-ce19-4d11-be4d-b7372bd7a23a","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"}

Sample syslog secure configuration

The following example shows how to configure a server CA for TLS. The ca_file must have the value /etc/opt/certs/ca.pem as that is the Private Cloud truststore file.

apiVersion: v1
data:
  relay: |
    exporters:
      logging:
        verbosity: basic
      syslog:
       network: tcp
       port: 6514
       endpoint: <rsyslog-hostname>
       tls:
         ca_file: /etc/opt/certs/ca.pem
       protocol: rfc3164
.
.
.
      pipelines:
        logs:
          exporters:
          - logging
          - syslog

Please note that this configuration will only work if the rsyslog server has TLS configured. Additional information on rsyslog TLS is available here: https://www.rsyslog.com/doc/master/tutorials/tls.html

We support TLS out of the box. mTLS is not supported – to configure mTLS, see TLS Configuration Settingsfor more information.

For added context, the following steps were done to test rsyslog using TLS. This test was done on a machine running RHEL 8.8.

The following lines were added to /etc/rsyslog.conf

module(
load="imtcp"
StreamDriver.Name="gtls"
StreamDriver.Mode="1"
StreamDriver.Authmode="anon"
) # needs to be done just once
input(type="imtcp" port="6514")

#### GLOBAL DIRECTIVES ####
global(
DefaultNetstreamDriver="gtls"
DefaultNetstreamDriverCAFile="/certs/myCA.pem"
DefaultNetstreamDriverCertFile="/certs/rsyslog.crt"
DefaultNetstreamDriverKeyFile="/certs/rsyslog.key"

If GnuTLS library is not already present, it must be installed:

yum install rsyslog-gnutls

The certs were created using a self-signed CA. The commands are:

# Create the CA private key
openssl genrsa -out myCA.key 2048

# Create the CA public key
openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1825 -out myCA.pem

# Create the server cert private key
openssl genrsa -out rsyslog.key 2048

# Create a certificate signing request using the private key above
openssl req -new -key rsyslog.key -out rsyslog.csr

# Create an ext file rsyslog.ext with the contents below
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = <rsyslog-hostname>

# Create the server cert
openssl x509 -req -in rsyslog.csr -CA myCA.pem -CAkey myCA.key -CAcreateserial -out rsyslog.crt -days 825 -sha256 -extfile rsyslog.ext

Import the CA cert myCA.pem into the miscellaneous section of the CA certificates from the Control Plane UI.

If you are not using a trusted CA cert, the server cert rsyslog.crt must also be imported.