Configuring audit spool alert notifications

You can enable and configure alerts for Ranger plugin supported services on Cloudera Manager that notify when audit spool files are accumulated.

Ranger stores plugin access audit events (audit logs) in Solr and in HDFS. Typically, you store audit logs in Solr for short-term auditing purposes and in HDFS for longer-term purposes. When the Solr server is down, Ranger plugin audit logs are stored in the spool directory as spool files. You configure the spool directory location using the following properties:

For Solr: xasecure.audit.destination.solr.batch.filespool.dir = /var/log/<service_name>/audit/solr/spool
For HDFS: xasecure.audit.destination.hdfs.batch.filespool.dir = /var/log/<service_name>/audit/hdfs/spool

If the Solr server goes down for a long period of time or, if a large number of audit events occur while the Solr server is down; then spool files accumulate in the spool directory. Spool file accumulation consumes system memory. Sometimes, audit records in spool files become corrupted, and may not be restored when the Solr server returns to a running state. Corrupted, un-restored records also cause spool file accumulation. This requires manual cleanup of corrupted spool files. An unnoticed large accumulation or "piling up" of spool files may fill the local filesystem and result in service failure.

After you enable spool directory metric usage for a service, an alert appears on the Cloudera Manager UI which notifies the user when spool files have piled up in the spool directory. The Cloudera Manager agent measures the disk usage of the spool directory and registers it as a metric value. This metric value is compared against a threshold value. The spool alert appears on the Cloudera Manager UI if the metric value is greater than the threshold value.

Single-level metrics for collecting the disk usage of Solr and Hdfs Ranger plugin spool directories are registered in Cloudera Manager, using the following disk usage metric names:

Solr: ranger_plugin_solr_spool_directory_size
HDFS: ranger_plugin_hdfs_spool_directory_size

The following table lists Ranger plugin supported service names and roles that support spool alerts.

Table 1. Ranger plugin supported services and their roles supporting audit spool alerts
Services	Roles
HDFS	NAMENODE
HIVE	HIVEMETASTORE
HIVE_ON_TEZ	HIVESERVER2
HBASE	MASTER, REGIONSERVER
YARN	RESOURCEMANAGER
IMPALA	IMPALAD, CATALOGSERVER
ATLAS	ATLAS_SERVER
KAFKA	KAFKA_BROKER
KNOX	KNOX_GATEWAY
KUDU	KUDU_MASTER
RANGER_KMS	RANGER_KMS_SERVER
RANGER_RAZ	RANGER_RAZ_SERVER
SCHEMAREGISTRY	SCHEMA_REGISTRY_SERVER
STREAMS_MESSAGING_MANAGER	STREAMS_MESSAGING_MANAGER_SERVER

To enable or disable spool directory alerts:

Go to Cloudera Manager > Ranger Plugin Supported Services page > Configuration.
In Search, type spool directory.

The following configuration properties display (this example uses the Kafka Broker role from Kafka service)

Figure 1. Audit Spool Alert Configurations for Kafka Broker
In Enable Spool Directory Metric Usage for <service-name>, check the box.
(un-checking the Enable box disables spool directory metric usage)
1. Refresh the role.
  For example, the Kafka service which supports Ranger Plugin in Kafka Broker role, go to Cloudera Manager > Kafka > Instances > Kafka Broker > Actions > Refresh Kafka Broker.
In Ranger Plugin Spool Directory Usage Thresholds for <service-name>, type values and select units.

By default the threshold is set to 1 GB. If the disk usage of the spool directory exceeds this threshold, an alert will be shown.

These spool alert details appear on the Ranger plugin service Role status page in the Cloudera Manager UI, as shown for Kafka Broker role in the following example:

Figure 2. Audit spool details for the Kafka Broker role

Also, an alert appears on the Cloudera Manager Home page next to the service name, as shown for Kafka service in the following example.

Figure 3. Audit file spool notification for Kafka service plugin on Cloudera Manager home page.

Health Test For Ranger Plugin Spool Alert — Figure 3. Audit file spool notification for Kafka service plugin on Cloudera Manager home page.

Optionally, you can create a graph for each spool alert metric.