This is the documentation for CDH 5.1.x. Documentation for other versions is available at Cloudera Documentation.

Auditing Impala Operations

To monitor how Impala data is being used within your organization, ensure that your Impala authorization and authentication policies are effective, and detect attempts at intrusion or unauthorized access to Impala data, you can use the auditing feature in Impala 1.1.1 and higher:

  • Enable auditing by including the option -audit_event_log_dir=directory_path in your impalad startup options. The path refers to a local directory on the server, not an HDFS directory.
  • Decide how many queries will be represented in each log files. By default, Impala starts a new log file every 5000 queries. To specify a different number, include the option -max_audit_event_log_file_size=number_of_queries in the impalad startup options. Limiting the size lets you manage disk space by archiving older logs, and reduce the amount of text to process when analyzing activity for a particular period.
  • Configure the Cloudera Navigator product to collect and consolidate the audit logs from all the nodes in the cluster.
  • Use the Cloudera Manager product to filter, visualize, and produce reports based on the audit data. (The Impala auditing feature works with Cloudera Manager 4.7 or higher.) Check the audit data to ensure that all activity is authorized and/or detect attempts at unauthorized access.

Continue reading:

Durability and Performance Considerations for Impala Auditing

The auditing feature only imposes performance overhead while auditing is enabled.

Because any Impala node can process a query, enable auditing on all nodes where the impalad daemon runs. Each node stores its own log files, in a directory in the local filesystem. The log data is periodically flushed to disk (through an fsync() system call) to avoid loss of audit data in case of a crash.

The runtime overhead of auditing applies to whichever node serves as the coordinator for the query, that is, the node you connect to when you issue the query. This might be the same node for all queries, or different applications or users might connect to and issue queries through different nodes.

To avoid excessive I/O overhead on busy coordinator nodes, Impala syncs the audit log data (using the fsync() system call) periodically rather than after every query. Currently, the fsync() calls are issued at a fixed interval, every 5 seconds.

By default, Impala avoids losing any audit log data in the case of an error during a logging operation (such as a disk full error), by immediately shutting down the impalad daemon on the node where the auditing problem occurred. You can override this setting by specifying the option -abort_on_failed_audit_event=false in the impalad startup options.

Format of the Audit Log Files

The audit log files represent the query information in JSON format, one query per line. Typically, rather than looking at the log files themselves, you use the Cloudera Navigator product to consolidate the log data from all Impala nodes and filter and visualize the results in useful ways. (If you do examine the raw log data, you might run the files through a JSON pretty-printer first.)

All the information about schema objects accessed by the query is encoded in a single nested record on the same line. For example, the audit log for an INSERT ... SELECT statement records that a select operation occurs on the source table and an insert operation occurs on the destination table. The audit log for a query against a view records the base table accessed by the view, or multiple base tables in the case of a view that includes a join query. Every Impala operation that corresponds to a SQL statement is recorded in the audit logs, whether the operation succeeds or fails. Impala records more information for a successful operation than for a failed one, because an unauthorized query is stopped immediately, before all the query planning is completed.

Impala records more information for a successful operation than for a failed one, because an unauthorized query is stopped immediately, before all the query planning is completed.

The information logged for each query includes:

  • Client session state:
    • Session ID
    • User name
    • Network address of the client connection
  • SQL statement details:
    • Query ID
    • Statement Type - DML, DDL, and so on
    • SQL statement text
    • Execution start time, in local time
    • Execution Status - Details on any errors that were encountered
    • Target Catalog Objects:
      • Object Type - Table, View, or Database
      • Fully qualified object name
      • Privilege - How the object is being used (SELECT, INSERT, CREATE, and so on)

Which Operations Are Audited

The kinds of SQL queries represented in the audit log are:

  • Queries that are prevented due to lack of authorization.
  • Queries that Impala can analyze and parse to determine that they are authorized. The audit data is recorded immediately after Impala finishes its analysis, before the query is actually executed.

The audit log does not contain entries for queries that could not be parsed and analyzed. For example, a query that fails due to a syntax error is not recorded in the audit log. The audit log also does not contain queries that fail due to a reference to a table that does not exist, if you would be authorized to access the table if it did exist.

Certain statements in the impala-shell interpreter, such as CONNECT, SUMMARY, PROFILE, SET, and QUIT, do not correspond to actual SQL queries, and these statements are not reflected in the audit log.

Reviewing the Audit Logs

You typically do not review the audit logs in raw form. The Cloudera Manager agent periodically transfers the log information into a back-end database where it can be examined in consolidated form. See the Cloudera Navigator documentation for details.

Page generated September 3, 2015.