Service Audit Events
Service audit events are the events generated by a given service running on the cluster. Users with the appropriate permissions (Auditing Viewer, Full Administrator) can view audit events in the Cloudera Navigator console or by using the APIs. Audit events can include the fields listed in the tables below.
The Cloudera Navigator console Audits includes events collected by Cloudera Manager: service lifecycle events (activate, create, delete, deploy, download, install, start, stop, update, upgrade, and so on) and user security-related events (add and delete user, login failed and succeeded). See Lifecycle and Security Auditing for more details on Cloudera Manager audit events.
| Display Name | Field in API | Field in Streaming | Description |
|---|---|---|---|
| Additional Info | additional_info | additionalInfo | JSON text that contains more details about an operation performed on entities in Navigator Metadata Server. |
| Allowed | allowed | allowed | Indicates whether the request to perform an operation failed or succeeded. A failure occurs if the user is not authorized to perform the action. |
| Collection Name | collection_name | collectionName | The name of the affected Solr collection. |
| Database Name | database_name | db
databaseName (Sentry) |
For Sentry, Hive, and Impala, the name of the database on which the operation was performed. |
| Delegation Token ID | delegation_token_id | delegationTokenId | Delegation token identifier generated by HDFS NameNode that is then used by clients when submitting a job to JobTracker. |
| Destination | dest | dst | Path of the final location of an HDFS file in a rename or move operation. |
| Entity ID | entity_id | entityId | Identifier of a Navigator Metadata Server entity. The ID can be retrieved using the Navigator Metadata Server API. |
| Event Time | timestamp | time | Date and time an action was performed. The Navigator Audit Server stores the timestamp in the timezone of the Navigator Audit Server. The Cloudera Navigator console displays the timestamp converted to the local timezone. Exported audit events contain the stored timestamp. |
| Family | family | family | HBase column family. |
| Impersonator | impersonator | impersonator | Name of user (service) that invokes an action on behalf of another user (service). Impersonator field always displays values when Sentry is not used with the cluster. For clusters that use Sentry, the Impersonator field displays values for all services other than Hive. |
| IP Address | ipAddress | ip | The IP address of the host where an action occurred. |
| Object Type | object_type | objType
objectType (Sentry) |
For Sentry, Hive, and Impala, the type of the object (TABLE, VIEW, DATABASE) on which operation was performed. |
| Operation | command | op | Commands executed by component. See Operations by Component for details. For Cloudera Navigator operations, see Navigator Metadata Server Sub Operations. |
| Operation Params | operation_params | operationParams | Solr query or update parameters used when performing the action. |
| Operation Text | operation_text |
opText operationText (Sentry) |
For Sentry, Hive, and Impala, the SQL query that was executed by user. For Hue, the user or group that was added, edited, or deleted. |
| Permissions | permissions | perms | HDFS permission of the file or directory on which the HDFS operation was performed. |
| Privilege | privilege | privilege | Privilege needed to perform an Impala operation. |
| Qualifier | qualifier | qualifier | HBase column qualifier. |
| Query ID | query_id | — | The query ID for an Impala operation. (Internal use only) |
| Resource | resource | — | A service-dependent combination of multiple fields generated during fetch. This field is not supported for filtering as it is not persisted. |
| Resource Path | resource_path | path
resourcePath (Sentry) |
HDFS URL of Hive objects (TABLE, VIEW, DATABASE, and so on). Used for HDFS, Sentry. |
| Service Name | service | service | The name of the service that performed the action. |
| Session ID | session_id | — | Impala session ID. (Internal use only) |
| Solr Version | solr_version | solrVersion | Solr version number. |
| Source | src | src | Path of the HDFS file or directory present in an HDFS operation. |
| Status | status | status | Status of an Impala operation providing more information on success or failure. |
| Stored Object Name | stored_object_name | name | Name of a policy, saved search, or audit report in Navigator Metadata Server. |
| Sub Operation | sub_operation | subOperation | Operations performed by Navigator Metadata Server are identified by subsystem (authorization, auditing, for example) and by sub-operation within that subsystem. See Navigator Metadata Server Sub Operations for details. |
| Table Name | table_name | table
tableName (Sentry) |
For Sentry, HBase, Hive, and Impala, the name of the table on which action was performed. |
| Usage Type | objUsageType | Hive only. | |
| Username | username | user | The name of the user that performed the action. |
Operations by Component
The Operation field of an audit event includes the actions taken (commands run) on the component. Operations for Cloudera Navigator (and sub-operations) are listed Navigator Metadata Server Sub Operations.
| Component | Action taken |
|---|---|
| HBase |
addColumn, append, assign, balance, balanceSwitch, checkAndDelete, checkAndPut, compact, compactSelection, createTable, delete, deleteColumn, deleteTable, disableTable, enableTable, exists, flush, get, getClosestRowBefore, grant, increment, incrementColumnValue, modifyColumn, modifyTable, move, put, revoke, scannerOpen, shutdown, split, stopMaster, unassign |
| HDFS | append, concat, create, createSymlink, delete, fsck, getfacl*, getfileinfo, listEncryptionZones, listSnapshottableDirectory, listStatus, mkdirs, open, rename, setfacl*, setOwner, setPermission, setReplication, setTimes |
| HiveServer2 /Beeline | ALTER_PARTITION_MERGE, ALTER_TABLE_MERGE, ALTERDATABASE, ALTERINDEX_PROPS, ALTERINDEX_REBUILD,
ALTERPARTITION_FILEFORMAT, ALTERPARTITION_LOCATION, ALTERPARTITION_PROTECTMODE, ALTERPARTITION_SERDEPROPERTIES, ALTERPARTITION_SERIALIZER, ALTERTABLE_ADDCOLS, ALTERTABLE_ADDPARTS, ALTERTABLE_ARCHIVE,
ALTERTABLE_CLUSTER_SORT, ALTERTABLE_DROPPARTS, ALTERTABLE_FILEFORMAT, ALTERTABLE_LOCATION, ALTERTABLE_PROPERTIES, ALTERTABLE_PROTECTMODE, ALTERTABLE_RENAME, ALTERTABLE_RENAMECOL,
ALTERTABLE_RENAMEPART, ALTERTABLE_REPLACECOLS, ALTERTABLE_SERDEPROPERTIES, ALTERTABLE_SERIALIZER, ALTERTABLE_TOUCH, ALTERTABLE_UNARCHIVE, ALTERVIEW_PROPERTIES, CREATEDATABASE, CREATEFUNCTION,
CREATEINDEX, CREATEROLE, CREATETABLE_AS_SELECT, CREATETABLE, CREATEVIEW, DESCDATABASE, DESCFUNCTION, DESCTABLE, DROPDATABASE, DROPFUNCTION, DROPINDEX, DROPROLE, DROPTABLE, DROPVIEW, EXPLAIN, EXPORT,
GRANT_PRIVILEGE, GRANT_ROLE, IMPORT, LOAD, LOCKTABLE, MSCK, QUERY, REVOKE_PRIVILEGE, REVOKE_ROLE, SHOW_GRANT, SHOW_ROLE_GRANT, SHOW_TABLESTATUS, SHOW_TBLPROPERTIES, SHOWDATABASES, SHOWFUNCTIONS,
SHOWINDEXES, SHOWLOCKS, SHOWPARTITIONS, SHOWTABLES, SWITCHDATABASE, UNLOCKTABLE
See also Data Manipulation Language statements Not supported: "Shutdown" option for the queue full policy. |
| Hue | ADD_LDAP_GROUPS, ADD_LDAP_USERS, CREATE_GROUP, CREATE_USER, DELETE_GROUP, DELETE_USER, DOWNLOAD, EDIT_GROUP, EDIT_PERMISSION, EDIT_USER, EXPORT, NAVIGATOR_ADD_TAG, NAVIGATOR_DELETE_TAG, SYNC_LDAP_USERS_GROUPS, USER_LOGIN, USER_LOGOUT |
| Impala |
CREATE ROLE, DELETE, DROP ROLE, GRANT privilege, GRANT ROLE, INSERT, Query, REVOKE privilege, REVOKE ROLE, SHOW GRANT ROLE, SHOW ROLE GRANT, UPDATE, Hive DDL and DML Statements Support |
| Sentry |
ADD_ROLE_TO_GROUP, CREATE_ROLE, DELETE_ROLE_FROM_GROUP, DROP_ROLE, GRANT_PRIVILEGE, REVOKE_PRIVILEGE |
| Solr | add, commit, CREATE, CREATEALIAS, CREATESHARD, DELETE, DELETEALIAS, deleteById, deleteByQuery, DELETESHARD, finish, LIST, LOAD_ON_STARTUP, LOAD, MERGEINDEXES, PERSIST, PREPRECOVERY, query, RELOAD, RENAME, REQUESTAPPLYUPDATES, REQUESTRECOVERY, REQUESTSYNCSHARD, rollback, SPLIT, SPLITSHARD, STATUS, SWAP, SYNCSHARD, TRANSIENT, UNLOAD |
HDFS Audit Logging for ACL Operations
HDFS audit logging for ACL operations has some variation based on the command options.
| Command | Option | Audit Event |
|---|---|---|
| getfacl | — | getAclStatus |
| setfacl | --b | removeAcl |
| setfacl | --k | removeDefaultAcl |
| setfacl | --m | modifyAclEntries |
| setfacl | --x | removeAclEntries |
| setfacl | --set | setAcl |
There is a difference in audit logging behavior based on how the ACL operations are run:
- Over FileSystem ACL APIs, all setfacl and getfacl operations produce audit log events.
- Over FsShell (that is, hadoop fs or hdfs dfs command lines):
- All setfacl operations produce audit log events.
- getfacl operations produce audit log events only if the file has ACLs set.
That is, setfacl operations always produce audit log events and getfacl operations always produce audit log events when ACLs are set.
Hive DDL and DML Statements Support
The table below lists Hive DDL and DML statements and whether Cloudera Navigator supports the operation with metadata and lineage extraction. This list applies to Cloudera Navigator 2.12 (Cloudera Manager 5.13) and later releases.
| Hive operations | Cloudera Navigator Support | Comment |
|---|---|---|
| Abort |  |
Operation does not generate data flow lineage. |
| Alter Table/Partition/Column |  |
Known Issue. ALTER TABLE RENAME TO does not create query entity. ALTER TABLE CHANGE column name does not create query operation entity. |
| Create Table | |
|
| Create/Drop Macro | |
Operation does not generate data flow lineage. |
| Create/Drop/Alter Index | |
Operation does not generate data flow lineage. |
| Create/Drop/Alter View | |
Operation does not generate data flow lineage. |
| Create/Drop/Alter/Use Database | |
Operation does not generate data flow lineage. |
| Create/Drop/Grant/Revoke Roles and Privileges | |
Operation does not generate data flow lineage. |
| Create/Drop/Reload Function | |
Operation does not generate data flow lineage. |
| DELETE | |
Requires ACID support. Hive ACID not supported. |
| Describe | |
Operation does not generate data flow lineage. |
| Drop/Truncate Table | |
|
| EXPORT | |
Known Issue. External tables can be exported to HDFS but Cloudera Navigator does not create a query entity for the EXPORT. |
| IMPORT | |
Known Issue. External tables can be imported from HDFS but Cloudera Navigator does not create a query entity for the IMPORT. |
| INSERT data into Hive Tables from queries | |
|
| INSERT data into the file system from queries | |
|
| INSERT values into tables from SQL | |
|
| LOAD | |
Known Issue. LOADing a CSV from HDFS into an existing Hive table does not generate lineage. |
| MERGE | |
Requires ACID support. Hive ACID not supported. |
| MSCK REPAIR | |
Tables track their respective partitions. Queries to create or repair partitions using MSCK are not captured as query entities. |
| Show | |
Operation does not generate data flow lineage. |
| UPDATE | |
Requires ACID support. Hive ACID not supported. |
Navigator Metadata Server Sub Operations
| Operation | Sub Operation |
|---|---|
| auditReport | createAuditReport, deleteAuditReport, fetchAllReports, updateAuditReport |
| authorization | deleteGroup, fetchGroup, fetchRoles, searchGroup, updateRoles |
| metadata | fetchAllMetadata, fetchMetadata, updateMetadata |
| policy | createPolicy, deletePolicy, deletePolicySchedule, fetchAllPolicies, fetchPolicySchedule, updatePolicy, updatePolicySchedule |
| savedSearch | createSavedSearch, deleteSavedSearch, fetchAllSavedSearches, fetchSavedSearch, updateSavedSearch |