Sentry Debugging and Failure Scenarios

This topic explains how Sentry deals with policy conflicts and how different CDH components respond when the Sentry service fails. It also shows you how to debug Sentry authorization request failures. Always first review the CDH Release Notes and the list of Known Issues for Sentry when you have issues.

Resolving Policy Conflicts

Sentry treats all policies independently. That means that for any operation, if Sentry finds an policy that allows it, that operation will be allowed. Consider an example with a table, test_db.test_tbl, whose HDFS directory is located at hdfs://user/hive/warehouse/test_db.db/test_tbl, and grant the following conflicting privileges to a user with the role, test_role. That is, you are granting ALL privilege to the role test_role on the URI, but only the SELECT privilege on the table itself. 
GRANT ALL ON URI 'hdfs:///user/hive/warehouse/test_db.db/test_tbl' to role test_role; 
USE test_db; 
GRANT SELECT ON TABLE test_tbl to role test_role;
With these privileges, all users with the role test_role will be able to carry out the EXPORT TABLE operation even though they should only have SELECT privileges on test_db.test_tbl: 
EXPORT TABLE <another-table-user-can-read> TO 'hdfs:///user/hive/warehouse/test_db.db/test_tbl'

Debugging Failed Sentry Authorization Requests

To figure out why Sentry has denied access in a specific instance, you can temporarily enable Sentry's debug level logging.

On systems managed by Cloudera manager, open the Hive service, go to the Configuration tab, and search for the HiveServer2 Logging Advanced Configuration Snippet (Safety Valve) parameter. Add the following lines to the parameter:

loggers = Sentry, lineage
logger.Sentry.name = org.apache.sentry
logger.Sentry.level=DEBUG

On systems not managed by Cloudera Manager, add log4j.logger.org.apache.sentry=DEBUG to the log4j.properties file on each host in the cluster, in the appropriate configuration directory for each service.

In the HiveServer2 logs, look for exceptions and messages that indicate the evaluations that Sentry makes. The logs below are an example of how debug logging can show why a request was denied. In this example, the user does not have the required privileges. 
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.binding.hive.authz.HiveAuthzBinding: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: Going to authorize statement QUERY for subject user3
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.binding.hive.authz.HiveAuthzBinding: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: requiredInputPrivileges = {Table=[SELECT], URI=[ALL], Column=[SELECT]}
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.binding.hive.authz.HiveAuthzBinding: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: inputHierarchyList = [[Server [name=server1], Database [name=db2], Table [name=t1], Column [name=fn]], [Server [name=server1], Database [name=db2], Table [name=t1], Column [name=ln]]]
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.provider.common.ResourceAuthorizationProvider: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: Authorization Request for Subject [name=user3] [Server [name=server1], Database [name=db2], Table [name=t1], Column [name=fn]] and [SELECT]
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.provider.common.ResourceAuthorizationProvider: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: requestPrivileges=[Server=server1->Db=db2->Table=t1->Column=fn->action=select->grantOption=false]
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.provider.common.ResourceAuthorizationProvider: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: PolicyEngine=org.apache.sentry.policy.engine.common.CommonPolicyEngine, PrivilegeFactory=org.apache.sentry.policy.engine.common.CommonPrivilegeFactory
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.provider.common.ResourceAuthorizationProvider: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: Get privileges for groups=[group3, othergroup, users], users=[user3], roleSet=ActiveRoleSet = [ roles = ALL ]
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.policy.engine.common.CommonPolicyEngine: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: Getting permissions for groups: [group3, othergroup, users], users: [user3]
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.policy.engine.common.CommonPolicyEngine: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: result = [server=server1->db=db1]
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.policy.common.CommonPrivilege: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: Create Privilege instance for privilegeStr=Server=server1->Db=db2->Table=t1->Column=fn->action=select->grantOption=false
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.policy.common.CommonPrivilege: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: Create Privilege instance for privilegeStr=server=server1->db=db1
2019-04-24 16:37:49,136 DEBUG org.apache.sentry.provider.common.ResourceAuthorizationProvider: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: ProviderPrivilege server=server1->db=db1, RequestPrivilege Server=server1->Db=db2->Table=t1->Column=fn->action=select->grantOption=false, RoleSet ActiveRoleSet = [ roles = ALL ] , Result false
2019-04-24 16:37:49,136 WARN  org.apache.sentry.binding.hive.conf.HiveAuthzConf: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: Using the deprecated config setting hive.sentry.failure.hooks instead of sentry.hive.failure.hooks
2019-04-24 16:37:49,137 INFO  org.apache.sentry.binding.hive.HiveAuthzBindingHook: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: No valid privileges
 Required privileges for this query: Server=server1->Db=db2->Table=t1->Column=fn->action=select->grantOption=false;
2019-04-24 16:37:49,137 ERROR org.apache.hadoop.hive.ql.Driver: [6d5e2fca-39aa-4557-90b5-3f9c84e6cd79 HiveServer2-Handler-Pool: Thread-80]: FAILED: SemanticException No valid privileges
 User user3 does not have privileges for QUERY
 The required privileges: Server=server1->Db=db2->Table=t1->Column=fn->action=select->grantOption=false;
org.apache.hadoop.hive.ql.parse.SemanticException: No valid privileges
 User user3 does not have privileges for QUERY
 The required privileges: Server=server1->Db=db2->Table=t1->Column=fn->action=select->grantOption=false;
        at org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:371)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:600)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1425)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1398)
        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:205)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
        at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:504)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:490)
        at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:295)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:507)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

Symptoms of Sentry Service Failure

If the Sentry service fails and you attempt to access the Hive warehouse, Hive, Impala and HDFS will behave as follows:
  • Hive: Queries to the Hive warehouse fail with an authentication error.
  • Impala: The Impala Catalog server caches Sentry privileges, so if Sentry goes down, Impala queries continue to work because they are authorized against this cached copy of the metadata. However, authorization DDLs such as CREATE ROLE or GRANT ROLE will fail.
  • HDFS/Sentry Synchronized Permissions: Affected HDFS files will continue to use a cached copy of the synchronized ACLs for a configurable period of time, after which they will fall back to the Hive System User and the Hive System Group (for example, hive:hive). The timeout value can be modified by adding the sentry.authorization-provider.cache-stale-threshold.ms parameter to the hdfs-site.xml Safety Valve in Cloudera Manager. The default timeout value is 60 seconds, but you can increase this value from several minutes to a few hours, as needed to accommodate large clusters.