Ranger RMS field issues - HDFS latency
Ranger RMS customers may experience intermittent high RPC queue and processing time.
Condition
When Apache Ranger (Ranger) Resource Mapping Server (RMS) is enabled, customers may intermittently encounter high Remote Procedure Call (RPC) queue time in the Hadoop Distributed File System (HDFS) NameNode, which results in jobs requiring more time than usual to finish. This is caused by the process of the Ranger HDFS plugin that needs to evaluate applicable Apache Hive (Hive) policies in addition to a set of HDFS policies for each HDFS location authorization. The evaluation process may cause access authorization latency of an additional 10-20 ms under heavy load, which in turn causes high NameNode RPC time.
- Components Affected:
- HDFS
- Products Affected:
- CDP Private Cloud Base
- Releases Affected:
- CDP Private Cloud Base 7.1.7 Service Pack (SP) 1 Cumulative Hotfix (CHF) 1 and higher versions
Cause
When Ranger RMS is enabled and a large number of access requests are made to the Ranger plugin in HDFS, it may take more than 10 ms to evaluate the access. If Ranger takes more than 10 ms to evaluate access in the middle of a write lock, all other calls will be blocked. This leads to slow response during the read/write operations on HDFS location, and increases the NameNode RPC queue time and processing time.
Remedy
As a workaround, add HDFS policies to authorize access for users and groups with heavy workloads, and configure RMS to skip chained plugin evaluation for these users and groups by performing the following steps:
