Cloudera Docs

ZooKeeper client error in Ranger RAZ

This document addresses a ZooKeeper client-side error encountered within the Apache Ranger RAZ (Ranger Authorization Service) environment. The issue arises from a buffer size mismatch between the ZooKeeper server and the Ranger RAZ client during large data transfers.

The Ranger RAZ client fails to process incoming or outgoing data packets from ZooKeeper, throwing the following exception in the application logs (or similar ZooKeeper/Curator Framework exceptions indicating a buffer overflow or length error):
[zkNettyNioEventLoopGroup-3-1]: Unexpected throwable
java.io.IOException: Packet len 1051047 is out of range!
at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:121) ~[zookeeper.jar:3.8.1.7.2.18.500-61]
at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientHandler.channelRead0(ClientCnxnSocketNetty.java:509) ~[zookeeper.jar:3.8.1.7.2.18.500-61]

The root cause is an insufficient client-side buffer size configured within the Apache Ranger RAZ client's ZooKeeper Curator framework.

  • Default limit mismatch: By default, Apache Ranger RAZ initializes its ZooKeeper client buffer (jute.maxbuffer) to 1MB. However, the ZooKeeper server ensemble in this environment is configured to handle up to 4MB packets.

  • Environment-specific payload: This issue is highly dependent on the specific environment's data volume (for example, highly complex security policies, large numbers of tokens, or extensive directory structures). Because the payload size fluctuates based on the environment, the issue could not be replicated in standard internal testing environments.

  • The failure mechanism: When Ranger RAZ attempts to read or write data packets exceeding 1MB, the underlying Apache Curator framework rejects the packet, resulting in an immediate connection or serialization failure.

To resolve this issue, the client-side buffer size must be increased to align precisely with the ZooKeeper server's maximum packet limit.

  1. Update the Ranger RAZ client configuration.

    Modify the Java options or configuration parameters for the Ranger RAZ client to pass the updated jute.maxbuffer property. Add or modify the following JVM argument in the Ranger RAZ startup script or environment configuration file:

    1. Go to the Ranger RAZ configuration page and search for Ranger RAZ Client Java Opts (ranger_raz_client_java_opts).
    2. Add the following to the configuration: 
      -Djute.maxbuffer=4194304
  2. Restart the Ranger RAZ service for the changes to take effect within the JVM.