Cloudera Docs

Quotas

Learn about Apache Kafka quotas, quota types, client groups, quota violation enforcement and detection, quota storage, as well as how you can configure quotas.

Kafka can enforce quotas on produce and fetch requests. Producers and consumers can use very high volumes of data. This can monopolize broker resources, cause network saturation, and generally deny service to other clients and the brokers themselves. Quotas protect against these issues and are important for large, multitenant clusters where a small set of clients using high volumes of data can degrade the user experience.

Quota types

There are two types of client quotas that can be enforced by Kafka brokers for each group of clients sharing a quota. These are as follows.

Network bandwidth quotas (byte-rate thresholds)
Network bandwidth quotas are defined as the byte rate threshold for each group of clients sharing a quota. Each group of clients can publish or fetch a maximum of X bytes/second per broker before clients are throttled.
Request rate quotas (CPU utilization thresholds)
Request rate quotas are defined as the percentage of time a client can utilize on request handler I/O threads and network threads of each broker within a quota window. A quota of n% represents n% of one thread, so the quota is out of a total capacity of ((num.io.threads + num.network.threads) * 100)%.
Each group of clients can use a total percentage of up to n% across all I/O and network threads in a quota window before being throttled. Since the number of threads allocated for I/O and network threads are typically based on the number of cores available on the broker host, request rate quotas represent the total percentage of CPU that each group of clients sharing the quota can use.

Client Groups

The identity of Kafka clients is the user principal, which represents an authenticated user in a secure cluster. In a cluster that supports unauthenticated clients, the user principal is a grouping of unauthenticated users chosen by the broker using a configurable PrincipalBuilder.

A client-id logically identifies an application making a request. A single client-id can span multiple producer and consumer instances. The tuple (user + client-id) defines a secure logical group of clients that share both user principal and client-id.

You can apply quotas to tuple (user + client-id), user, or client-id groups. For a given connection, the most specific quota matching the connection is applied. All connections of a quota group share the quota configured for the group. For example, if the tuple (user="test-user", client-id="test-client") has a produce quota of 10 MB/sec, then that quota is shared across all producer instances using the test-user user with the test-client client-id.

Quota violation detection

To detect quota violations quickly, byte-rate and thread utilization are measured over multiple small windows. For example, 30 windows that are each 1 second long. Having large measurement windows, for example, 10 windows that are 30 seconds each, leads to large bursts of traffic followed by long delays.

Quota violation enforcement

When a client exceeds its specified quota, the broker attempts to throttle the client instead of returning an error message. Throttling happens using the following logic.

  1. The broker calculates the amount of delay needed to bring a client under its quota.
  2. The broker sends a response to the client that includes the delay that the broker calculated. If the broker is responding to a fetch request, the response only contains the delay. The data that was requested is not included in the response.
  3. The broker mutes the channel to the client. Any additional requests that the broker receives from the client are only processed after the delay is over.
  4. If a client receives a response that includes a delay, the client refrains from sending further requests to the broker. This means that requests form a throttled client are blocked by both broker and client.

Quota storage and precedence

Quota configurations are stored in ZooKeeper. Specifically, user and tuple (user + client-id) quota configurations are written to ZooKeeper under /config/users, and client-id quota configurations are written under /config/clients.

The order of precedence for quota configuration is as follows.

  1. /config/users/[***USER***]/clients/[***CLIENT ID***]
  2. /config/users/[***USER***]/clients/<default>
  3. /config/users/[***USER***]
  4. /config/users/<default>/clients/[***CLIENT ID***]
  5. /config/users/<default>/clients/<default>
  6. /config/users/<default>
  7. /config/clients/[***CLIENT ID***]
  8. /config/clients/<default>

Configuring quotas

Learn how to configure Apache Kafka quotas.

By default, each client receives an unlimited quota. However, customizing quotas is possible. You can define quota configuration on the level of tuple (user + client-id), user, and client-id groups. In addition, there are two categories of quota configuration, default and custom.

For example, you can define a default quota configuration that applies to all clients, but also specify custom configurations that only apply to a specific subset of clients. Both default and custom configurations can be specified for any of the levels. Additionally, changes to quota configuration are dynamic and are effective immediately. This means that a broker restart is not required for the configuration to take effect.

Configuration is done with the kafka-configs tool using the --alter, --add-config, --entity-type, --entity-name, and --entity-default options.

The following collects various example commands that configure quotas as well as an example demonstrating how you can describe quotas.

Custom quota configuration examples
  • Configure a custom quota for a tuple (user + client-id)
    kafka-configs --bootstrap-server [***HOST***]:[***PORT***] --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name [***USER NAME***] --entity-type clients --entity-name [***CLIENT-ID***]
  • Configure a custom quota for a user
    kafka-configs --bootstrap-server [***HOST***]:[***PORT***] --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name [***USER NAME***]
  • Configure a custom quota for a client-id
    kafka-configs --bootstrap-server [***HOST***]:[***PORT***] --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type clients --entity-name [***CLIENT-ID***]
Default quota configuration examples

Notice that default quota configurations are set by specifying the --entity-default option instead of --entity-name.

  • Configure a default client-id quota for a user
    kafka-configs --bootstrap-server [***HOST***]:[***PORT***] --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name [***USER NAME***] --entity-type clients --entity-default
  • Configure a default quota for a user
    kafka-configs --bootstrap-server [***HOST***]:[***PORT***] --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-default
  • Configure a default quota for a client-id
    kafka-configs --bootstrap-server [***HOST***]:[***PORT***] --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type clients --entity-default
Describing quotas

In addition to configuring quotas using kafka-configs, you can also describe the quota configuration for any specific tuple (user + client-id), user, or client-id group. This is done by using the --describe option. For example, the quota configuration of a specific tuple can be retrieved with the following command.

kafka-configs --bootstrap-server [***HOST***]:[***PORT***] --describe  --entity-type users --entity-name [***USER NAME***] --entity-type clients --entity-name [***CLIENT-ID***]