Memory
The flag --memory_limit_hard_bytes
determines the maximum amount of memory that a Kudu tablet server may use. The amount of memory
used by a tablet server scales with data size, write workload, and read concurrency.
The following table provides numbers that can be used to compute a rough estimate of memory usage:
| Type | Multiplier | Description |
|---|---|---|
| Memory required per TB of data on disk | 1.5GB per 1TB data on disk | Amount of memory per unit of data on disk required for basic operation of the tablet server. |
| Hot Replicas' MemRowSets and DeltaMemStores | minimum 128MB per hot replica | Minimum amount of data to flush per MemRowSet flush. For most use cases, updates should be rare compared to inserts, so the DeltaMemStores should be very small. |
| Scans | 256KB per column per core for read-heavy tables | Amount of memory used by scanners, and which will be constantly needed for tables which are constantly read. |
| Block Cache |
Fixed by --block_cache_capacity_mb(default 512MB)
|
Amount of memory reserved for use by the block cache. |
Using this information for the example load gives the following breakdown of memory usage:
| Type | Amount |
|---|---|
| 8TB data on disk | 8TB * 1.5GB / 1TB = 12GB |
| 200 hot replicas | 200 * 128MB = 25.6GB |
| 1 40-column, frequently-scanned table | 40 * 40 * 256KB = 409.6MB |
| Block Cache |
--block_cache_capacity_mb=512 = 512MB
|
| Expected memory usage | 38.5GB |
| Recommended hard limit | 52GB |
Using this as a rough estimate of Kudu’s memory usage, select a memory limit so that the expected memory usage of Kudu is around 50-75% of the hard limit.
