Controlling memory usage

When you enable WriteBufferManager for a Flink application, the block-cache size becomes the single most important option in limiting the memory size. You can set it 256 mb as a starting value, and later increase it based on your production experience.

It is configured with the following option:
  • state.backend.rocksdb.block.cache-size: 256mb

The block cache serves as the in-memory cache for both reads and writes for the RocksDB instances. Flink creates an independent RocksDB instance for each stateful (keyed state) operator and subtask, so the minimum RocksDB native memory requirement is num_task_slots * num_stateful_ops * block_cache_size. The block size should be chosen depending on the size of the state and number of keys, but 256 MB is a good starting value.

If the read performance is insufficient, the block-size can be increased to 512 MB, 1 GB or even 2 GB on machines with large memory sizes.

Once you made the calculation, you need to ensure that the YARN container has enough extra memory on top of the TM heap size to cover the native memory requirement. This can be controlled by the following settings: