RocksDB state backend configuration

You can use RocksDB as a state backend when your Flink streaming application requires a larger state. With ClouderaConfigurableOptionsFactory, you are able to configure RocksDB for YARN applications to have a limit on memory usage supported by the Write Buffer Manager.

The default RocksDB state backend configuration available in Flink is not suitable for YARN applications with large state, as it does not effectively limit the native memory usage of the embedded RocksDB database. To effectively enforce a limit on memory usage, support for Write Buffer Manager is introduced in the ClouderaConfigurableOptionsFactory.

ClouderaConfigurableOptionsFactory can be added to the Flink configuration file to get a good initial configuration of the state backend:
state.backend: rocksdb
state.backend.rocksdb.options-factory: org.apache.flink.contrib.streaming.state.ClouderaConfigurableOptionsFactory

For the default RocksDB state backend configuration, see the Apache Flink documentation. See more information about the Write Buffer Manager here.