RocksDB state backend configuration
You can use RocksDB as a state backend when your Flink streaming application requires a larger state. With ClouderaConfigurableOptionsFactory, you are able to configure RocksDB for YARN applications to have a limit on memory usage supported by the Write Buffer Manager.
The default RocksDB state backend configuration available in Flink is not suitable for YARN
applications with large state, as it does not effectively limit the native memory usage of the
embedded RocksDB database. To effectively enforce a limit on memory usage, support for Write
Buffer Manager is introduced in the
ClouderaConfigurableOptionsFactorycan be added to the Flink configuration file to get a good initial configuration of the state backend:
state.backend: rocksdb state.backend.rocksdb.options-factory: org.apache.flink.contrib.streaming.state.ClouderaConfigurableOptionsFactory