Cloudera Docs

Configuring Kafka brokers

Learn how you can update Kafka broker properties in your Kafka resource. Additionally, learn which broker properties are configurable and which are managed by Strimzi.

Updating broker configuration

You update broker configuration by editing your Kafka and KafkaNodePool resources.

You can update your Kafka and KafkaNodePool resource with kubectl edit. Which resource you update depends on what exact broker configurations you want to change.

Most broker configuration properties are specified in your Kafka resource. For example, properties like the default replication factor (default.replication.factor), minimum in sync replicas (min.insync.replicas), as well as many others. The KafkaNodePool resource contains configuration related to replicas, roles, and storage. Additionally, it can contain configuration related to CPU and memory resources, JVM options, as well as templates.

  1. Edit your resource. 
    kubectl edit [***RESOURCE***] --namespace [***NAMESPACE***]

    Running kubectl edit opens the resource manifest in an editor.

  2. Make your changes.
  3. Save the file.
Once the changes are saved, a rolling update is triggered and the brokers restart one after the other with the applied changes.

Configurable broker properties and exceptions

Learn which Kafka broker properties you can configure in the Kafka resource and which are managed by Strimzi.

Kafka broker properties are configured by adding them to spec.kafka.config in your Kafka resource. The values can be on of the following JSON types:
  • String
  • Number
  • Boolean

You can find full reference of the available broker properties in the Apache Kafka documentation. While all properties can be specified, some properties are managed by Strimzi. Broker properties managed by Strimzi generally cannot be configured, however, there are a few exceptions.

If spec.kafka.config contains a broker property that cannot be changed, it is disregarded, and a warning message is logged to the Strimzi Cluster Operator log. All other supported properties are forwarded to Kafka.

Properties managed by Strimzi

Strimzi takes care of configuring and managing options related to the following.

  • Security (encryption, authentication, and authorization)
  • Listener configuration
  • Broker ID configuration
  • Configuration of log data directories
  • Inter-broker communication
  • ZooKeeper connectivity

This means that the properties with the following prefixes cannot be set.

  • controller
  • cruise.control.metrics.reporter.bootstrap.
  • cruise.control.metrics.topic
  • host.name
  • inter.broker.listener.name
  • listener.
  • listeners.
  • log.dir
  • password.
  • port
  • process.roles
  • sasl.
  • security.
  • servers,node.id
  • ssl.
  • super.user
  • zookeeper.clientCnxnSocket
  • zookeeper.connect
  • zookeeper.set.acl
  • zookeeper.ssl

Exceptions

There are a few exceptions within the list of broker properties managed by Strimzi. These properties are forwarded to Kafka rather than being disregarded. The properties are as follows:

  • Any ssl configuration for supported TLS versions and cipher suites
  • Configuration for the zookeeper.connection.timeout.ms property to set the maximum time allowed for establishing a ZooKeeper connection.
  • The following Cruise Control metrics properties:
    • cruise.control.metrics.topic.num.partitions
    • cruise.control.metrics.topic.replication.factor
    • cruise.control.metrics.topic.retention.ms
    • cruise.control.metrics.topic.auto.create.retries
    • cruise.control.metrics.topic.auto.create.timeout.ms
    • cruise.control.metrics.topic.min.insync.replicas
  • The following controller properties:
    • controller.quorum.election.backoff.max.ms
    • controller.quorum.election.timeout.ms
    • controller.quorum.fetch.timeout.ms