Troubleshooting KRaft controller scaling

Checking KRaft Quorum state

If any of the add or remove commands are failing, the KRaft Quorum state can be checked to start the investigation using the following command:

kafka-metadata-quorum --bootstrap-controller example.com:9192 describe --status

Verifying that a controller joined the quorum

There are two ways to verify that a KRaft Controller successfully joined the quorum:

• Command success: If the Add KRaft Controller to Quorum command completes successfully, the controller has been added to the quorum.
• Health test: Cloudera Manager provides a health test that displays Concerning Health status if a KRaft Controller role is not part of the quorum. If the controller role shows healthy status, it is part of the quorum.

Handling dangling KRaft Controller roles

In some cases it is possible that a given KRaft Controller role is not actually a member of the KRaft Quorum despite never explicitly leaving it. For example, this can be the case when the metadata log directory was corrupted, removed, or recreated.

If there are dangling KRaft Controller roles present, meaning that a KRaft Controller role is active but not part of the Quorum, Cloudera Manager will change the health status of the role to Concerning Health.

To resolve the issue, the KRaft Controller role must be readded to the Quorum by completing a Remove KRaft Controller from Quorum → Add KRaft Controller to Quorum cycle.