Scaling down Kafka brokers

Learn how you can downscale Kafka brokers in a Streams Messaging cluster.

The Kafka brokers in a Streams Messaging cluster can be downscaled. During a downscale operation, broker nodes are decommissioned from the Broker host group. Data and partitions of the decommissioned nodes are automatically moved to other nodes.

There are two methods that can be used to downscale a cluster. You can either have Data Hub choose which nodes to decommission or manually specify which nodes should be decommissioned. The following steps walk you through both methods.

Ensure that you have reviewed Scaling Kafka broker nodes.
Ensure that the cluster, its hosts, and all its services are healthy.
Ensure that the Kafka brokers are commissioned and running.
Ensure that the Cruise Control service is running.
Using the Cruise Control REST API, verify that Cruise Control is reporting that all partitions are healthy.
important
An access level (admin, user or viewer) must be set for the user calling the API endpoint in Cruise Control. For more information, see Cruise Control REST API endpoints.
This can be done by calling the state API endpoint and ensuring the following in the response:
- numValidPartitions is equal to numTotalPartitions
- monitoringCoveragePct is 100.0
API call example:
```
https://[***MY-DATA-HUB-CLUSTER.COM***]/cdp-proxy-api/cruise-control/kafkacruisecontrol/state?json=true
```
You can find the API base URL on the Endpoints tab of the Cluster Details page. Management Console > Data Hub Clusters > [***YOUR CLUSTER***] > Endpoints.
Do not restart Kafka or Cruise Control during a downscale operation.
Do not create new partitions during a downscale operation.

Log in to the CDP web interface.
Go to Management Console > Data Hub Clusters.
Find and select the Data Hub cluster you want to scale.
Decommission nodes.
You can either let Data Hub choose which nodes to decommission or you can select the nodes that you want to have decommissioned.
- Let Data Hub choose which nodes to decommission
- Manually specify which nodes to decommission
Click Actions > Resize.

Select the Broker host group.

Set the desired size of the host group.
You can enter a number or use the plus and minus buttons to adjust the number of nodes.

Click Resize.
note
Data Hub chooses which nodes to decommission based on how suitable it is to remove a specific node. As a result, the nodes selected by Data Hub might not be the nodes that have the highest node ID.
Go to Hardware.

Scroll down to the Broker host group.

Select the nodes you want to remove.

Click the Delete (trash bin) icon.

Click Yes to confirm deletion.
Access the Cloudera Manager instance managing the cluster.
Restart all services that have stale configurations.

The cluster is downscaled. The number of Kafka broker nodes available in the Broker host group is equal to the configured number of nodes. Partitions are automatically moved from the decommissioned brokers. Once no load is left on the broker, the broker is fully decommissioned and removed from the Broker host group.