Scaling down Kafka brokers
Learn how you can downscale Kafka brokers in a Streams Messaging cluster.
The Kafka brokers in a Streams Messaging cluster can be downscaled. During a downscale operation, broker nodes are decommissioned from the Broker host group. Data and partitions of the decommissioned nodes are automatically moved to other nodes.
There are two methods that can be used to downscale a cluster. You can either have Data Hub choose which nodes to decommission or manually specify which nodes should be decommissioned. The following steps walk you through both methods.
- Ensure that you have reviewed Scaling Kafka broker nodes.
- Ensure that the cluster, its hosts, and all its services are healthy.
- Ensure that the Kafka brokers are commissioned and running.
- Ensure that the Cruise Control service is running.
- Using the Cruise Control REST API, verify that Cruise Control is reporting that all
partitions are healthy. This can be done by calling the
stateAPI endpoint and ensuring the following in the response:
numValidPartitionsis equal to
You can find the API base URL on the Endpoints tab of the Cluster Details page. .
- Do not restart Kafka or Cruise Control during a downscale operation.
- Do not create new partitions during a downscale operation.
- Log in to the CDP web interface.
- Go to .
- Find and select the Data Hub cluster you want to scale.
- Decommission nodes.
You can either let Data Hub choose which nodes to decommission or you can select the nodes that you want to have decommissioned.
- Click .
- Select the Broker host group.
- Set the desired size of the host group.
You can enter a number or use the plus and minus buttons to adjust the number of nodes.
- Click Resize.
- Go to Hardware.
- Scroll down to the Broker host group.
- Select the nodes you want to remove.
- Click the Delete (trash bin) icon.
- Click Yes to confirm deletion.
- Access the Cloudera Manager instance managing the cluster.
- Restart all services that have stale configurations.