Scaling Kafka brokers
Learn how to scale the number of Kafka brokers in a CDP Private Cloud Base cluster.
The number of Kafka brokers provisioned in a CDP Private Cloud Base cluster can be scaled. Scaling is done by first configuring the Kafka and Cruise Control services and then adding or removing (decommission and delete) Kafka Broker role instances in the cluster. After broker roles are added or removed, Cruise Control automatically detects the change and rebalances partitions between brokers.
Configuring Kafka and Cruise Control for scaling
Learn how to configure Kafka and Cruise Control to be able to scale a Kafka cluster.
- For the Kafka service, you must ensure that Enable Kafka Decommission is selected.
- For the Cruise Control service, you must enable self healing and configure Cruise Control to use specific anomaly notifier and anomaly finder classes.
-
Configure Kafka.
- In Cloudera Manager, select the Kafka service.
- Go to Configuration.
- Find and select the Enable Kafka Decommission property.
-
Configure Cruise Control.
- Restart both the Kafka and Cruise Control service.
Scaling up Kafka brokers
Learn how you can upscale Kafka brokers in a Streams Messaging cluster.
The Kafka brokers in a Streams Messaging cluster can be upscaled by adding new Kafka broker instances to your cluster. The newly added Kafka broker is automatically detected by Cruise Control, a rebalance is triggered once they become available, and partitions are moved to the new broker.
- Ensure that the cluster, its hosts, and all its services are healthy.
- Ensure that the Kafka brokers are commissioned and running.
- Ensure that the Cruise Control service is running.
- Ensure that Kafka and Cruise Control is configured for scaling. For more information, see Configuring Kafka and Cruise Control for scaling.
- Ensure that one or more hosts are available that do not have a Kafka Broker role deployed on them. For more information on how to add a new host, see Adding a Host to a Cluster.
Scaling down Kafka brokers
Learn how you can downscale Kafka brokers in a Streams Messaging cluster.
The Kafka brokers in a Streams Messaging cluster can be downscaled by decommissioning one or more Kafka broker role instances. Data and partitions of the decommissioned broker are automatically moved to other Kafka brokers. The decommissioned broker instances can be deleted after the downscale process is finished.
- Ensure that the cluster, its hosts, and all its services are healthy.
- Ensure that the Kafka brokers are commissioned and running.
- Ensure that the Cruise Control service is running.
- Ensure that Kafka and Cruise Control is configured for scaling. For more information, see Configuring Kafka and Cruise Control for scaling.
- Do not restart Kafka or Cruise Control during a downscale operation.
- Do not create new partitions during a downscale operation.
- In Cloudera Manager, select the Kafka service.
- Go to Instances.
- Select the Kafka broker roles you want to decommission.
- Click Actions for Selected>Decommission.
- Review the list of brokers that will be decommissioned and click Decommission.
- Wait until the decommission process is finished and click Close.
- Restart the Kafka service.
- Go to .
- Select the instances that you want to delete.
- Click .