Cloudera Docs

Scaling Kafka brokers

Learn how to scale the number of Kafka brokers in a CDP Private Cloud Base cluster.

The number of Kafka brokers provisioned in a CDP Private Cloud Base cluster can be scaled. Scaling is done by first configuring the Kafka and Cruise Control services and then adding or removing (decommission and delete) Kafka Broker role instances in the cluster. After broker roles are added or removed, Cruise Control automatically detects the change and rebalances partitions between brokers.

Configuring Kafka and Cruise Control for scaling

Learn how to configure Kafka and Cruise Control to be able to scale a Kafka cluster.

Before you can scale a Kafka cluster, configuring both the Kafka and Cruise Control service is required:
  • For the Kafka service, you must ensure that Enable Kafka Decommission is selected.
  • For the Cruise Control service, you must enable self healing and configure Cruise Control to use specific anomaly notifier and anomaly finder classes.
  1. Configure Kafka.
    1. In Cloudera Manager, select the Kafka service.
    2. Go to Configuration.
    3. Find and select the Enable Kafka Decommission property.
  2. Configure Cruise Control.
    1. In Cloudera Manager, select the Cruise Control service.
    2. Go to Configuration.
    3. Find and select the Self Healing Enabled property.
    4. Find the Anomaly Notifier Class property.
    5. Replace the currently configured class with the following: 
      com.linkedin.kafka.cruisecontrol.detector.notifier.SelfHealingNotifier
    6. Find the Metric Anomaly Finder Class property.
    7. Add the following class to the configuration: 
      com.cloudera.kafka.cruisecontrol.detector.EmptyBrokerAnomalyFinder
  3. Restart both the Kafka and Cruise Control service.
After configuring Kafka and Cruise Control, you can upscale or downscale the Kafka clusters.

Scaling up Kafka brokers

Learn how you can upscale Kafka brokers in a Streams Messaging cluster.

The Kafka brokers in a Streams Messaging cluster can be upscaled by adding new Kafka broker instances to your cluster. The newly added Kafka broker is automatically detected by Cruise Control, a rebalance is triggered once they become available, and partitions are moved to the new broker.

  • Ensure that the cluster, its hosts, and all its services are healthy.
  • Ensure that the Kafka brokers are commissioned and running.
  • Ensure that the Cruise Control service is running.
  • Ensure that Kafka and Cruise Control is configured for scaling. For more information, see Configuring Kafka and Cruise Control for scaling.
  • Ensure that one or more hosts are available that do not have a Kafka Broker role deployed on them. For more information on how to add a new host, see Adding a Host to a Cluster.
  1. In Cloudera Manager, select the Kafka service.
  2. Go to Instances.
  3. Click Add Role Instances.
  4. Click Select hosts>Custom… found under Kafka broker.
  5. Select one or more hosts and click OK.
  6. Click Continue.
  7. Optional: Review and configure the properties available on the Review Changes page based on your cluster and requirements.
  8. Click Finish.
  9. Select the newly added role instances.
  10. Click Actions for Selected>Start.
  11. Review the list of instances that will be started and click Start.
  12. Restart the Kafka service.
After the Kafka broker is added to your cluster, Cruise Control automatically detects it and starts the partition reassignment process.

Scaling down Kafka brokers

Learn how you can downscale Kafka brokers in a Streams Messaging cluster.

The Kafka brokers in a Streams Messaging cluster can be downscaled by decommissioning one or more Kafka broker role instances. Data and partitions of the decommissioned broker are automatically moved to other Kafka brokers. The decommissioned broker instances can be deleted after the downscale process is finished.

  • Ensure that the cluster, its hosts, and all its services are healthy.
  • Ensure that the Kafka brokers are commissioned and running.
  • Ensure that the Cruise Control service is running.
  • Ensure that Kafka and Cruise Control is configured for scaling. For more information, see Configuring Kafka and Cruise Control for scaling.
  • Do not restart Kafka or Cruise Control during a downscale operation.
  • Do not create new partitions during a downscale operation.
  1. In Cloudera Manager, select the Kafka service.
  2. Go to Instances.
  3. Select the Kafka broker roles you want to decommission.
  4. Click Actions for Selected>Decommission.
  5. Review the list of brokers that will be decommissioned and click Decommission.
  6. Wait until the decommission process is finished and click Close.
  7. Restart the Kafka service.
Cruise Control automatically decommissions the Kafka broker and reassigns the partitions to the other brokers.
After the broker is decommissioned, you can fully remove the role instance from the cluster by deleting it.
  1. Go to Kafka>Instances.
  2. Select the instances that you want to delete.
  3. Click Actions for Selected>Delete.