Scaling down Kafka Connect

Learn how you can downscale Kafka Connect in a Streams Messaging cluster.

Kafka Connect in a Streams Messaging cluster can be downscaled. The downscale operation differs depending on which Streams Messaging cluster template you are using. In the Light Duty template, Kafka Connect is downscaled by decommissioning nodes from the Broker host group. In the Heavy Duty template, Kafka Connect is downscaled by decommissioning nodes from the Connect host group.

There are two methods that can be used to downscale a cluster. You can either have Data Hub choose which nodes to decommission or manually specify which nodes should be decommissioned. The following steps walk you through both methods.

Ensure that you have reviewed Scaling Kafka Connect.
Ensure that the cluster, its hosts, and all its services are healthy.
Ensure that the Kafka brokers are commissioned and running.
Do not restart Kafka or Cruise Control during a downscale operation.
Do not create new partitions during a downscale operation.
If using the Light Duty cluster template ensure the following:
- Ensure that the Cruise Control Service is running.
- Using the Cruise Control REST API, verify that Cruise Control is reporting that all partitions are healthy.
  important
  An access level (admin, user or viewer) must be set for the user calling the API endpoint in Cruise Control. For more information, see Cruise Control REST API endpoints.
  This can be done by calling the state API endpoint and ensuring the following in the response:
  - numValidPartitions is equal to numTotalPartitions
  - monitoringCoveragePct is 100.0
  API call example:
```
https://[***MY-DATA-HUB-CLUSTER.COM***]/cdp-proxy-api/cruise-control/kafkacruisecontrol/state?json=true
```
  You can find the API base URL on the Endpoints tab of the Cluster Details page. Management Console > Data Hub Clusters > [***YOUR CLUSTER***] > Endpoints.
This is required because in the Light Duty template, Kafka Connect is deployed on the same node as Kafka brokers. While Kafka Connect does not require Cruise Control for scaling, Kafka brokers do.

Log in to the CDP web interface.
Go to Management Console > Data Hub Clusters.
Find and select the Data Hub cluster you want to scale.
Decommission nodes.
You can either let Data Hub choose which nodes to decommission or you can select the nodes that you want to have decommissioned.
- Let Data Hub choose which nodes to decommission
- Manually specify which nodes to decommission
Click Actions > Resize.

Select the host group you want to scale.
For Light Duty clusters, select the Broker host group. For Heavy Duty clusters, select the Connect host group.

Set the desired size of the host group.
You can enter a number or use the plus and minus buttons to adjust the number of nodes.

Click Resize.
note
Data Hub chooses which nodes to decommission based on how suitable it is to remove a specific node. As a result, the nodes selected by Data Hub might not be the nodes that have the highest node ID.
Go to Hardware.

Scroll down to the appropriate host group.
For Light Duty clusters, go to the Broker host group. For Heavy Duty clusters, go to the Connect host group.

Select the nodes you want to remove.

Click the Delete (trash bin) icon.

Click Yes to confirm deletion.
Access the Cloudera Manager instance managing the cluster.
Restart all services that have stale configurations.

The cluster is downscaled. In Light Duty clusters, the number of nodes available in the Broker host group is equal to the configured number of nodes. In Heavy Duty clusters, the number of nodes available in the Connect host group is equal to the configured number of nodes. Connectors and tasks assigned to the decommissioned nodes are redistributed among remaining connect workers