Prerequisites for queue autoscaling (Technical Preview)
You must configure Load-based autoscaling in Data Hub before configuring queue autoscaling. Load-based autoscaling scales the host group up or down based upon YARN’s assessment of pending demand and available capacity. Load-based autoscaling can help control costs while providing on-demand cluster capacity.
When you configure a load-based autoscaling policy, you choose a minimum and maximum number of nodes for the host group. The policy does not scale below or beyond the range of nodes that you define, regardless of the demand on the cluster. You also define a cool down period, which is the number of minutes to wait before another autoscaling operation is performed.
- In the Data Hub Clusters homepage, click Create Data Hub and select the desired environment and cluster definition. Define the general cluster settings.
- Under Auto Scaling, click the slider button to enable autoscaling. Autoscaling is disabled by default.
- Click Add Autoscale Policy and select Load-Based, and then click Add.
Define the policy parameters.
Autoscale policy Parameters Parameter Description Name Enter a unique name for the policy. Host Group Select the host group that you want to scale. The list of available host groups is determined by which host groups include services that can be scaled. Target Enter a minimum and maximum number of nodes for the policy. During any autoscaling operation, the number of nodes do not exceed the predefined maximum or drop below the predefined minimum. Cooldown After an autoscaling event occurs, enter the amount of time in minutes to wait before performing another scaling operation. This means that the scaling events during cooldown time are dropped.
- Click Add. The policy appears under Auto Scaling on the Provision Data Hub page.