Prerequisites for queue autoscaling (Technical Preview)
You must configure Load-based autoscaling in Data Hub before configuring queue
autoscaling. Load-based autoscaling scales the host group up or down based upon YARN’s
assessment of pending demand and available capacity. Load-based autoscaling can help control
costs while providing on-demand cluster capacity.
When you configure a load-based autoscaling policy, you choose a minimum and
maximum number of nodes for the host group. The policy does not scale below or beyond the
range of nodes that you define, regardless of the demand on the cluster. You also define a
cool down period, which is the number of minutes to wait before another autoscaling operation
is performed.
In the Data Hub Clusters homepage, click Create Data
Hub and select the desired environment and cluster definition. Define the
general cluster settings.
Under Auto Scaling, click the slider button to enable
autoscaling. Autoscaling is disabled by default.
Click Add Autoscale Policy and select
Load-Based, and then click Add.
Define the policy parameters.
Autoscale policy Parameters
Parameter Description
Name
Enter a unique name for the policy.
Host Group
Select the host group that you want to scale. The list of available host
groups is determined by which host groups include services that can be
scaled.
Target
Enter a minimum and maximum number of nodes for the policy. During any
autoscaling operation, the number of nodes do not exceed the predefined maximum
or drop below the predefined minimum.
Cooldown
After an autoscaling event occurs, enter the amount of time in minutes to
wait before performing another scaling operation. This means that the scaling
events during cooldown time are dropped.
Click Add. The policy appears under Auto Scaling
on the Provision Data Hub page.
