Downscaling of resources
Resources are scaled up when the jobs are submitted. The downscale of all the autoscaled compute nodes happens only after all the submitted jobs are completed.
- Enable autoscale.
- Submit job-1 to occupy the whole cluster (auto scaled cluster - 3X). Additional compute nodes are added.
- Submit job-2 to occupy the original cluster size ( X - 100%).
This does not downscale all the autoscaled compute nodes because of job-2's shuffle data being present in N-nodes where containers are run.
- After the completion of job-2, downscaling of all the nodes that have been autoscaled in Step 2 is observed.