Cloudera Docs

Downscaling of resources

Resources are scaled up when the jobs are submitted. The downscale of all the autoscaled compute nodes happens only after all the submitted jobs are completed.

The following example scenario explains the autoscaling and downscaling process:
  1. Enable autoscale.
  2. Submit job-1 to occupy the whole cluster (auto scaled cluster - 3X). Additional compute nodes are added.
  3. Submit job-2 to occupy the original cluster size ( X - 100%).
  4. Kill job-1.

    This does not downscale all the autoscaled compute nodes because of job-2's shuffle data being present in N-nodes where containers are run.

  5. After the completion of job-2, downscaling of all the nodes that have been autoscaled in Step 2 is observed.