Using Spot Instances

To help you manage your cloud resource costs, Cloudera supports AWS Spot instances. Spot instances are Amazon EC2 instances that you can bid on. They run just like On-Demand instances, except that they are not provisioned until the instance price falls below your bid price. Amazon will terminate the instances when the instance price exceeds your bid price. Spot instances allow you to add capacity to your clusters at a low price.

Altus Director supports Spot instances for worker roles on the following services:
  • YARN
  • Hive on Spark
  • Hive on MapReduce
  • Spark
For all other components and for master roles, Cloudera recommends using on-demand instances.
Although Spot instances are supported with Altus Director 2.0 and higher, Cloudera recommends using Altus Director 2.5 and higher together with Cloudera Manager and CDH 5.12 and higher for Spot instances. Beginning with these versions, Cloudera has improved cluster resiliency to support Spot instances better. Now, if Amazon terminates a Spot instance during a bootstrap or update operation, Altus Director will shrink the terminated instance out of the cluster, as long as the affected instance group still meets the minimum count requirement. For this reason, Cloudera recommends specifying a minimum count of zero for instance groups that use Spot instances. (See Best Practices for Using Spot Instances below.)

For more information about using Spot instances, see the Amazon EC2 documentation. For help with bidding on Spot instances, see the Spot Bid Advisor.

Planning for Spot Instances

It is normal for Spot instances on a cluster to disappear over time. However, Cloudera Manager does not see that these instances are terminated. If you use Cloudera Manager to restart a cluster that contains a Spot instance group, and the Spot instances have terminated, the restart fails. If you are modifying any group in the cluster that has lost Spot instances, do not select the Restart checkbox.

If your bid price is so low that you do not obtain an instance when the group is created, you will have 0 instances in your group. If this happens, you can:
  • Delete the entire group.
  • Add more instances to the group.
  • Delete unprovisioned instances from the group (only as part of adding more instances to the group).
  • Retry (repair) existing instances.
You cannot do the following:
  • Change the bid price for spot instances.
  • Delete all instances without adding more, due to the minimum instance count requirement.

The bid price for Spot instances is set in an instance template. This template is associated with a group. Although you can modify the group, you cannot change the bid price. Therefore, if you set the bid price too low for successful provisioning, you must delete the group where that price is set and create a new group with the higher bid price. You must also delete the current group and create a new one if you want to drop the bid price.

Specifying Spot Instances

To specify Spot instances, create a new instance template and use this template for your instance group. In the Advanced Options section of the Create New Instance Template wizard, check the Use Spot Instances checkbox and enter a value in the Spot bid field.


Spot Blocks: Specifying a Duration for Spot Instances

The AWS Spot block feature enables you to specify a fixed duration ranging from one to six hours for Spot instances. Spot instances with a predefined duration use a fixed hourly price that remains in effect for the Spot instance while it runs. The price for instances with a Spot block will not be as low as that for ordinary Spot instances, but will be lower than that of on-demand instances, and Spot block instances are guaranteed to run for the specified duration, after which they are terminated.

To configure an instance template for Spot block instances, check the Use Spot Instances checkbox, enter a value in the Spot bid field, and choose a value from one to six hours (in intervals of 60 minutes) in the Spot Block Duration dropdown.


See Specifying a Duration for Your Spot Instances in the AWS documentation for more information about Spot blocks.

Best Practices for Using Spot Instances

  • Use a Spot instance worker group in conjunction with an On-Demand worker group. This ensures that the cluster can redo computational tasks run on Spot instances that could be terminated before the tasks are finished.
  • Use Spot instances only in contexts where the loss of the instance can be tolerated, as in a worker group. Do not use Spot instances for master nodes or for data storage.
  • Use a minimum count of 0 for Spot instance groups. If you use a number above 0, the cluster will likely enter a failed state.

Additional best practices for using spot instances can be found in Spot Instance Interruptions in the AWS documentation.