Menu
Sign In to the Console
Amazon Elastic Compute Cloud
User Guide for Linux Instances

AWS Documentation » Amazon EC2 » User Guide for Linux Instances » Amazon EC2 Instances » Instance Purchasing Options » Spot Instances » Spot Instance Interruptions

Spot Instance Interruptions

Demand for Spot instances can vary significantly from moment to moment, and the availability of Spot instances can also vary significantly depending on how many unused EC2 instances are available. In addition, no matter how high you bid, it is still possible that your Spot instance will be interrupted. Therefore, you must ensure that your application is prepared for a Spot instance interruption. We strongly recommend that you do not use Spot instances for applications that can't be interrupted.

The following are the possible reasons that Amazon EC2 will interrupt your Spot instances:

  • Price—The Spot price is greater than your bid price.

  • Capacity—If there are not enough unused EC2 instances to meet the demand for Spot instances, Amazon EC2 interrupts Spot instances, starting with those instances with the lowest bid prices. If there are several Spot instances with the same bid price, the order in which the instances are interrupted is determined at random.

  • Constraints—If your request includes a constraint such as a launch group or an Availability Zone group, these Spot instances are terminated as a group when the constraint can no longer be met.

Interruption Behavior

You can specify whether Amazon EC2 should stop or terminate Spot instances when they are interrupted. The default is to terminate Spot instances when they are interrupted. You can change the behavior so that Amazon EC2 stops Spot instances when they are interrupted if the following requirements are met.

Requirements

  • For a Spot instance request, the type must be persistent, not one-time. You cannot specify a launch group or an Availability Zone group in the Spot instance request.

  • For a Spot fleet request, the type must be maintain, not request.

  • The root volume must be an EBS volume, not an instance store volume.

After a Spot instance is stopped by the Spot service, it can only be restarted by the Spot service, and only using the same launch configuration. When capacity is available that matches the Availability Zone and instance type of a Spot instance that is stopped, the Spot instance is started. With Spot fleet, if capacity is available only with a different Availability Zone or instance type, Spot fleet launches a new Spot instance using the launch configuration with available capacity.

While a Spot instance is stopped, you can modify some of its instance attributes, but not the instance type. If you detach or delete an EBS volume, it is not attached when the Spot instance is started. If you detach the root volume and the Spot service attempts to start the Spot instance, instance start fails and the Spot service terminates the stopped instance.

You can terminate a Spot instance while it is stopped. If you cancel a Spot request or a Spot fleet, the Spot service terminates any associated Spot instances that are stopped.

While a Spot instance is stopped, you are charged only for the EBS volumes, which are preserved. With Spot fleet, if you have many stopped instances, you can exceed the limit on the number of EBS volumes for your account.

Preparing for Interruptions

Here are some best practices to follow when you use Spot instances:

  • Choose a reasonable bid price. Your bid price should be high enough to make it likely that your request will be fulfilled, but not higher than you are willing to pay. This is important because if the supply is low for an extended period of time, the Spot price can remain high during that period because it is based on the highest bid prices. We strongly recommend against bidding above the price for On-Demand instances.

  • Ensure that your instance is ready to go as soon as the request is fulfilled by using an Amazon Machine Image (AMI) that contains the required software configuration. You can also use user data to run commands at start-up.

  • Store important data regularly in a place that won't be affected when the Spot instance terminates. For example, you can use Amazon S3, Amazon EBS, or DynamoDB.

  • Divide the work into small tasks (using a Grid, Hadoop, or queue-based architecture) or use checkpoints so that you can save your work frequently.

  • Use Spot instance interruption notices to monitor the status of your Spot instances.

  • While we make every effort to provide this warning as soon as possible, it is possible that your Spot instance will be terminated before the warning can be made available. Test your application to ensure that it handles an unexpected instance termination gracefully, even if you are testing for interruption notices. You can do so by running the application using an On-Demand instance and then terminating the On-Demand instance yourself.

Spot Instance Interruption Notices

The best way to protect against Spot instance interruption is to architect your application to be fault tolerant. In addition, you can take advantage of Spot instance interruption notices, which provide a two-minute warning before Amazon EC2 must interrupt your Spot instance. We recommend that you check for these warnings every 5 seconds.

This warning is made available to the applications on your Spot instance using an item in the instance metadata. You should use the instance-action item. The termination-time item is maintained for backward compatibility.

instance-action

If your Spot instance is marked to be stopped or terminated by the Spot service, the instance-action item is present in your instance metadata. You can retrieve it as follows. 

Copy
[ec2-user ~]$ curl http://169.254.169.254/latest/meta-data/spot/instance-action

The instance-action item specifies the action (stop or terminate) and the approximate time in UTC when the action will occur. The following example indicates the time that this instance will be stopped: 

{"action": "stop", "time": "2017-09-18T08:22:00"}

The following example indicates the time that this instance will be terminated:

{"action": "terminate", "time": "2017-09-18T08:22:00"}

termination-time

If your Spot instance is marked for termination by the Spot service, the termination-time item is present in your instance metadata. This item is maintained for backward compatibility. You can retrieve it as follows. 

Copy
[ec2-user ~]$ if curl -s http://169.254.169.254/latest/meta-data/spot/termination-time | grep -q .*T.*Z; then echo terminated; fi

The termination-time item specifies the approximate time in UTC when the instance will receive the shutdown signal. For example: 

2015-01-05T18:02:00Z

If Amazon EC2 is not preparing to terminate the instance, or if you terminated the Spot instance yourself, the termination-time item is either not present (so you receive an HTTP 404 error) or contains a value that is not a time value.

If Amazon EC2 fails to terminate the instance, the Spot bid status is set to fulfilled. Note that termination-time remains in the instance metadata with the original approximate time, which is now in the past.