Auto-Attach EBS-volume to a New Spot Instance?

Question

I am experimenting with EC2 spot instances, and am needing some data to be retained between terminations. Now as I understand it, when the current price goes above my max. bid, it will be automatically terminated. I assume any init scripts I have will be run on shutdown so I can push data off to the EBS before unmounting.

My question is, how can I automatically mount the same EBS volume on the new spot instance once the price goes down, since it won't have any of my init scripts that I would've loaded onto the root volume the first time?

Do I have to create a custom AMI, or is there some other way to achieve this?

Solution 1

If you simply want the instance to start out with a fixed starting point each time it runs, then you could:

1. Create your own AMI and run that as the instance each time, or

2. Specify a user-data script that installs and configures software to your specifications each time a new instance is started from a standard base AMI.

If you need to keep state between instance runs, then you'll need to save the data somewhere outside of the instance/AMI. For example:

1. The data could be kept up to date on S3, SimpleDB, DynamoDB, RDS, etc.

2. You could designate a permanent EBS volume that the instance dynamically attaches and mounts at startup. This could be done with a user-data script.

Solution 2

Check out the ec2-spotter project on GitHub. It was created to solve the problem you describe.

Issues (#1 thru #4) described in @mcenzm's answer are real, and are handled correctly in ec2-spotter.

Solution 3

I have been trying this for a while now, and there are still a number of issues with EC2 spot instances and attached storage.

1. The volume you want to attach may not be in the same zone as the instance ? Zones cannot be specified at launch.
2. Internally attaching with the init script will work, but is a little asynchronous so you will need to test for it or sleep 10 say.

3. I cannot understand why the AWS console shows a "partition" as the primary block device. Maybe to deny us pirate windows images on another partition ? (sda1 instead of sda). Good luck finding the partition table.

   

4. Currently hard to attach using CLI from external point of control, which makes it hard to use snapshots for versioning.

5. So in general I back-up ...tar.bz2 to another box, and can then populate the "work area" upon startup. This is only useful for small volumes of data, so baselining a new ami with both volumes is a good idea. Spot instances are really good for "work unit" or "restartable" work anyway so the concept of fetching work from a server is well established. Given you are waiting 9 minutes in the extreme for a launch, you may not mind formatting your (as small as possible) disk.

   6. EBS is still pretty flaky and you need to pay extra for "optimised". It is still faster than trying to upload your completed work in 90s or so when the instance is "priced out".

   7. I suspect this will all change with any "elastic file" offering.

Comments

[Jeff]

Specifying a user-data script that installs sounds interesting; How could this be done when a spot request starts a new instance automatically? I already have a permanent EBS volume storing data, and will probably continue to use that

[Eric Hammond]

When you create the spot request, you specify user-data that should be passed in to each new spot instance.

[Jeff]

Ah I did not realize with cloud-init putting #! first in user-data executed a script. Thank you

[Eric Hammond]

@Jeff: Yeah, I invented that and popular AMIs have adopted it :-) alestic.com/2009/06/ec2-user-data-scripts

[Derek Morrison]

Along this line, is it possible to have a spot instance mount a particular EBS volume as its root disk when started?

[Eric Hammond]

@DerekMorrison: You might want to submit that as a new question on serverfault.com

[Derek Morrison]

@EricHammond Thanks for the suggestion. I've asked it here: serverfault.com/questions/565979/…

[Jeremy]

Dunno who downvoted this, but it means I can stop wasting time trying to code it! Love it.

[Morgan Christiansson]

1. You specify the subnet which is tied to an AZ zone. 2. Waiting until /dev/xvdx appears is easy. 3. How is this relevant?