Launching spot instances on AWS from the Command Line

It is no secret that I love Amazon Web Services, and that we run all of our business on AWS. One of the things I fave the most is spot instances. The spot market is an auction market for unused resources. For literally a penny an hour you can fire up a server to try out something and shut it down when you are done. At a penny an hour you can run it for a long time for whatever reason you want. Pinterest uses this technique for their production load as well.

This weekend an r3.large which comes with 15GB of memory and SSD disks was costing 2 pennies an hour in the us-west-2c availability zone. So when I wanted to experiment around with few database related things it was a no brainer to fire one up on the spot market. Now, it takes all of a minute to zip through the screens on the AWS console to setup the spot request, and once fulfilled another minute or so to start up the machine and you are up and running. A side note, 20 years ago I was managing a super computer at UW-Madison. A grant from IBM for over a $1 million. That big ass machine had less memory than the instance I started up for 2 pennies. I was managing the software and another person was managing the hardware. It took up a large room of space with special cooling system. The fact that 20 years later I can fire up a machine with more processing power and more memory in 2 minutes for 2 pennies is incredible, a testament to Moore’s law and to AWS redefining the modern computing landscape.

At any rate, since I am trying out new things today; including firebase, but that’s a story for another post, I decided I want to fire up the instance from the command line and I want to script it so I can re-use it later.

I use the official aws cli for my command line needs. It works very well on my ubuntu instances as well as on my Macbook running OSX. So I figured I will look up the command line switches in couple of minutes and be on my way. Things did not turn out that well. The documentation for requesting a spot instance is lacking. They have couple of bland examples that did not really fit my needs. For one thing, if you run their example you will end up with 5 running instances that you cannot connect to remotely via ssh

aws ec2 request-spot-instances --spot-price "0.050" --instance-count 5 --type "one-time" --launch-specification "{"ImageId":"ami-a43909e1","InstanceType":

because they do not tell you to specify the Key. That’s fine if you do not need to connect to the machine and if the instance is going to bootstrap itself to do whatever function it needs. But how do you do that when you are not specifying user data or cloud-init data? Unless you had already baked the startup into your AMI. But then they do not tell you that, and that kindda defeats the whole purpose of me starting up an instance in two seconds for trying out stuff.

I want my instance to run in a VPC, register itself with Route53, to create and attach an EBS volume, to install Percona MySQL, install latest ubuntu patches, copy down databases from s3 and restore them locally, and have a public IP address. But it is not clear how i can configure what I need from the command line so that my startup scripts run successfully. I already have the shell scripts to do what I need on startup, but how do I pass it to my spot instance?

The documentation was not really helpful with that. I tried to look at the docs for ec2 run-instances which was more thorough but a lot of options did not translate well to the request-spot-instance command. I could not for example figure out how to pass security group by name. Could not figure out how to launch into a VPC and ask for a public IP Address at the same time, the cli complained that the options are mutually exclusive. The tedious format of the nested JSON classes separate by escaped double quotes on a single line was also error prune.

Eventually, through trial and error and using the options for a dry-run and for generate-cli-skeleton i figured most of it out. But I still needed to pass my user data to the instance, and preferably to tag the instance so i can have the amazing cli53 register it for me in route53 with no fuss. I could not find an option to tag the instance from the aws cli, so i ended up hard coding the tags in my user data and then tagging the instance at first boot, which is not ideal but worked.

But what about the user data? One option is to pass it as string on the command line. My user data script is too long to make this possible. As I said, it created and attaches volumes, downloads a bunch of software, restores databases from s3, yada yada… The ec2 run-instances command has an option to specify the user data as a file on the command line such as: file://my_userdata_script, but that did not work. The ec2 request-spot-instances command actually has a separate option to pass all the input json as a file. That worked but did not let me pass my user data in either.

So eventually, I ended up storing the user data in a separate file, base64 encoding it from my request spot instances script and passing it as a bash variable to the ec2 request-spot-instances script.

As explained earlier my user data scripts tags the instance but I did not solve the issue with public IP address. I ended up specifying it on the subnet setup in the VPC instead. If you launch into a default VPC that would be set for you any how. If you are launching into a non default VPC like I am then you need to switch the setting on the subnet to make the public IP address the default setting.

My final script looks like this, note that you will want to use your own account identifiers for the IAM roles, security groups, etc…

[code language=”bash”]
UserData=$(base64 < userdata-current)

aws ec2 request-spot-instances –region $region –spot-price $price –launch-specification "{ "KeyName": "$key", "ImageId": "ami-3d50120d" , "UserData": "$UserData", "InstanceType": "r3.large" , "Placement": {"AvailabilityZone": "$zone"}, "IamInstanceProfile": {"Arn": "$iamrole"}, "SecurityGroupIds": ["$sg"],"SubnetId": "$subnet" }"


And now my instance is happily busily restoring databases and running my tests scripts, for couple of pennies an hour.

Once you have the instance up and running there are few handy commands to manage things. To check the recent prices:

aws ec2 describe-spot-price-history –start-time $(date -j -u +”%Y%m%dT%H0000″) –product “Linux/UNIX (Amazon VPC)” –instance-type “r3.large” –availability-zone $zone

This will give you the spot prices as of the current hour.

To check on your request status, which you will need to do a lot initially till you figure out all the command switches as you will run into a lot of “failed due to bad parameters” cases:

aws ec2 describe-spot-instance-requests

Finally to cancel your request

aws ec2 cancel-spot-instance-requests –spot-instance-request-ids $requestId