ReleaseEngineering/How To/Manage spot AMIs

Problem

Something wrong with spot AMIs

Background

Spot AMIs are generated daily (almost) from scratch daily by aws_create_instance.py

The scripts uses base AMIs generated by aws_create_ami.py. These are generated manually and contain only base system. Particular base AMIs used to generate final spot AMIs are listed in the instance configs (e.g. bld-linux64).

The script takes a base AMI, puppetizes it, and cleans up some files. Spot AMIs use cloud-init to bootstrap their hostnames specified it instance specific user-data.

Troubleshooting

To find AMI a particular instance uses you can run the following:

ssh cltbld@instnace_ip curl http://169.254.169.254/latest/meta-data/ami-id

Verify that it matches latest AMIs in get_spot_amis.py output:

# example output
$ python scripts/get_spot_amis.py
bld-linux64, us-east-1: ami-7af30012
try-linux64, us-east-1: ami-208c7f48
tst-linux64, us-east-1: ami-fa8f7c92
tst-linux32, us-east-1: ami-188f7c70
bld-linux64, us-west-2: ami-3f46340f
try-linux64, us-west-2: ami-23473513
tst-linux64, us-west-2: ami-954032a5
tst-linux32, us-west-2: ami-e74032d7

Delete the AMIs using AWS Web Console.