148
edits
ReleaseEngineering/How To/Work with Golden AMIs: Difference between revisions
Jump to navigation
Jump to search
ReleaseEngineering/How To/Work with Golden AMIs (view source)
Revision as of 14:22, 17 November 2017
, 17 November 2017Updated some links and provided some examples.
ChrisCooper (talk | contribs) No edit summary |
(Updated some links and provided some examples.) |
||
| Line 27: | Line 27: | ||
== How it works == | == How it works == | ||
=== Base AMI === | === Base AMI === | ||
Base AMIs are created by [ | Base AMIs are created by [https://github.com/mozilla-releng/build-cloud-tools/blob/master/cloudtools/scripts/aws_create_ami.py create_ami.py] script manually and published in the corresponding config files, e.g. [https://github.com/mozilla-releng/build-cloud-tools/blob/master/configs/bld-linux64#L6 configs/bld-linux64]. These AMIs are used as a base image for “golden” AMIs. | ||
=== Golden AMI === | === Golden AMI === | ||
“Golden” AMIs are generated on aws- | “Golden” AMIs are generated on aws-manager2 by multiple daily cronjobs, e.g. [http://hg.mozilla.org/build/puppet/file/a09b4e59ae96/modules/aws_manager/manifests/cron.pp#l79 modules/aws_manager/manifests/cron.pp] | ||
Instances used to generate “golden” AMIs use DNS to make puppet work and rely on proper DNS entries. To reduce possibility of IP collisions they live in a different subnet and require --ignore-subnet-check as a paramter. | Instances used to generate “golden” AMIs use DNS to make puppet work and rely on proper DNS entries. To reduce possibility of IP collisions they live in a different subnet and require --ignore-subnet-check as a paramter. | ||
| Line 36: | Line 36: | ||
If the process is stuck for some reason you need to terminate the instances and kill the process. | If the process is stuck for some reason you need to terminate the instances and kill the process. | ||
The generated AMIs are published to [https://s3.amazonaws.com/mozilla-releng-amis/amis.json https://s3.amazonaws.com/mozilla-releng-amis/amis.json] by [ | The generated AMIs are published to [https://s3.amazonaws.com/mozilla-releng-amis/amis.json https://s3.amazonaws.com/mozilla-releng-amis/amis.json] by [https://github.com/mozilla-releng/build-cloud-tools/blob/master/cloudtools/scripts/aws_publish_amis.py scripts/aws_publish_amis.py] scheduled by [http://hg.mozilla.org/build/puppet/file/a09b4e59ae96/modules/aws_manager/manifests/cron.pp#l65 modules/aws_manager/manifests/cron.pp]. The published file is used by running spot instances to determine available AMIs and if they are need to be terminated if the AMI is out of date. | ||
Once a day [ | Once a day [https://github.com/mozilla-releng/build-cloud-tools/blob/master/cloudtools/scripts/delete_old_spot_amis.py scripts/delete_old_spot_amis.py] scheduled at [http://hg.mozilla.org/build/puppet/file/a09b4e59ae96/modules/aws_manager/manifests/cron.pp#l71 modules/aws_manager/manifests/cron.pp#l71] deletes old AMIs leaving latest 10 AMIs. | ||
== Troubleshooting == | == Troubleshooting == | ||
| Line 46: | Line 46: | ||
=== How to determine the IP of a spot instance which is not in DNS === | === How to determine the IP of a spot instance which is not in DNS === | ||
See [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves]. Running the script with “status” sub-command shows the IP used by instances. | See [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves]. Running the script with “status” sub-command shows the IP used by instances. | ||
<pre> | |||
python aws_manage_instances.py status b-2008-spot-006 | |||
2017-11-17 06:15:42,273 - INFO - Found b-2008-spot-006 (i-04ad9421626d050de)... | |||
Name: b-2008-spot-006 | |||
ID: i-04ad9421626d050de | |||
IP: 10.134.55.105 | |||
Enabled: True | |||
State: running | |||
Tags: Name -> b-2008-spot-006, moz-type -> b-2008, FQDN -> b-2008-spot-006.build.releng.use1.mozilla.com, moz-state -> ready | |||
</pre> | |||
=== Something is happening to the spot instances === | === Something is happening to the spot instances === | ||
| Line 52: | Line 62: | ||
=== How to force AMI generation === | === How to force AMI generation === | ||
There are cronjobs on aws- | There are cronjobs on aws-manager2 responsible for this. Running the underlying scripts would regenerate “golden” AMIs and copy then to other regions. | ||
The cronjobs can be listed by running | The cronjobs can be listed by running | ||
ls /etc/cron.d/*golden* | ls -lt /etc/cron.d/*golden* | ||
The underlying wrapper scripts live in | The underlying wrapper scripts live in | ||
| Line 61: | Line 71: | ||
To regenerate AMIs one should run those scripts as “buildduty” user. Consider using screen/tmux because it may take up to 2 hours to generate some AMIs. | To regenerate AMIs one should run those scripts as “buildduty” user. Consider using screen/tmux because it may take up to 2 hours to generate some AMIs. | ||
<pre> | |||
e.g. screen -S "bld-linux64-ec2-golden" | |||
</pre> | |||