48
edits
Auto-tools/Projects/EC2Builder: Difference between revisions
no edit summary
No edit summary |
No edit summary |
||
| (9 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
= EC2 Build Server = | = EC2 Build Server == | ||
The EC2 Build server seeks to allow building mozilla-central's different changesets quickly with the help of Amazon's Elastic Cloud service. The tools that will use EC2's build server potentially include '''mozregression''' and '''mozremotebuilder'''<br><br> | The EC2 Build server seeks to allow building mozilla-central's different changesets quickly with the help of Amazon's Elastic Cloud service. The tools that will use EC2's build server potentially include '''mozregression''' and '''mozremotebuilder'''<br><br> | ||
| Line 13: | Line 13: | ||
EC2 Small Windows Instance / Windows Firefox Build: ??? hrs | EC2 Small Windows Instance / Windows Firefox Build: ??? hrs | ||
EC2 Medium High-CPU Instance / Linux Firefox Build (1 Core): 2 hrs | EC2 Medium High-CPU Instance / Linux Firefox Build (1 Core): 1.2 hrs | ||
EC2 Medium High-CPU Instance / Windows Firefox Build (1 Core): 2.75 hrs | EC2 Medium High-CPU Instance / Windows Firefox Build (1 Core): 2.75 hrs | ||
| Line 27: | Line 27: | ||
Medium High-CPU Linux: $0.17 / hr | Medium High-CPU Linux: $0.17 / hr | ||
Medium High-CPU Windows: $0.29 / hr | Medium High-CPU Windows: $0.29 / hr | ||
Worst-case (24-hour always-on) cost assuming limit of 5 instances running at once: $1162.35 / month | |||
Storage and bandwidth costs haven't yet been calculated. | |||
'''The cost of EC2 has been found to be untenable at this current time and this project has been discontinued. Please see mozremotebuilder for the alternative that is being worked on right now.''' | |||
== About EC2 == | == About EC2 == | ||
EC2 is Amazon's cloud computing service. Most commonly used for site hosting, there is a separate service called Elastic Load Balancer that provides hooks for spawning new servers under load <br> | EC2 is Amazon's cloud computing service. Most commonly used for site hosting, there is a separate service called Elastic Load Balancer that provides hooks for spawning new servers under load <br> | ||
E.g | E.g | ||
CPU > x% would trigger new virtual server | CPU > x% would trigger new virtual server | ||
CPU < x% would turn off virtual certain servers | CPU < x% would turn off virtual certain servers | ||
Since we're using it to do builds, we thought about maybe using elastic load balancer (builds usually put the system at CPU close to 100%) but it seemed more effective to spin up 1 server per build and to queue excess jobs (limit active servers to some number, like 5, for cost reasons). Amazon provides a Queuing service as well. <br> | Since we're using it to do builds, we thought about maybe using elastic load balancer (builds usually put the system at CPU close to 100%) but it seemed more effective to spin up 1 server per build and to queue excess jobs (limit active servers to some number, like 5, for cost reasons). Amazon provides a Queuing service as well. <br> | ||
== EBS-backed Instances vs S3-backed Instances == | == EBS-backed Instances vs S3-backed Instances == | ||
| Line 44: | Line 51: | ||
Relatively recently, EBS (Elastic Block Storage) has also become an option. Instead of mounting a drive stored in S3, you can store the AMI in elastic block storage. This allows faster boot up, as well as an option to "stop" an image and preserve its state prior to stopping. There is also no size limit (or well, it's such a high limit that we don't really care) to the drive. However it is more expensive because instead of just paying hourly running cost + storage cost, we pay additional storage cost. We do save a bit of money on the AMI storage: we're only charged for the memory that our additional modifications to the AMI have made, whereas an S3 instance is charged for storing the whole AMI. | Relatively recently, EBS (Elastic Block Storage) has also become an option. Instead of mounting a drive stored in S3, you can store the AMI in elastic block storage. This allows faster boot up, as well as an option to "stop" an image and preserve its state prior to stopping. There is also no size limit (or well, it's such a high limit that we don't really care) to the drive. However it is more expensive because instead of just paying hourly running cost + storage cost, we pay additional storage cost. We do save a bit of money on the AMI storage: we're only charged for the memory that our additional modifications to the AMI have made, whereas an S3 instance is charged for storing the whole AMI. | ||
It wasn't super clear how much of a difference this made from the price perspective, but from a performance perspective EBS is clearly superior and AMI creation and everything related to that is significantly easier given that Amazon provides GUI tools and snapshot tools for AMI creation. As a result the AMIs that are prepared right now are EBS-backed and not S3-backed. | It wasn't super clear how much of a difference this made from the price perspective, but from a performance perspective EBS is clearly superior and AMI creation and everything related to that is significantly easier given that Amazon provides GUI tools and snapshot tools for AMI creation. As a result the AMIs that are prepared right now are EBS-backed and not S3-backed. | ||
== Preliminary Design == | |||
One server remains always-on (probably an in-house box of some sort). It contains the keypair necessary to access Amazon. A python script listens to mozilla pulse messages and simultaneously listens to requests on a socket port. A client program will send a changeset # to the socket server, and the server will in turn spin up an instance (using synchronization primitives to keep track of how many resources are in use), or it will queue the job if too many instances are already running. When a job completes, the server downloads the built binary from the EC2 instance, shuts down the instance (freeing the resource unless another queued job exists in which case it gets the job and executes it), and serves the binary via HTTP. It provides the download URL via a pulse message. The client program knows its build is complete when the pulse message comes through. Binaries older than 3 days are deleted via a cron job. <br> | |||
http://i.imgur.com/fg0zN.gif | |||
== Side Project == | |||
Create a wrapper that pushes to mozillapulse | |||
== Other ideas == | |||
Create parameter that makes build servers clone try instead of moz-central and build try requests! | |||
Create | |||
== Proposed Usage == | == Proposed Usage == | ||