Auto-tools/Projects/EC2Builder: Difference between revisions

EC2 is Amazon's cloud computing service. Most commonly used for site hosting, there is a separate service called Elastic Load Balancer that provides hooks for spawning new servers under load (CPU > x% would trigger new virtual server, CPU < x% would turn off virtual certain servers). Since we're using it to do builds, we thought about maybe using elastic load balancer (builds usually put the system at CPU close to 100%) but it seemed more effective to spin up 1 server per build and to queue excess jobs (limit active servers to some number, like 5, for cost reasons). Amazon provides a Queuing service as well.

When AWS first came out, EC2 only supported S3-backed instances. What this means is that EC2 instances have no state -- they simply take a frozen virtual image called an AMI, boot it up, and mount a "drive" that's really some memory allocated in Amazon's simple storage service (S3). Mounting the S3 drive can take about 5 minutes (in reality I think it's even longer than that) because there's movement of the image from S3 storage to EC2's local memory. Also there's a 10gb size limit (the max bucket size for S3). This is impractical for Windows under most circumstances from what I've read.

It wasn't clear which was better from a price perspective, but from a performance perspective EBS is clearly superior and AMI creation and everything related to that is significantly easier given that Amazon provides GUI tools and snapshot tools for AMI creation. As a result the AMIs that are prepared right now are EBS-backed and not S3-backed.

One server remains always-on (probably an in-house box of some sort). It contains the keypair necessary to access Amazon. A python script listens to mozilla pulse messages and simultaneously listens to requests on a socket port. A client program will send a changeset # to the socket server, and the server will in turn spin up an instance (using synchronization primitives to keep track of how many resources are in use), or it will queue the job if too many instances are already running. When a job completes, the server downloads the built binary from the EC2 instance, shuts down the instance (freeing the resource unless another queued job exists in which case it gets the job and executes it), and serves the binary via HTTP. It provides the download URL via a pulse message. The client program knows its build is complete when the pulse message comes through. Binaries older than 3 days are deleted via a cron job.