Introduction

Mozilla releases have historically been small, consisting of no more than 10-20 official files. With Firefox 1.0, we crossed a threshold that led us to managing 300+ files. With Firefox 1.0.1, we found ourselves pushing 450+ official files along with updates to all of our Firefox 1.0 clients. We need a better story for how we do releases, track them, and offer them to our users.

Mister is an integration piece between UMO, Bouncer, the build systems, QA, and the release process. It closes the gap between these services so we can push out new releases and updates in a timely and uniform manner for all of our users (across products, versions, platforms, and locales). It provides greater visibility of changes over time and offers an audit trail that we can use to provide a 10,000 foot view of what's happened over time to our code.

Our Current Story

is a bad one. Currently, builds are manually posted to stage.mozilla.org and the releases page is hand-edited to make releases available. The ftp staging server is home to 99 different user accounts, many belonging to people no longer involved with mozilla.org, and there are definite security risks and potential for damage if an unauthorized file were to be posted.

(fill in how tinderbox machines post builds, etc...)

The Story We Want

Mozilla Build Status

The Tinderbox front page is nice and was groundbreaking when introduced in 1998 but it's outdated and doesn't provide the full reach we need to offer the audit trail and functionality required for soothing our enormous build and release pains. Where each build system notifies the Tinderbox front page now, we would shift the Tinderbox recipient email address to feed into this system and offer the Tinderbox front page we know and love as one of the interfaces people can use to track builds and releases.

Manual Release File Registration

Mister should allow registration of new release files (given some token for auth and authz) to prepopulate the release data store. For example, a simple Perl script could parse a release directory and then submit some RPC to add new records for each file and group them together under a (product, version) tuple.

Automated Build Registration for Our Build Farm

Build systems should issue a notice to Mister when a new build is available. Similar to how a file registration script could be invoked from the command-line, Mister could receive a notice directly from the build systems that new files are available. Automating this build registration dovetails nicely with the holy grail of a "nightly update channel" for our early adopters and QA/testing community. Those who want to be on the bleeding edge could update to code that had just been compiled. If they find it too unstable, they could fall back to any of a hundred previous builds or any of twenty previous releases.

We should transmit this registration from our build systems to Mister via email. It works incredibly well for the Tinderbox systems (though with the occasional, generally tolerable hiccup, of course) and we should allow the same delayed delivery announcement from the build systems to the central release store.

Build History

Tracking builds as they are published would provide invaluable history to our teams and community members to audit when a release was made available and what notes were made on it when, by what people.

• Outstanding bugs could be linked from the notes to Bugzilla where comments could be easily tracked.
• Release notes could be generated by compiling the list of fixed bugs based on these notes.
• Previous build configurations could be reviewed and build logs stored in the system.

Greater Visibility of Changes

Tracking code changes is a difficult job and it behooves us to help ourselves by making tools which help us do this better. We need to offer this collection from a web-interface so that we can pass URLs around describing the total set of changes. We need to track not just changes we receive but changes we expect to receive so we can have a higher visibility of what fixes we require before we can close the code and begin BFTs, cut release candidates, and so forth.

...Between Builds

As builds are registered with the system, we can use Bonsai to track exactly the set of changes that vary a new build from the system's previous build.

...Between Releases

If the system tracks code changes all along (with the help of Bonsai), and release files are linked back to a specific build, we can easily compile an exact listing of changes that occurred to the code since some previous release.

Issuing Releases to Our Users

A lot of work goes into preparing the website and FTP site for a release, along with mirror coordination. Bouncer helps us by automating the job of monitoring the mirrors and providing accurate download links for files that we wish to serve to our clients. In an ideal world, Bouncer would be able to get its world-view of what files are available by peeking into the release data store and then truck over to the mirrors to see what's available where.

Making the Release Live with the Push of a Button

Though there's a lot of prep work prior to release, the act of making the release public should be the equivalent of flipping a switch or pushing a Big Button. Pushing the button should serve as a simple way to make the release available to different audiences. First to testers (who would be typing in a special download link to grab the release), then to our full audience.

Issuing Updates to Our Users

Where now the update RDF files are written and modified by hand (by me), we should move to a place where they are generated automatically upon request by the client. We can mitigate the hit we take generating these dynamically by placing a caching mechanism between this service and the client.

Making the Update Live with the Push of a Button

Making the update live for our users is a very complex topic. An update or patch may exist for all of our users, only one platform for one version, or only one locale on one platform for one version. We need to find a way to make the patch available for testers to verify it works as expected, and then to make the same patch available to our larger audience. The configuration of the update must change as little as possible from the time it is signed off on by QA to the time the release time hits the go button.

In case it's not obvious, this is to ensure that the act of making the update live itself does not break the configuration of the update that worked for the testers.

High-Coupling With Client Capabilities

The capabilities of the client should be encoded in Mister so that RDFs it generates work correctly with our deployed clients. We will require excellent communication between the server- and client-side engineers to ensure that we have a proper handle on what functionality we have available to us.

Localisations

When we have such a system in place, trusted localisers should be given access to customize the strings that are presented to the user. Creating this set of localised strings would be included in the required steps to stand-up any new localisation that a volunteer sought to bring online.

As the master en-US strings are changed, all localised strings should be flagged as needing translation to simplify the job of noticing when a locale requires an update.

Contact Information

Lead

• Owner, Mozilla Foundation Lead: Chase Phillips, chase on irc.m.o