From MozillaWiki
Jump to: navigation, search

Quick Links For You

About Us

We ship Firefox to the world!

Release Engineering does this by:

  • Developing world-class, scalable, open, secure, and maintainable infrastructure for application builds and tests, on a diverse set of desktop and mobile platforms
  • Building fast and reliable pipelines to transform source code into shipped products

Contacting Release Engineering

Our team is located on 3 continents and in 5 time zones. This allows us to provide better service, especially if you contact us using one of the following methods:

  • To request a new service, please file a bug
  • To ask a general question, ping in #releng on IRC, or email release@m.c
  • To request status on an existing request, set the "needinfo" flag in the bug.

Our team works normal business hours for our respective time zones. If you need to escalate an issue that is both urgent and important, use the following methods:

  1. Use #releng in IRC or email release@m.c, starting your message with 'URGENT:'. If no response in 15 minutes, then
  2. Be sure the bug severity is set to "blocker"
  3. Call one or more of the following people (nagios page preferred if you have access):

If you get voice mail, please leave your callback number.


If you are interested in getting involved with Release Engineering, please see our contributing page! We'd love to hear from you.


Mozilla's Release Engineering:

Name Profile Twitter Blog
Ben Hearsum bhearsum @bhearsum Ben's blog
Justin Wood Callek "Callek's Blog"
Chris AtLee catlee @chrisatlee chris' random ramblings: programming, photography, media, and anything else that strikes my fancy
Jordan Lund jlund ghost blog
Nick Thomas nthomas @nthomasftang F'tang F'tang Release wrangling at the bottom of the world
Rail Aliiev rail @railaliiev Rail's blog
Aki Sasaki aki @escapewindow escapewindow
Mihai Tabara mtabara @mihaitabara
Rok Garbas @garbas Rok's blog
Johan Lorenzo jlorenzo Johan's blog
Simon Fraser sfraser


Releng Offerings

As mentioned above, Releng offer pipelines to transform source code into shipped products. The following sections break down how that happens. Each section can be understood as both isolated and combined parts.

Initiating a Release

To start a release, Release Management submit a form in Ship It with what product they would like to release, what revision to use, and a number of other inputs.

Configuring & Scheduling

Ship It then creates Taskcluster tasks that are needed to take a builds and release them to users. These tasks make up the release graph that are derived from templates within taskgraph and are submitted via action tasks.


The signing of build artifacts and binaries. Some features of this:

  • signing is done through a client/server model
  • integrated with Taskcluster thanks to signingscript in conjunction with scriptworker, a securely managed Taskcluster worker type. signingscript tasks are part of the task graph that is generated from Ship It
  • keys are generated with modern standards
  • keys are secured by our infra and rotated regularly
  • coming soon: self served signing through exposed API


As important release artifacts are created, they are then uploaded to archive.m.o (FTP replacement) where they can served from and live permanently for historical or longstanding support reasons.

Like signing, uploading is:

Install & Update Tests

Like product tests, releases likewise need tests. This ensures that initial downloads install correctly on all supported platforms and updates work as expected, all before actually releasing. Right now this is done through our Update and Final Verify tests. Which, like all other sections are part of the release graph.

TODO: link to documentation and source of these

Shipping & Updates

Finally, we release the product via both initial downloads and as new updates.

For Desktop Firefox:

  • Initial installs are served via updating the download page, (Bedrock) and updating Bouncer, Mozilla's mirroring and redirection server.
  • Updates are handled by our own application, Balrog. Balrog is our server side component of the update system

For Android Products, including Firefox:


Best Practices



Release Engineering has historically been involved with the Build and Release pipeline of a product. As part of that, we have some responsibility to help ensure that the build (continuous integration) and release (deployment) pipeline infrastructure is in good running health. We have two functional groups that monitor and respond to any operational issues that come up.


Buildduty provides support for Firefox's build and test infrastructure

The role of Buildduty has grown beyond Release Engineering's scope. There is a dedicated team of 6 people, providing 24/7 support of Mozilla's build and test infrastructure.

See the Buildduty home page and manifesto for more information.


Releaseduty provides support for Firefox's release infrastructure and is currently handled by Releng

See the ReleaseEngineering/Releaseduty home page for more information.

Releng Applications

  • Applications - Various applications and services that RelEng provides
  • TODO update application list with: Services, Scriptworker, ChainOfTrust

Mozilla Applications

  • Taskcluster - our task execution tool for providing continuous integration and release pipelines
  • Treeherder - aggregates all the build, test, and release tasks as a reporting dashboard
  • PuppetAgain - Release Operations tool and puppet configs for configurating build and release machines. Both for workers and hosting applications
  • TODO - add Release Operations replacement for Slaveapi: Controller

External Third Party Dependencies

  • AWS
  • Heroku
  • Papertrail
  • Nagios
  • Sentry


below pages should be deleted or be given a "warning" outdated box where appropriate.


VCS Sync


Platform Support