ReleaseEngineering

About Us

We ship Firefox to the world!

Release Engineering does this by:

• Developing world-class, scalable, open, secure, and maintainable infrastructure for application builds and tests, on a diverse set of desktop and mobile platforms
• Building fast and reliable pipelines to transform source code into shipped products

Contacting Release Engineering

Our team is located on 3 continents and in 5 time zones. This allows us to provide better service, especially if you contact us using one of the following methods:

• To request a new service, please file a bug
• To ask a general question, ping in #releng on IRC, or email release@m.c
• To request status on an existing request, set the "needinfo" flag in the bug.

Our team works normal business hours for our respective time zones. If you need to escalate an issue that is both urgent and important, use the following methods:

1. Use #releng in IRC or email release@m.c, starting your message with 'URGENT:'. If no response in 15 minutes, then
2. Be sure the bug severity is set to "blocker"
3. Call one or more of the following people (nagios page preferred if you have access):
   • Chris AtLee Eastern Time
   • Jordan Lund Pacific Time

If you get voice mail, please leave your callback number.

Contributing

If you are interested in getting involved with Release Engineering, please see our contributing page! We'd love to hear from you.

Team

Mozilla's Release Engineering:

• twitter: @MozRelEng
• blogs: Planet Release Engineering
• calendar: Google Calendar
• mailing list: release-engineering@lists.mozilla.org

Name	Profile	Twitter	Blog
Ben Hearsum	bhearsum	@bhearsum	Ben's blog
Justin Wood	Callek		"Callek's Blog"
Chris AtLee	catlee	@chrisatlee	chris' random ramblings: programming, photography, media, and anything else that strikes my fancy
Jordan Lund	jlund
Nick Thomas	nthomas	@nthomasftang	F'tang F'tang Release wrangling at the bottom of the world
Rail Aliiev	rail	@railaliiev	Rail's blog
Aki Sasaki	aki	@escapewindow	escapewindow
Mihai Tabara	mtabara	@mihaitabara
Rok Garbas		@garbas	Rok's blog
Johan Lorenzo	jlorenzo		Johan's blog
Simon Fraser	sfraser

Interns

• Interns

Releng Offerings

As mentioned above, Releng offer pipelines to transform source code into shipped products. The following sections break down how that happens. Each section can be understood as both isolated and combined parts.

Initiating a Release

To start a release, Release Management submit a form in Ship It with what product they would like to release, what revision to use, and a number of other inputs.

Configuring & Scheduling

Ship It then creates Taskcluster tasks that are needed to take a builds and release them to users. These tasks make up the release graph that are derived from templates within taskgraph and are submitted via action tasks.

Signing

The signing of build artifacts and binaries. Some features of this:

• signing is done through a client/server model
• integrated with Taskcluster thanks to signingscript in conjunction with scriptworker, a securely managed Taskcluster worker type. signingscript tasks are part of the task graph that is generated from Ship It
• keys are generated with modern standards
• keys are secured by our infra and rotated regularly
• coming soon: self served signing through exposed API

Uploading

As important release artifacts are created, they are then uploaded to archive.m.o (FTP replacement) where they can served from and live permanently for historical or longstanding support reasons.

Like signing, uploading is:

• integrated with Taskcluster thanks to beetmoverscript in conjunction with scriptworker, a securely managed Taskcluster worker type. signingscript tasks are part of the task graph that is generated from Ship It

Install & Update Tests

Like product tests, releases likewise need tests. This ensures that initial downloads install correctly on all supported platforms and updates work as expected, all before actually releasing. Right now this is done through our Update and Final Verify tests. Which, like all other sections are part of the release graph.

TODO: link to documentation and source of these

Shipping & Updates

Finally, we release the product via both initial downloads and as new updates.

For Desktop Firefox:

• Initial installs are served via updating the download page, mozilla.com (Bedrock) and updating Bouncer, Mozilla's mirroring and redirection server.
• Updates are handled by our own application, Balrog. Balrog is our server side component of the update system
  • this is integrated with Taskcluster thanks to balrogscript in conjunction with scriptworker, a securely managed Taskcluster worker type. balrogscript tasks are part of the task graph that is generated from Ship It

For Android Products, including Firefox:

• releases are submitted to the Play Store
  • this is integrated with Taskcluster thanks to pushapkscript in conjunction with scriptworker, a securely managed Taskcluster worker type. pushapkscript tasks are part of the task graph that is generated from Ship It

Development

Best Practices

• Development Best Practices - read this when you're starting something new
• Deployment - How we deploy our tools, and how do we update them.
• Wiki Guidelines - how to wiki garden for fun and profit

Bugzilla

• Flags
• Whiteboard
• Triage

Operations

Release Engineering has historically been involved with the Build and Release pipeline of a product. As part of that, we have some responsibility to help ensure that the build (continuous integration) and release (deployment) pipeline infrastructure is in good running health. We have two functional groups that monitor and respond to any operational issues that come up.

Buildduty

Buildduty provides support for Firefox's build and test infrastructure

The role of Buildduty has grown beyond Release Engineering's scope. There is a dedicated team of 6 people, providing 24/7 support of Mozilla's build and test infrastructure.

See the Buildduty home page and manifesto for more information.

Releaseduty

Releaseduty provides support for Firefox's release infrastructure and is currently handled by Releng

See the ReleaseEngineering/Releaseduty home page for more information.

Releng Applications

• Applications - Various applications and services that RelEng provides
• TODO update application list with: Services, Scriptworker, ChainOfTrust

Mozilla Applications

• Taskcluster - our task execution tool for providing continuous integration and release pipelines
• Treeherder - aggregates all the build, test, and release tasks as a reporting dashboard
• PuppetAgain - Release Operations tool and puppet configs for configurating build and release machines. Both for workers and hosting applications
• TODO - add Release Operations replacement for Slaveapi: Controller

External Third Party Dependencies

• AWS
• Heroku
• Papertrail
• Nagios
• Sentry

Deprecated

below pages should be deleted or be given a "warning" outdated box where appropriate.

Buildbot

• Master
  • Master Naming
  • Master Setup
  • Upgrading Buildbot
  • Preproduction
  • Masters (authentication required)
  • Queue directories
• Slave
  • Buildslave Versions
  • Buildslave Startup Process
  • Slave naming
• Development
  • Virtualenv - How to set up and use python virtual environments
  • Methods for testing your changes
  • Buildbot Best Practices
  • Landing Buildbot Master Changes
• References
  • Buildbot Database Schemas

VCS Sync

• HowTo
• History

Releases

• Release Checklist and Build Notes
• Releng Post Mortem Meetings
• What is releaseduty?
• Release Promotion
• Update Generation
  • Patcher tags for release updates

Platform Support

• Reference Platforms
• Slave Allocator
• Official Platform Support Checklist

Tryserver

• Tryserver As Branch
• TryServer
• TryChooser

Testing

• Graph Server Notes
• Intermittent Errors
• Requesting A Slave
• Testing a new Signing Server

Subpages

• Presentations
• How Tos
• Releng Docs outdated
• Special:PrefixIndex/ReleaseEngineering/ - all subpages of ReleaseEngineering in the wiki
• Buildbot Migration - outdated What builds are produced where? Buildbot, TaskCluster, or other?
• ReleaseEngineering/Obsolete Pages
• day 1 checklist
• day -1 checklist