Releasing Partner supported features in Firefox

This document outlines how we will co-produce and ship new features in Firefox based on partner functionality. It focuses on the technical process and toolchain and does not provide background on how Mozilla product teams choose features to ship.

The goals for this process are to provide:

• flexibility for partners
• this is meant as a template to start from, however projects consisting of code that will land into mozilla-central and ship as a part of our products will need to follow some standards outlined for the code repository
• verifiable change control
• update, a/b testing and beta channels for partner functionality
• clarity on change history related to Mozilla product (eg. ability to reproduce a build of Firefox for a given version)

Legal Requirements

All code contribution agreements will be covered within the Partner agreement, vetted by both Partner and Mozilla. The essence, from a code contribution perspective, is that Partner contributed code will be released under an open source license compatible with Mozilla products, and that they are allowing us to distribute their code to our users. Contributions will also have to go through normal Mozilla review processes [e.g. technical, security, privacy and QA] in order to be accepted into the core product. The actual details and agreements are handled by our Business and Legal teams.

Product features through system add-ons

System addons are our integration mechanism for Firefox features that are released with the standard Firefox distribution. System addons are essentially the same as any Add-on, however there may be some minor APIs available only to system addons to help provide a slightly tighter integration with Firefox.

Using add-ons as a mechanism for integrating new features into firefox allows us to utilize a number of features and systems making this integration easier.

• Test Pilot

 Used to distribute potential new features or upgrades to a group of users who have opted in to testing.

• AMO

 Add-ons can be distributed on a beta channel in AMO, allowing Partners to
 independently release new functionality to a subset of their users prior to
 landing in official Firefox releases.

• Firefox

 System add-ons are a part of Firefox.  Unlike typical add-ons, they are
 presented to users as a core feature and are not [currently] displayed in the
 Firefox addon manager.  Users will not be able to [typically] disable the
 system addons.

• Staged rollouts

 Occasionally it may be desirable to move a feature release faster than the
 typical Firefox release cycle.  This is an internal system that allows addon
 updates to be shipped in a stagged manor to release users.  A strict process
 is in place to ensure quality levels prior to rolling out to release users
 and will be described later in this document.

System Addons are also a part of the GoFaster project where Mozilla will push certain features into product at a rate faster than the typical product release cycle.

Partner addons will closely follow the GoFaster release process with possibly some minor modifications.

Releases fall into 3 categories:

• Updating the core Firefox distribution
• Releasing a beta version of the addon for testing purposes
• Updating an existing release of Firefox

Code Repositories

Mozilla maintains a set of partner related code repositories in Github. These repositories are linked into our build systems to produce Firefox distributions, partner Add-ons and more. Once Mozilla enters an agreement to produce products or features with a Partner, we'll create a private (or public depending on the project) code repository. The Partner will have read-only access to this repository, but may fork the repository into their own account. Updates provided back to Mozilla can be done through pull requests on the original repo.

Git supports tagging which will be used to tag both the Partner version and, if necessary, the Firefox version for each release. Git branches and/or forked repositories should be used for development work. Once the development cycle is complete, a pull request is made and code is reviewed prior to merging into the master branch. The master branch is then tagged with the new Partner version. Periodically the master branch will be merged into mozilla-central, and [git] tagged at that time with the Firefox version.

• master branch is always the latest release (beta or stable)
• work is done in forks and branches, developers should fork their own copy, or a team may choose to use one forked copy

Tagging of a new version

> git tag -a 1.0.1 -m "1.0.1 release"

# Mozilla merges into mozilla-central and will tag the Firefox release version.
# each version of firefox will use the latest tagged version in the repository.
# If Firefox 48.0.3 is being built, and the latest tag is 48.0, code at that
# tag will be used.

# Start development cycle for next version

> git branch 1.0.2-beta
> git checkout 1.0.2-beta

# branch is cloned to forked repositories for work
# changes are pulled back to the main repository via pull requests
# Addon should be uploaded to AMO for an AMO review
# once the AMO review is complete, create a pull request
# pull requests are reviewed by mozilla staff, once accepted are merged into
# the main github repository.
# when ready, branch is merged to master in the main repository

> git checkout master
> git merge 1.0.2-beta
> git tag -a 1.0.2 -m "1.0.2 release"
> git push origin 1.0.2
> git branch -d 1.0.2-beta 

Updating system addon in mozilla-central

The repository should be tagged prior to pulling into mozilla-central. It is important to note, for Partners who work within private repositories initially, at this stage the code will exist in our public repository.

> git show 1.0.1

commit 56e179790ccf8212a2e36b3261309453a21ff5c0
Author: somebody
Date:   ...date stamp...

> git tag -a fx-47.0 56e179
> git push origin fx-47.0

A Mozilla employee will handle moving the code into mozilla-central after tagging the repository. If necessary, code drops may be uplifted on the release train to address bugs are provide an earlier release schedule.

Issue Tracking

Mozilla uses Bugzilla for issue tracking. As much as possible, Partners should also utilize Bugzilla but may do so in combination with another issue tracker (e.g. github). Our Product releases track issues through Bugzilla therefore any issues that must be flagged and tracked for any release will need to be in Bugzilla. Our users also will often report bugs into Bugzilla. We will have a component created for any system addon, Partners should configure an account to watch for new issues reported against their addon. Issues tracking new feature development, etc. are more easily supported on external issue trackers if the Partner prefers.

Development and Review Process

It would be good to browse through the development resources at least to be familiar with what resources are on our wiki that may be useful. The code reviewprocess is good to be familiar with as well as understanding what we have in the reviewer checklist. In any case, we will be available to help you through the process.

Firefox Release schedule

Firefox is developed following a train model where each release will go through development (aka Nightly on mozilla-central), alpha (aka Aurora or Dev Edition), beta then release. Each phase is roughly a 6 week period though adjustments are made periodically for seasonal reasons (e.g. December holiday seasons). We maintain a handy Release Calendar you can refer to for a detailed and updated schedule.

It is expected that system addons will typically ride this train, requiring a full 18 week schedule prior to release. Under some agreements that schedule will be shortened to provide a quicker release cycle. Testing and review processes may be more stringent in this case, and is outlined under the GoFaster program. GoFaster Google doc

Test Pilot

[TBD]

AMO Release Channel

Add-ons released via AMO can be used as an alternative distribution channel. The addon should maintain the same versioning and addon id as what is distributed in the Firefox release. This will allow the addon to replace the builtin system addon. If the AMO version is removed, the builtin system addon will be reenabled. It is preferable that this is only used for releasing beta versions of the addon, or as a testing mechanism with a subset of users.

GoFaster Release Channel

The GoFaster program provides a process by which a system addon can be uplifted faster than the regular Firefox release schedule. This can be used to fast track new features and/or critical bug fixes. It is still preferable to let new features settle on the Firefox Beta channel prior to release. GoFaster is relatively new with only limited internal use at the time of this writing, therefore the process is subject to change as we refine the process.

Uplift to Beta

For non critical fixes we will require some stabilization time prior to uplift. This helps ensure a quality release for Firefox. Since changes have been reviewed via AMO and the github pull requests, only a cursory review if any should be necessary at this point. You should be familiar with the uplift criteria, though your partner engineering contact should help you changes get through the process.

Criteria:

• Changes must uplift prior to Beta 6
• Changes must stabilize on nightly and aurora prior to beta uplift
• All changes must be final prior to uplift
• Critical fixes may land during or after Beta 6, these will require additional scrutiny by the release management team.

Process:

• Create a new bug for the addon under the Firefox product using the General category. [NOTE: We will find a better product/component combination]
  • example subject: merge release of ABC system addon
  • example description:

   ABC is ready to release a new version of it's addon.  
   The partner repository is located at https://github.com/mozilla-partners/ABC .  
   Please land the updates into nightly using the git tag "1.0.2".

• • Include a summary of changes in the description
  • Create an attachment with a link to the diff on github. The attachment in bugzilla is necessary for uplift requests.
• Once the changes have landed into nightly (mozilla-central repository) let the changes sit for 2-5 days. Then you may request uplift to Aurora and Beta.
  • edit the attachment and add the approval‑mozilla‑aurora? and approval‑mozilla‑beta? flags (setting both to ?)
  • Fill in the form that has been added to the comment field.
  • Release engineers will see the flags and approve or ask additional questions.

Uplift to Release Channel

Uplifting changes to the release channel should be a last resort, reserved for critical fixes, unless the changes are otherwise part of a plan in coordination with Mozilla. A partner engineer will assist with the release uplift process.

Process:

• Changes must have already gone through the above uplift process and should be on the Beta channel.
• “Intent to ship” bug should be created in Bugzilla [TBD product/component for bug?]. The bug should link to all bugs, issues and patches related to the version of the addon that will ship to release.
• Partner Engineering will send an intent to ship email to the release drivers (see gdoc for details).
  • Uplift request made by Partner Engineering (mozilla-approval-release = “?”)
  • Uplift approval made by Release Management [RelMan] (mozilla-approval-release = “+”)
  • relnote-firefox tracking flag set by dev with suggested wording for relnote
• Dev and QA will test changes
• QA confirms changes on Beta channel
• RelMan schedules release
• Partner Engineering will push the change by landing the code in moz-release branch and balrog configuration change.
• The release will start with a throttled rollout to 10% of users.
• The release will push to 100% of users after 24-48 hours with no regressions.
• If necessary, Communications will publish information about the update after the update has been unthrottled.