THESE NOTES ARE DEPRECATED AS OF 2016-10-28.

VIEW UPDATED PROCESS HERE: https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process

This is the process used to release system add-ons. This Wiki page was created based on this Google Document.

New System add-on planning and development phase

Initial Notification

Once a decision has been made to implement a new system add-on, dev owner will send an “Intent to Implement” email to release-drivers@mozilla.org and gofaster@mozilla.org and firefox-dev@mozilla.org mailing list.

• Reference (bug, repository, Wiki):
• Overview of the add-on:
• Timeline (including development and testing):

The responsible EPM will add this as a card to https://trello.com/b/moJCpVCD/go-faster-system-add-on-pipeline and tracked in the Go Faster checkins.

Implementation Complete

Devs are encouraged to give QA a heads-up a few weeks before the expected system add-on implementation complete date. This will help QA plan testing resources and staffing needs.

Testing system add-on

System add-on code and uplifts will go through testing via Nightly, Aurora and Beta channels. QA team will work with devs to create and review a test plan and test schedule.

Uplift process for GoFaster fixes

Beta Uplift criteria - Best practices

After Beta 6, we only uplift fixes for:

• Recent critical regressions, blocking bugs
• top crashers affecting browser stability and
• sec-high/crit vulnerabilities

Keeping in mind the general Beta uplift criteria, we recommend that GoFaster changes be planned such that:

• Bulk of new system add-on work needs to land in m-b before Beta6
• Before uplifting to Beta, the patches need to stabilize on Nightly, Aurora.
• All ground work needed to get the system add-on working should be code complete and uplifted to moz-beta before Beta6. Only fit-and-finish type fixes will be uplifted beyond Beta6 in Beta cycle
• A few minor uplifts can land post Beta6 if deemed critical.
• <Need to add a suggested timeline for localization related changes that need uplifting>

Process to Push updates to release channel

System add-ons updates that are pushed live on release channel should follow the steps below:

Verification on pre-release channel

Update must be tested on Beta/Try (by dev) and validated (by QA) before requesting push to release.

Notification to release-drivers@mozilla.org and gofaster@mozilla.org

“Intent to ship” tracking bug should link to all the bugs that were fixed as part of the system add-on update. This will be needed eventually for sec advisory pushes that happen through GoFaster.

Intent to ship email (see draft below) to <release-drivers> and <gofaster> ML and a corresponding bug to track. Plan to send this email 4-5 days before update-go-live date.

• uplift request made by dev (mozilla-approval-release = “?”)
• uplift approval made by RelMan (mozilla-approval-release = “+”)
• relnote-firefox tracking flag set by dev with suggested wording for relnote
• steps to push the change live

Multiple system add-on updates

For multiple system add-on updates that need to be pushed in parallel, we need to combine test efforts so these occur in parallel to save QA and ship time.

• We need one “intent-to-ship” tracking bug for each system add-on update.

Uplift request

Before landing the code, dev owner will request mozilla-release uplift. Uplift request will be reviewed by RelMan

Updating release notes to announce system add-on update

If approved, RelMan will create a corresponding release note with tag “UPDATE [datestamp-of-update-go-live]” in the main release notes and set to public after the update is pushed live.

Staged rollout

• Starting with staged rollout to 10% users, dev owner will push the change by landing the code in moz-release branch and balrog configuration change.
• If 24-48 hours of data indicates no regressions, the system add-on update will be pushed out to 100% end-users.
• Comms will publish blogs only after system add-on update is unthrottled and at 100%.

“Intent to Ship” email

• Bug Id: <bug #>
• System add-on: <add-on name> <add-on version>
• Committed to Moz-Release: <Yes/No, Commit Id if Yes>
• Verification: <Indicate fix was verified or new feature was signed-off>
• Other areas of impact: <describe any impact to stability/security/perf/mem-usage>
• Proposed date to go live: <go-live-date>
• Relnote suggested wording: <Wording suggested by dev in bug when nom’ing for relnote addition, also add a link to the code change/commit>

System add-on updates on pre-release vs. release channel

It is important to mention that updates on pre-release channel involve uplifting changes to mozilla-central/mozilla-aurora/mozilla-beta branch and the subsequent build contains the updated system add-on and no changes are pushed via balrog server.

If there are changes to the system add-on update process (update server like Balrog), we will test the update mechanism on Beta channel before pushing to release. Otherwise we can go straight to release channel and push directly after QA signs off on the system add-on updates.

For updates to release channel, the patch is uplifted to mozilla-release but the update is pushed to release channel via balrog.

Scheduling of system add-on updates on release channel

GoFaster team discussed the idea of releasing system add-on updates on pre-fixed days of the week. Instead of doing pre-fixed days, we will list out system add-on update blackout dates - these are the days when we should *not* plan to ship an update.

Blackout dates

• Mozilla-wide work weeks (June and December)
• Holidays: July 4th, December 25th, December 31st
• Week of release go-live

History of release version and system add-on updates pushed

Ian made a comment about maintaining a history of all releases and add-on versions pushed to the respective release. Balrog and release notes can be used to extrapolate this info.

We may need some tooling support to make this happen. TBD

Open Issues

• This Wiki seems to focus mostly on in-tree addons with possible uplift. But not necessarily addons that are specifically developed for old versions
• When will QA begin testing? When the add-on work is complete or when the add-on implementation code enters Aurora cycle?
• When do we add telemetry probes? This should be included in test plan discussion. Is there a way to do this for system add-ons?
• System Add-on A/B Testing
• Laura brought this up as a topic of discussion for a future meeting

Ideas for release process v2

Consider setting a release cadence for system add-ons if we start seeing a lot of pushes during the release cycle.

Some suggestions: Add-on updates (that are not critical) are pushed on Tuesday and Thursday of the week only. This builds more predictability in dev, QA and release efforts. Updates that fix critical issues can occur on a as-need basis.