Compatibility/System Addon/Release Process

Interventions Releases

Module Owner: Dennis Schubert

Generally speaking, new interventions are landed in the Firefox repo as soon as they're written, and then they just ride the trains.

In cases that are more urgent, we can speed up the process by

• uplifting interventions into Beta at any time during the cycle,
• uplifting the interventions into Release to be shipped with the next dot-release,
• shipping an interventions add-on update out-of-band.

It is up to each release's Interventions Owner to decide if a WebCompat bug is important enough to warrant a fast process.

The normal process: Letting interventions ride the train

For low-priority interventions, the simplest way to ship is to land them in the tree and have them ride the trains like any code change.

1. Build and test the interventions as usual. Make sure to bump the version number in manifest.json accordingly somewhere in your patch stack.
2. Submit the patch to Phabricator, the webcompat-reviewers group will be added automatically. Land the stack whenever the reviews are done.
3. Make sure that for any intervention you added, you set the sitepatch-applied label GitHub or the [webcompat:sitepatch-applied] whiteboard flag on Bugzilla. Likewise, remove those labels for all interventions you removed.

Your work is done here!

Uplifting new interventions into Beta or Release

For now, just request uplifts via the Phabricator-based process. Make sure the uplift includes a version number change, because you might run into issues otherwise. In the future, we will explore always uplifting the entire addon code to the trains, but this is a future ToDO.

Shipping interventions with an out-of-band System Addon Update

This process will involve: two WebCompat engineers, one Release Engineer, one Release Manager, and however many people it takes to run tests.

Prerequisites

Before you can do any of the work outlined here, you need to make sure that your Mozilla LDAP account is a member of the vpn_cloudops_shipit and xpi_system_build groups. If you are not sure, check your LDAP profile here (Mozilla-internal). If one or both groups are missing, file a bug similar to bug 1683143 (Mozilla-internal).

You also need to be able to connect to the Mozilla Corporate VPN to use ShipIt. See this guide (Mozilla-internal) if you're currently not set up for that.

Preparations

As opposed to cherry-picking individual commits or crafting a custom build of the addon, we're always shipping the Interventions addon based on whatever is in the current firefox repo - that is, everything that's currently included in Nightly. This reduces the risks around crafting a wrong patch, missing a file, etc - and we can easily verify the interventions by using a Nightly build.

• Your interventions need to be landed in the firefox repo, autoland is not enough. If you're working on a tight schedule, ask a sheriff, who should be able to land your code quicker.
• File a tracking bug like this one as soon as you can. Use this to give a heads-up to the involved Release Owner, as well as the on-duty Release Engineer. If you need QA resources, give them a heads-up, too.

Kicking off the build process

1. Make sure you're connected to the Mozilla Corp VPN, specifically the US-west endpoint.
2. Sign into Ship-It. In the header menu, go to Releases > Extensions > New.
3. From the loaded list, select the webcompat extension. Afterwards, select a revision. The current head commit is fine, just make sure it actually includes your interventions. Click the "create release" button and submit the release.
4. When the release has been submitted and you click the "close" button, it will send ton the list of all active XPI releases. Search for the right webcompat addon release, click on "Build", and click "schedule".
5. When that's done, the "Build" will turn into "Build task", which is a clickable link to the job in Taskcluster. It might take a few moments before the link works, but it should work eventually.

Testing the dep-signed .xpi

1. Wait until all the tasks in the previously started build job are done. It can take 10-15 minutes, so this is your chance to relax and stretch.
2. When all jobs are done, open the dep-signing-webcompat task and copy the URL to the public/build/webcompat.xpi artifact. Ideally, post that for reference into the tracking bug.
3. Download an unbranded release build - you can find instructions on how to get it here.
   • Note: if you are on macOS, you might get a "Nightly is damaged and can’t be opened" error. If so, you can do what mozregression is doing and sign the unbranded build yourself. You can do that with codesign -v --force --deep --sign - Nightly.app && xattr -r -d com.apple.quarantine ./Nightly.app.
4. Launch that unbranded build with a blank profile. Head to about:config, create a new bool pref named xpinstall.signatures.dev-root and set it to true.
5. Then, open the link to the dep-signed .xpi from the build step earlier. Firefox should install it happily.

You can now test the update. Check the version number on about:support, test that about:compat works, test the intervention. If everything looks fine, you can proceed with

Shipping sign-offs, testing the real .xpi

To get a "real release-signed .xpi", you need to:

1. Head back to Ship-It. Click the "Ship" "button", which will open a popup asking for a sign-off. It's fine if you, as the person writing the interventions, do one of the sign-offs. Do so.
2. Get another WebCompat team member to be the second-sign off. They should be familiar with the intervention, and also have done some quick testing with the dep-signed .xpi.
3. When the second sign-off is there, Ship-It will automatically start the "shipping" phase. Don't worry, it's not actually shipping anything to users yet - it's just building a shippable .xpi. Click the "Ship task" link, which will again bring you to Taskcluster.
4. After you got your sign-off and the shipping phase in Ship-It was scheduled, you will no longer see the build in the "Pending Releases" view. That's expected. In the main menu, head to Releases > Extensions > Recent.
5. If all tasks aren't done yet, wait. When they're done, open the release-signing-webcompat task, and copy the link to the public/build/webcompat.xpi artifact. Be careful with this link, as it is a real release-signed .xpi that any Firefox will just install. Do not publicly share this link.
6. Start a regular release Firefox, with a new profile. Then open the Link to the artifact. Firefox will install it happily. Run the same tests again, just to make sure.

If you're satisfied with the testing results, proceed with

Getting the update on a Balrog staging channel, and final testing

Now, you need to involve Release Engineering.

1. Find out which RelEng is on duty - you can check the subject of the releaseduty Matrix room.
2. Ask them to add Balrog rules for the webcompat release on the staging channel. The staging channels have a -sysaddon suffix, so depending on if you're testing Release or Beta, the channel names would be release-sysaddon and beta-sysaddon respectively.

When they have staged the update, you can test the whole process.

1. Post the testing instructions on Bugzilla. You can use this comment as a template. A few notes:
   • The value for extensions.systemAddon.update.url is the original value of that pref, except that the %CHANNEL% variable is replaced with whatever staging channel you're targetting.
   • The Browser Console command is not necessarily needed - but since the background update check happens in the background and only once per day, it's highly advisable to use it. It will trigger the exact same update step that the background task would do.

Be very careful with testing, this is your final chance to catch issues. This is also the stage where you should be involving QA, or any other internal users who might help validate the fix. Make sure that all testing steps and all testing results are documented in the Bugzilla bug.

Shipping it.

When all testing is completed and nothing looks concerning, you can finally press the big button - metaphorically speaking.

1. Write a Bugzilla bug to explicitly document which channels and versions you want to target.
   • Balrog targets both channels and versions. So, when talking to Release Engineering, you need to be exact about which channels and versions you target. For example, if you want to target the current Release and the current Beta version, be sure to specify something like "we want to target release for 145.*, as well as beta and aurora for 146.*.
   • Firefox Developer Edition is technically Beta, but it's not using the beta channel. That is aurora, so make sure to include it if needed.
   • Firefox will happily downgrade (or rather, shadow) built-in versions. It's therefore never advisable to target Nightly in a train-hop. Also, if you, for example, shipped a train-hop to 145.0, and then uplift a bunch of new interventions into 145.0.2 a few weeks later, the train-hop update will override that. If you think this will become an issue, Release Engineers should be able to help you out by specifically targeting individual dot-versions as opposed to a whole major version.
2. Make sure RelEng sees the comment - either by ni? them, or by sending them a message. Release Engineering will need to involve Release Management to get the necessary sign-offs.

When Release Engineering pushes the update live, you're done. Congratulations, you just made the web better for our users - and they won't even notice it.

Speeding up the rollout

We do not have a real mechanism to speed up the deployment. Clients check for addon updates once per day, and we don't have a way to tell clients "do it NOW". However, if time presses or you want to do accelerated testing of something, you can motivate individual Firefox installs do update immediately.

• You can open the Browser Console/Browser Toolbox, and paste ChromeUtils.importESModule("resource://gre/modules/AddonManager.sys.mjs", { global: "shared" }).AddonManagerPrivate.backgroundUpdateCheck();.
• Alternatively, you can open about:config and set app.update.lastUpdateTime.addon-background-update-timer to 0. That will not trigger an update right away, but it will speed it up a lot. Since this is a process that's easily available to users, it's fine to instruct them to do so if there is demand.

Observing the rollout

Telemetry data includes the version numbers of system-addons, so you can leverage that. You can fork this query (Mozilla-internal). Be sure to adjust:

• Inside the base CTE
  • The submission_timestamp condition to be close to the beginning of the rollout to avoid querying unnecessary data
  • The mozfun.norm.truncate_version(application.version, 'major') condition to match the major version(s) you're targeting.
• Inside the proxy CTE
  • The payload_info_addons condition to target the interventions addon version you're interested in.

The resulting graph is a percentage of client checkins that have the new version installed. The query is using the live telemetry data, so you're as close to real-time as it gets. Note that it is totally fine and expected that the line goes down sometimes. During certain times of the day, a lot of clients wake up for the first time, and they might do a telemetry checkin before they get the update.

You should usually see a 40-50% rollout after three hours or so. After 24 hours, the chart should be around 90%.