Release Management/Release Process Checklist Documentation

< Release Management

The goal of this page is to document the Release Process Checklist being used by Firefox Release Managers to track each release throughout the cycle. Any changes to this documentation or the checklist should be reflected in both documents.

Contents

Nightly Checklist

Given the nature of how nightly builds are created and shipped, the role of the release manager during this phase of the cycle skews much more heavily to the monitoring aspect rather than release mechanics.

Prior to the start of the Nightly cycle, the following tasks need to be performed before Merge Day

  1. Create a Release Checklist before the cycle starts.
  2. Create the Milestones Document for the Release Checklist.
    • Use the sheet linked in the Release Checklist.
  3. Create a Firefox Retrospective Notes document.
    • Use the template from Google drive.
    • It is useful to document during the release cycle.
  4. Create a folder in the Release Tracking Google Drive for the release.
    • Move the Release Checklist and Firefox Retrospective Notes to this folder.
  5. Create a release notes skeleton in Nucleus.
    1. Click Admin Interface
    2. Click the Releases.
    3. Select the previous Nightly release notes (enable the checkbox beside the release).
    4. At the top of the page, expand the Action dropdown
    5. Select Copy releases.
    6. Click Go.
    7. Select the copy.
    8. In the Version text box edit the version to the current Nightly version.
      • Example: 99.0a1
    9. Set the Release Date to match when Central was bumped to Nightly
    10. Remove the release notes from the previous release.
    11. For additional information on the Release Notes process see The Firefox release notes process wiki page.
  6. Ensure that Bugzilla is bumped to the Nightly version, see BMO/new-version
    • Note: This is performed on the Friday before Merge Day.
    • Find the bug that tracks the bump.
      • Example bug
      • Please Note: The Milestone bump is performed by Bugzilla admins, see the ticket assignee.
    1. Update the Release Tracking flags in Bugzilla
    2. Go to Bugzilla > Administration > Release Tracking Flags
    3. Add the cf_tracking_firefox and cf_status_firefox for the Nightly release
      • Example: If 97 is becoming the Nightly, then add cf_tracking_firefox97 and cf_status_firefox97.
    4. Deactivate the oldest cf_tracking_firefox flag.
      • Example: If 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then deactivate cf_status_firefox94.
    5. Edit the current cf_tracking_firefox_esr tracking flag to add the new release as a Value and deactivate the oldest release.
      • Example: If esr is based on 91, 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then edit cf_tracking_firefox_esr91 to add a Value 97+ set to mozilla-next-drivers and deactivate the Value 94+
    6. Add the cf_tracking_thunderbird and cf_tracking_thunderbird for the Nightly release.
      • Example: If 97 is becoming the Nightly, then add cf_tracking_thunderbird97 and cf_tracking_thunderbird97
    7. Deactivate the oldest cf_tracking_thunderbird flag.
      • Example: If 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then deactivate cf_tracking_thunderbird94.
    8. Edit the current cf_tracking_thunderbird_esr tracking flag to add the new release as a Value and deactivate the oldest release.
      • Example: If esr is based on 91, 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then edit cf_tracking_firefox_esr91 to add a Value 97+ set to mozilla-next-drivers and deactivate the Value 94+.
    9. Add a comment on the bug that the tracking flags were updated.
  7. Update the Google calendar Releases Scheduling.
    1. Open the following URL and change the version to the Nightly release.
    2. Download the generated ical file.
    3. Review locally on a calendar app of your choice
    4. Open Google Calendar.
    5. Click + beside Other Calendars.
    6. Select Import.
    7. Select the ical file previously downloaded.
    8. Select the Releases Scheduling calendar.
    9. Click Import.
    10. Please Note: Importing the ical to Google Calendar is a one-shot operation. If you notice incorrect info or dates after importing, correct them directly in Google Calendar. Importing the ical file again will cause duplication.
  8. Ensure the REO (Regression Engineering Owner) is listed on Release Owners.
    • If required check the Google Sheet REO Rotation by Director.
    • Update the wiki with information from the spreadsheet.
      • Ensure that the REO and RelEng owners are listed.
      • Ensure the dates are accurate.
    • If required reach out to the Director to find out who the REO is.
    • Ensure they are invited to the Weekly Regression Triage.
  9. Ensure the Release Engineer on Release Duty is listed on Release Owners.

At the start of the Nightly cycle, the following tasks need to be performed on Merge Day

  1. Verify that Central was bumped to the Nightly version.
    • Check version_display.txt
    • Please Note: This is performed on Merge Day of the release going to beta, if it is incorrect talk to the RelMan on duty for Beta cycle.
  2. Make the release notes public in Nucleus
  3. Verify that the android-components version and GV version was updated.
    • version.txt should be xxx.0.0, where xxx matches the nightly version of desktop.
    • The version in Gecko.kt should match a nightly build of GeckoView as available on maven
    • See example PR
    • If the versions are not updated, then sync with android developers on the #releaseduty-moblie Slack channel.
  4. Verify that the version in Fenix and Focus was updated.
    • The version should be xxx.0b1, where xxx matches the nightly version of desktop.
    • If the versions are not updated, then edit version.txt and commit.
    • Please Note: The version on main always contains b1
  5. Verify the firefox_versions.json is correct.
    • Please Note: Usually this json update is performed the day after merge day
    • Please Note: fx trains and various scripts in other tools use this data.
    • Verify the following are correct:
      • FIREFOX_NIGHTLY, NEXT_MERGE_DATE, NEXT_SOFTFREEZE_DATE
    • If these are incorrect, contact the RelEng on duty via the #release-duty channel in Matrix. It is possible a PR was missed during the RelEng merge day steps.
    • Please Note: There are also date changes that are monitored by autonag. For example, if Template:NextReleaseDate was not updated an autonag email will be sent.

On a daily (or thereabouts) basis, the following items should be monitored during the Nightly cycle

  1. Review Pending tracking-firefoxXX requests
    • Example Filter
    • Ensure that the tickets are progressing, follow-up as required.
  2. Review Open tracking-firefoxXX+ and blocking bugs
    • Example Filter
    • Ensure that the tickets are progressing, follow-up as required.
  3. Review Newly-filed regression bugs
    • Example Filter
    • Also see the Bugs Lists section in the Platform Wiki Page
    • Ensure that new tickets are prioritized.
    • Ensure that carry-over tickets are still on the team’s radar.
    • Review with the REO in the weekly regression meeting.
  4. Review stability rates and reported crash spikes
    • Leverage information from the automated emails, monitor top crashes in Crash Stats, and monitor Mission Control
    • Ensure that there are bugs tracking stability rates and crash spikes.
    • Ensure these tickets are tracked and prioritized.
    • Please Note: If ever there are stats/bugs that cannot be explained/identified, then reach out to the team for assistance on the #stability Matrix channel, it may be an issue with the tooling issue.
  5. Review the Release Notes requests for Nightly Tickets.
    • Example Filter
    • Check through tickets in the release that are tagged for release notes. Review and add these notes to the Nightly Release Notes in Nucleus.
  6. Review tickets to check anything that should require a release note.
    • Leverage fx-trains nightly to check patches landed in central.
    • Reach out to the patch developer to check.

On a weekly (or thereabouts) basis, the following items should be monitored during the Nightly cycle

  1. Review and land mozilla-central repo-update HSTS HPKP remote-settings tld-suffixes patches.
    • Every Monday and Thursday revisions are automatically created for review in Phabricator.
    • Monitor email notifications sent by Phabricator: “No Bug, mozilla-central repo-update HSTS HPKP remote-settings tld-suffixes”
    • Review and land the patch to autoland.
  2. Review the tracking Mana page for the release.
    • The tracking page can be found as a child of Firefox Release Tracking
    • Ensure that features targeting the release are on track, reach out as needed if any are slipping.
  3. Update the Weekly Release Tracking Digest document, as requested on #fx-weekly-digest.
    • Update the Release Milestones (Details) section to add Status (On Track, Delayed), Ship Date, and additional details depending on where the nightly is in the cycle for example coming up to the Soft Freeze.
    • Review the dates in the Release Milestones section and update the status.

As available, the following items should be monitored during the Nightly cycle

  1. Review Test Plans for features targeting the release. QA shares test plans via email. For example:
    • Become familiar with how the feature works.
    • Understand how the team ensures it is of sufficient quality to ship in that release.
    • Provide feedback on the risk analysis and mitigations put in place.
  2. Review mid-Nightly and pre-Beta testing reports QA shares mid-Nightly and pre-Beta testing reports via email. For example:
    • If there are any red or yellow reports review the recommendations/plan and the bugs mentioned in the report.
    • If QA flagged that the feature is behind a pref verify that this is expected.

At the end of the Nightly cycle, the following tasks need to be performed prior to and during Merge Day

  1. Send Nightly Soft Code Freeze reminder to dev-platform and firefox-dev.
    • Send this at the start of the week before the Soft Code Freeze.
    • Remind developers that the window for landing riskier fixes is coming to a close until after the version bump.
    • Example email
  2. Prepare the Beta release notes in Nucleus
    1. Click Admin Interface
    2. Click the Releases
    3. Select the Nightly release notes (enable the checkbox beside the release)
    4. At the top of the page, expand the Action dropdown
    5. Select Copy release
    6. Click Go
    7. In the Channel dropdown select Beta
    8. In the Version text box edit the version to the Beta version
      • Example: 96.0beta
    9. Replace the content in the Text box with content from a previous beta release
    10. Set the Release Date to the date when the first Beta is released
    11. Ensure to remove any notes that are only applicable to nightly
  3. Sync with RelEng to confirm the RelEng on duty for the Central to Beta merge.
    • Sync on Friday before Merge Day
    • Use the #releaseduty channel on Matrix.
    • Align to perform the merge early on Merge Day, for example, 9/10am ET.
  4. Sync with the Sheriff team to confirm Central is in a good state so that the Merge Day email can be sent.
    • Sync on Merge Day morning
    • Use the #sheriffs channel on Matrix.
    • Check the Beta simulations document for the release, this document is emailed to Release Management at the start of the night cycle.
    • Ensure that central does not currently have any known severe regressions or performance issues that would be carried into beta.
  5. Once you have confirmed central is in a good state, send the Merge Day email to release-drivers and release signoff
  6. Sync with RelEng on the Central to Beta merge
    • Use the #releaseduty channel on Matrix to mention the merge day email was sent
    • Sync with the RelEng on duty. The RelEng on duty is listed in the Matrix channel description
  7. Create a Desktop Beta Checklist and Mobile Beta Checklist for beta 1
    • Copy the Desktop Beta Checklist template and the Mobile Beta Checklist
    • Please Note: Depending on the Beta, some steps are irrelevant. Set the Status to N/A for these steps.
  8. Wait for RelEng to complete the central to beta merge.
    • Use the #releaseduty channel on Matrix to synchronize if needed.
    • RelEng will also reply to the Merge Day email
  9. Announce the soft freeze is over
    • Once the merge is complete, reply to the soft freeze reminder email
    • Example: “Hi, the Gecko 100 version bump landed on central as planned and the soft freeze is over.”


Beta Checklist

Once a release moves to the Beta cycle, the daily tasks performed during the Nightly cycle will continue to be carried out as new bug reports come in from a wider audience and new features move through the QA cycle towards shipping. There is also an added triage step during Beta - monitoring the “missed uplifts” email or queries to find issues fixed in Nightly but that still affect Beta.

The following tasks need to be performed on Merge Day at the start of the Beta cycle

  1. After the central to beta merge, review any pending uplifts that may be required before proceeding with a beta 1 build
    • This should not be a common occurrence, however, an example would be a crash fix that may have landed in central post-merge.
  2. Once the merge is complete, set up Desktop and DevEdition builds in Ship-It
    1. Connect to the VPN
    2. Log into Ship-It
    3. Click New Release
      • Shit-it new release.png
    4. For Desktop complete as follows and click Create Release:
      • Please note: Revision should be the tip. The Partial updates field is automatic.
      • Screen Shot 2022-05-31 at 3.38.42 PM.png
    5. For DevEditon complete as follows and click Create Release
      • Please note: Revision should be the tip. The Partial updates field is automatic.
      • Screen Shot 2022-05-31 at 3.40.59 PM.png
  3. Start the Desktop and DevEditon in Ship-it via the Promote Task
    1. View the Pending Releases in Ship-It
    2. Click the Promote Task for Desktop and continue through the pop-ups
    3. Click the Promote Task for DevEditon and continue through the pop-ups
  4. Confirm the builds have started
    • Taskcluster email: “[desktop] Build of firefox xxx.0b1 build 1”
    • Taskcluster email: “[desktop] Build of devedition xxx.0b1 build 1”
  5. Confirm the notification is sent when builds finish
    • Taskcluser email: “firefox xxx.0b1 build1/mozilla-beta is in the candidates directory”
    • Taskcluser email: “devedition xxx.0b1 build1/mozilla-beta is in the candidates directory”
  6. Schedule a push to CDN from ship-it (Desktop & DevEdition) via Push
    1. Click Push
      • Screen Shot 2022-05-31 at 3.42.21 PM.png
    2. Click Schedule
      • Screen Shot 2022-05-31 at 3.44.18 PM.png
  7. Confirm the notification email is sent when CDN push finishes
    • Taskcluser email: “firefox xxx.0b1 build1/mozilla-beta has been pushed to cdntest”
    • Taskcluser email: “devedition xxx.0b1 build1/mozilla-beta has been pushed to cdntest”
  8. Branch android-components from main to releases/xxx.0
    • xxx matches the desktop release version.
    • This branch is used for beta and rc builds. This step can be performed before the Beta 1 build has shipped to the beta channel, geckoview is built during the promote phase.
  9. Branch fenix from main to releases_vxxx.0.0
    • xxx matches the desktop release version.
    • This branch is used for beta and rc builds.
  10. Branch focus-android from main to releases_vxxx.0
    • xxx matches the desktop release version.
    • This branch is used for beta and rc builds.
  11. Bump Android Components to Beta on xxx release branch with PR
    • Create a PR to make the following changes in buildSrc/src/main/java/Gecko.kt
      • Change the GeckoChannel from nightly to beta
        • From val channel = GeckoChannel.NIGHTLY to val channel = GeckoChannel.BETA
    • Let the Github Action bump the geckoview version like in 12547.
      • Please Note: The beta 1 GeckoView package number for beta can be found in Maven. This is built during the desktop build.
  12. Update Android Components xxx.0.0 tag with Releasenotes, Changelog, & Milestone links like this
    • xxx matches the desktop release version
  13. Create AC release in Ship it
    • Once the PR to change the GeckoChannel and the PR to bump the geckoview have both merged, go to ship-it and create new release.
  14. Ship AC release in Ship it
    • Click the Ship button
  15. Change the Android Components version in Fenix
    • buildSrc/src/main/java/AndroidComponents.kt - AndroidComponents version should be xxx.0.0, where xxx matches the desktop release version.
    • Example
    • Please Note: You will need to wait for the android components build to complete before performing this step.
  16. Change the Android Components version in Focus
    • buildSrc/src/main/java/AndroidComponents.kt - AndroidComponents version should be xxx.0.0, where xxx matches the desktop release version.
    • Example
    • Please Note: You will need to wait for the android components build to complete before performing this step.
  17. Set up a Fenix build in Ship-it
    • Please note: you can only start a ship-it build on a revision that has a completed decision task
  18. Start a Fenix build from ship-it via Promote
  19. Ship Fenix from ship-it via Ship
  20. Setup a Focus build in Ship-it
    • Please note: you can only start a ship-it build on a revision that has a completed decision task
  21. Start a Focus build from ship-it via Promote
  22. Ship Focus from ship-it via Ship
  23. Monitor for QA sign-off on functional testing, before proceeding with Step 24.
    • Please Note: Build Validation sign-off is usually provided the day after Merge Day.
    • QA will post a message to the #qa-coordination channel in Slack when they complete functional testing.
    • Build validation testing is tracked here
  24. Once QA has signed off, push Desktop and DevEdition to Beta in Ship-It via Ship.
  25. Verify that the Balrog rule changes are live and correctly set.
  26. Email release-signoff with a confirmation that updates are live.
  27. Activate automated betas in Ship-It
    1. At the bottom of the page, click the red x beside Firefox Desktop: Beta
      • Screen Shot 2022-05-31 at 3.51.31 PM.png
    2. Click Enable
      • Screen Shot 2022-06-01 at 9.47.26 AM.png
    3. Perform the same actions for Firefox Developer Edition: Beta
    • Please Note: Automated Betas are built at 21:00 UTC every Sunday, Tuesday, and Thursday as determined here.
  28. Make the Beta release notes public in Nucleus
    • Enable the Is Public checkbox and save
      • Screen Shot 2022-06-01 at 9.49.18 AM.png
    • Share the Beta release notes URL in #release-notes Slack channel
      • Also mention, if anyone has any feature worth a release note but not yet listed, they can nominate it via the relnote-firefox ? flag in Bugzilla
  29. Monitor for QA sign-off on Fenix/Focus build validation before proceeding with Steps 30 and 31.
    • Please Note: Build Validation sign-off is usually provided the day after Fenix/Focus builds are produced.
    • Focus and Fenix QA sign-off are sent via email, with the following subjects:
      • [mobile] Focus Beta xxx.0.0-beta.1 - Manual testing sign-off [GREEN]
      • [Mobile] Firefox Beta xxx.0.0-beta.1 - Manual testing sign-off[GREEN]
  30. Once QA has signed off, roll out Fenix to the production track at 50% in Google Play
  31. Once QA has signed off, roll out Focus to the Closed testing - Foxfooding track at 100% in Google Play
  32. Monitor for QA sign-off on desktop update testing
    • QA will post a message to the #qa-coordination channel in Slack when they complete update testing.

The following tasks need to be performed daily during the Beta cycle

  1. Monitor Uplift Requests
    • Example Filter
    • Accept or reject the beta uplift requests.
    • See Uplift Rules for guidelines
    • Please Note: During the RC week, also monitor release uplift requests.
    • Please Note: When pushing uplifts be conscious of timing around the automated beta builds. Pushing an uplift will trigger CI, the automated beta build will not trigger until the CI completes.
    • Please Note: If an uplift contains string changes, then l10n approval is required and the commit message must end with l10n=approver_name. Add a ni on the bug for approval before taking the uplift.
  2. Monitor Pending tracking-firefoxXX requests
    • Example Filter
    • Ensure that the tickets are progressing, and follow up as required.
  3. Monitor Open tracking-firefoxXX+ and blocking bugs
    • Example Filter
    • Ensure that the tickets are progressing, and follow up as required.
  4. Monitor Newly-filed regression bugs
    • New regressions bugs can be found here
    • Review with the REO - ensure that new tickets are prioritized and ensure that carry-over tickets are still on the team’s radar
  5. Monitor stability rates and reported crash spikes
    • Leverage the automated emails, monitor top crashes in crash-stats, and monitor Mission Control
    • Ensure that there are tickets tracking issues with stability rates and crash spikes
  6. Monitor the Release Notes requests for Beta Tickets.
    • Example Filter
    • Check through tickets in the release that are tagged for release notes. Review and add these notes to the Beta Release Notes in Nucleus.

The following tasks need to be performed for each Desktop beta build (Sunday, Tuesday, Thursday) during the Beta cycle

Please Note: The Beta build pipeline is also monitored by the Sheriffs, if there is an intermittent test failure they will rerun the failed job. If there is a permanent failure or an issue in the build infrastructure, then it may require canceling the build. Discuss in the #releaseduty Matrix channel. If the decision is made to cancel, then view the pending release in ship-it and click cancel. To create a new build, follow the same steps as outlined in One Time - Start of Beta Cycle to produce build 2 of the current beta release.

  1. Create a tab in the release tracking sheet “xxx.0bx”
    • Please note: After the Desktop beta has shipped, the tab can be hidden in order to minimize clutter.
  2. Review tracking-firefoxXX+ bugs and approval requests.
    • This step is performed as part of the daily activities, tracked in the checklist for confirmation.
  3. Verify all approved bugs landed on mozilla-beta.
    • This step is performed as part of the daily activities, tracked in the checklist for confirmation.
  4. Confirm the builds have started.
    • Taskcluster email: “[desktop] Build of firefox xx.0bx build 1”
    • Taskcluster email: “[desktop] Build of devedition xx.0bx build 1”
    • Please Note: The automated beta builds schedule is determined here.
  5. Confirm the notification is sent when builds finish.
    • Taskcluster email: “firefox xx.0bx build1/mozilla-beta is in the candidates directory”
    • Taskcluster email: “devedition xx.0bx build1/mozilla-beta is in the candidates directory”
  6. Confirm the notification is sent when CDN push finishes.
    • Taskcluster email: “firefox xx.0bx build1/mozilla-beta has been pushed to cdntest”
    • Taskcluster email: “devedition xx.0bx build1/mozilla-beta has been pushed to cdntest”
  7. Verify that the Balrog rule changes are live and correctly set.
  8. Email release-signoff with a confirmation that updates are live.
    • Beta 2 Subject: “[desktop] Firefox Desktop & DevEdition xxx.0b2 are live”
    • Beta 2 Body: “Firefox Desktop & DevEdition xxx.0b2 are live on the Beta and #* Aurora channels at 50% and 100% rollout, respectively.”
    • Beta 3+ Subject: “[desktop] Firefox Desktop & DevEdition xxx.0bx are live @ 100% rollout [EOM]”
    • Beta 3+ Body: N/A
    • Please Note: For the final beta release email, mention it was the last beta.
      • Subject: [desktop] Firefox Desktop & DevEdition xxx.0bx are live @ 100% rollout, this was the last beta [EOM]
  9. Monitor for QA sign-off on build validation for both update/functional testing.

The following tasks need to be performed for each Fenix/Focus beta build (Sunday, Thursday, or as needed) during the Beta cycle

Please Note: We aim to produce Fenix and Focus builds based on the Desktop Beta builds every Sunday and Thursday. However, if there is a significant fix on the Android application layer, or in a Tuesday beta, then it may be necessary to then build for Fenix and Focus.

  1. Confirm if new Fenix and Focus builds are required.
    • For example, if there were any uplifts that impact Android or if there were commits in the Fenix/Focus release branches.
  2. Create a tab in the release tracking sheet “Mobile xxx.0bx”
    • Please note: After the Fenix/Focus beta has shipped, the tab can be hidden in order to minimize clutter.
  3. Confirm Android Components has the latest beta GeckoView build.
    • android-components repo
    • If it is not up-to-date, check the status of the PR pending automatically created when a new GeckoView is available in Maven.
  4. Create AC release in Ship it
  5. Ship AC release in Ship it
  6. Confirm the Android Components bump was merged into the Fenix/Focus release branch.
    • fenix/buildSrc/src/main/java/AndroidComponents.kt
    • focus-android/buildSrc/src/main/java/AndroidComponents.kt
    • If it is not up-to-date, check the status of the PR pending automatically created when a new Android Components package is available.
  7. Check if there are any pending PRs for Fenix/Focus release branch before proceeding with the steps to build Fenix/Focus
  8. Create a new release in Fenix build in ship-it.
  9. Start Fenix build from ship-it via Promote.
  10. Ship Fenix from ship-it via Ship.
    • Confirm taskcluster notification is sent when the push completes "fenix xxx.0bx build1 has been pushed to the closed testing track on Google Play"
  11. Create a new release in Focus build in ship-it.
  12. Start Focus build from ship-it via Promote.
  13. Ship Focus from ship-it via Ship.
    • Confirm taskcluster notification is sent when the push completes "focus-android xxx.0bx build1 has been pushed to the closed testing track on Google Play"
  14. Monitor for QA sign-off on Fenix/Focus build validation before proceeding with Step 15 and 16.
    • Focus and Fenix QA sign-off are sent via email, with the following subjects:
      • [Mobile] Focus Beta xxx.0.0-beta.x - Manual testing sign-off [GREEN]
      • [Mobile] Firefox Beta xxx.0.0-beta.x - Manual testing sign-off[GREEN]
    • Please Note: Build Validation sign-off is usually provided the day after Fenix/Focus builds are produced. If the sign-off is yellow/red, then follow-up on the relevant team Slack channel. Ensure a plan is in place or a decision is made to block/proceed with the release.
  15. Once QA has signed off, roll out Fenix to the production track in Google Play
    • See Firefox for Android Beta
    • Please Note: Beta 1 rollout is 50%, for Beta 2 and greater rollout at 100%.
    • Reply to the QA sign-off email
      • “Thanks, published to Production at xxx% rollout.”
  16. Once QA has signed off, roll out Focus to the Closed testing - Foxfooding track in Google Play
    • See Focus (Beta for Testers)
    • Please Note: Beta 1 rollout is 100%.
    • Reply to the QA sign-off email
      • “Thanks, published to the Foxfooding channel at xxx% rollout..”

As available, the following items should be monitored during the Beta cycle

  1. QA shares beta testing and pre-release testing reports via email. Review the test reports for the release. For example:
    • If there are any red or yellow reports, review the recommendations/plan and the bugs mentioned in the report. For example:
      • Track the bug reported as blockers.
      • If there is no clear plan to address the report, follow up to ensure that engineering is reviewing.
    • If QA mentions the feature is behind a pref verify that this is expected.
  2. Phabricator patches for remote-settings updates are generated automatically. Typically this is every Monday and Thursday. These patches must be included in the beta uplifts.
    • Monitor email notifications sent by Phabricator: “No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes”
    • Review the patch.
    • Commit the patch to beta via the moz-phab tool
      • moz-phab patch <patch id> --no-bookmark --apply-to here
    • Edit the commit message to add the approval.
  3. Update the Beta Quality Metrics in the Mana release tracking page, a child page of Firefox Release Tracking is created per release.
    • Add a new column In the Beta Quality Metrics table using the current beta version as the title.
    • Add data from each of corresponding the metrics URLs (Bugzilla/telemetry) in the Metric Name column.
      • Ensure to add the Firefox metric, not the target metric.
      • While gathering the telemetry data metrics, take a look at the metric over time to ensure the value is stable.
      • Please Note: Ensure to update this prior to the Tuesday channel meeting.
  4. Update the Weekly Digest document, as requested on the #fx-weekly-digest Slack channel.
    • Update the Release Milestones (Details) section to add Status (On Track, Delayed), Ship Date, information on the current beta release for both Desktop/DeveEdition and Fenix/Focus, information on where we are in the Beta cycle for example when coming up to end of early beta.
    • Review the Release Milestones section dates and update the status.
    • Update the Quality Metrics section to add information on the number of outstanding regressions, if there are any release blockers, and an overview on the trend of the Beta Quality Metrics metrics.

The following needs to be performed at the End of Early Beta

Please Note: This is performed on the Friday at the midday point of the beta cycle, see the Releases Scheduling google calendar. This is typically after beta 6 has been released. This should be done in its own push and before pushing any other uplifts to beta.

  1. Open build/defines.sh
  2. Clear the EARLY_BETA_OR_EARLIER flag:
    • EARLY_BETA_OR_EARLIER=1 to EARLY_BETA_OR_EARLIER=
  3. Commit the changes:
    • hg commit -m "No bug - post beta 6, unset EARLY_BETA_OR_EARLIER. a=me"
  4. Push the changes to beta.

The following needs to be performed at the end of the Beta Cycle

  1. Disable automated beta builds in Ship-it
  2. Please Note: Monitor beta/release uplift requests between the final beta build and the beta to release merge. It is possible that a late uplift request may come in for a regression/stability fix, evaluate if this is a blocker for the RC and if it should be uplifted to beta before the beta to release merge and RC build.

RC Checklist

Once a release moves to the Release cycle, preparations for the release candidates begin. Until go-live, the daily tasks are similar to the beta cycle in terms of monitoring and if needed create a new release candidate.

The following tasks need to be performed at the start of RC week (week before go-live)

  1. Sync with RelEng to merge Beta to Release.
  2. Once the merge is complete, set up the Desktop build in Ship-It.
    1. Connect to the VPN.
    2. Log into Ship-It.
    3. Click New Release.
      • Shit-it new release.png
    4. Complete as follows and click Create Release:
      • Screen Shot 2022-06-07 at 9.26.10 AM.png
      • Please note: Revision should be the tip. The Partial updates field is automatic.
    5. Click Submit.
      • Screen Shot 2022-06-07 at 9.27.24 AM.png
  3. Start build from Ship-It via Promote RC.
    • Screenshot 2022-06-07 at 09-28-44 RelMan Activities Overview.png
    • Click the Schedule button.
      • Screenshot 2022-06-07 at 09-31-56 RelMan Activities Overview.png
  4. Confirm the build has started.
    • Taskcluster email: “[desktop] Build of firefox xx.0 build 1”
  5. Confirm the notification is sent when builds finish
    • Taskcluser email: “firefox xxx.0 build1/mozilla-release is in the candidates directory”
  6. Push Desktop to Beta at 100% from Ship-It via Ship RC
    • Screenshot 2022-06-07 at 09-32-46 RelMan Activities Overview.png
    • Click the Schedule button.
      • Screenshot 2022-06-07 at 10-28-30 RelMan Activities Overview.png
  7. Verify that the Balrog rule changes are live and correctly set.
    1. View the Balrog Rules for Firefox Beta.
    2. Verify that the Background Rate for Firefox-xx.0-build1 is 100.
      • Screenshot 2022-06-07 at 10-39-15 RelMan Activities Overview.png
      • Please Note: The Fallback Mapping is the previous beta release.
  8. Email release-signoff with a confirmation that updates are live.
  9. Bump the Fenix version number from xxx.0b# to xxx.1.0.
  10. Bump the Focus version number from xxx.0b# to xxx.1.0.
  11. Change Android Components from Beta to Release.
    1. Fork the release branch.
    2. Change and commit the following:
      • buildSrc/src/main/java/Gecko.kt
        • Change the GeckoView Version to the RC1 GeckoView package
          • Please Note: The package number for RC1 can be found in Maven
        • Change the GeckoChannel from beta to release
          • From val channel = GeckoChannel.BETA to val channel = GeckoChannel.RELEASE
    3. Create a PR to merge the fork into the release branch and request review from another release management team member.
    4. Once the PR is approved and the CI tests pass, merge the PR.
  12. Check if there are any pending PRs for Fenix/Focus release branch before proceeding with the steps to build Fenix/Focus.
    • fenix
    • focus-android
    • Example Filter: is:pr is:open base: releases_v101.0
    • If the pending PR is reviewed and is waiting on CI, then where possible wait until the PR has been merged.
    • If ever unsure, reach out on the relevant team Slack channel #fenix-team or #focus-browser.
  13. Wait for the following to finish before proceeding with the steps to build Fenix/Focus:
    • Android Components build is finished.
    • Android Components version bump PR created for Fenix/Focus release branch.
    • Android Components version bump PR merged into the Fenix/Focus release branch.
  14. Set up Fenix build in Ship-It.
    • Follow the same steps as creating a beta build.
  15. Set up Focus/Klar build in Ship-It.
    • Follow the same steps as creating a beta build.
  16. Start Fenix build from Ship-It via Promote.
  17. Start Focus/Klar build from Ship-It via Promote.
  18. Verify the package in the Microsoft Store submission is the correct version.
    • Please Note: The submission is automatically created as part of the RC build pipeline.
    • Verify that correct package was uploaded, the version displayed in the submission matches the release version
  19. Submit the Desktop RC build to the Microsoft Store for review
    • Please Note: The Microsoft Store review can take several days, it’s recommended to submit for review at the end of RC week. If there is a blocker found later, you can always resubmit.
    1. Navigate to the Microsoft Partner Center
    2. Select Firefox from Windows & Xbox > Overview
    3. Select the pending submission
    4. In Packages enable gradual rollout and set to 25%
      • Ensure to enable the option to provide the newest package when users manually check for updates.
    5. In Submission Options set to Automatically Publish on the go-live date and time
    6. Submit to the store for review
    • Please see App submissions documentation for additional information on the Microsoft Store submission process.

As available, the following should be monitored/performed during RC week

  1. Phabricator patches for remote-settings updates are generated automatically. Typically this is every Monday and Thursday. These patches must be included in release/esr uplifts.
    • Monitor email notifications sent by Phabricator: “No Bug, mozilla-release repo-update HSTS HPKP remote-settings tld-suffixes”/“No Bug, mozilla-esr91 repo-update HSTS HPKP remote-settings tld-suffixes”
    • Review the patch.
    • Commit the patch to release/esr via the moz-phab tool
      • moz-phab patch <patch id> --no-bookmark --apply-to here
    • Edit the commit message to add the approval.
  2. Update the Weekly Digest document, as requested on #fx-weekly-digest.
    • Update the Release Milestones (Details) section to add Status (On Track, Delayed), Ship Date.
    • Review the Release Milestones section dates and update the status.
  3. Monitor for QA sign-off on Fenix/Focus build validation before proceeding with Steps 4 to 9.
    • Please Note: Build Validation sign-off is usually provided the day after Fenix/Focus builds are produced.
    • Focus and Fenix QA sign-off are sent via email, with the following subjects:
      • [Mobile] Firefox RC xxx.x.x - Manual testing sign-off[GREEN]
      • [Mobile] Focus RC xxx.x.x - Manual and automated testing sign-off [Green]
  4. Ship Fenix from ship-it via Ship
  5. Roll out Fenix to the Production track at 5% in google play
  6. Ship Focus from ship-it via Ship
  7. Roll out Focus/Klar to the Production track at 5% and Open Testing track at 100% in google play
  8. Upload Fenix to Samsung Store with 5% rollout when approved
    • Upload x86 and x64 apks (armeabi-v7a/arm64-v8a)
    • Set the rollout rate to 5%
    • Set to publish automatically
    • Please note: It can take up to 2 days for review.
  9. Upload Focus to Samsung Store with 5% rollout when approved
    • Perform the same steps as Step 8
  10. Verify that QA have signed off manual testing for Desktop
    • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
  11. Verify that QA have signed off update tests on Beta
    • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
  12. Verify that QA have signed off update tests on release-localtest
    • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here

RC Uplifts

This tab is for tracking bugs that are being tracked for possible uplift to the mozilla-release repository for RC builds. The primary objectives are:

  • Track whether there are any drivers for a respin of the RC builds during RC week.
  • Assess whether Desktop, Mobile, or both are affected by the issues noted.
  • Verify that all drivers have had an explicit decision made.

The following tasks need to be performed during RC week (the week before go-live)

  1. Monitor Uplift Requests for Beta/Release
    • For Example
    • Track if there are any drivers for an RC respin during RC week.
    • Assess whether Desktop, Mobile, or both are affected by the issues noted.
    • Verify that all drivers have had an explicit decision made.
  2. If a new RC is required, for example, a release blocker was identified, then create a new RC checklist tab and perform the same steps as for the initial RC but with the following exceptions:
    1. Cancel the previous RC build in ship-it
    2. Delete the Microsoft Store submission before creating a new RC build
      • Please Note: The RC build will fail if trying to create an MSIX submission when there is already a pending submission
      • To cancel and delete the pending submission you may first need to change it from publishing automatically to publishing manually
  3. Determine if a new RC build impacts Android Components.
    • If a new Android Components build is not required, then there is no impact on the Fenix/Focus builds and no action is required.
    • If a new Android Components build is required, then repeat the RC steps with the following exceptions:
      • Android Components was already switched from Beta to Release.
      • If Fenix/Focus/Klar already rolled out - Replace the 5% rollout of Fenix/Focus/Klar in Google Play and the Samsung Store.
      • If Fenix/Focus/Klar has not already rolled out - cancel the Fenix/Focus RC builds in ship-it.


Go-Live Checklist

Once RC week is over and the release candidate is ready to go-live, preparations for go-live begin. After go-live, monitoring continues and if needed preparing for potential dot releases.

The following tasks need to be performed prior to go-live

  1. Ensure that feedback for release notes was gathered by UX/Product
    • Monitor the #release-notes Slack channel
    • For more information on the UX/Product release notes doc process see here
  2. Incorporate feedback into draft release notes.
    • Add a release in Nucleus for Firefox, Firefox for Android, Firefox for iOS
    • Duplicate the previous release, edit the version, dates, and remove the previous release notes. For example Firefox, Firefox for Android, Firefox for iOS
    • Create release notes entries as outlined in the document shared on the #release-notes Slack channel
    • Please Note: If images are included with the release note then create a PR for Bedrock, example PR
    • Share a link to the staged release notes on #release-notes
  3. Ensure that Legal signoff on release notes was requested by UX/Product
    • Please Note: This step is not always required, if the release notes are not atypical
  4. Review tracking-firefoxXX+ bugs and confirm there are no blocking issues
  5. Check for crash spikes with RC builds
  6. Schedule push to CDN from Ship-It
    • Click Push
    • Screenshot 2022-06-07 at 10-45-18 RelMan Activities Overview.png
    • Click Schedule
    • Screenshot 2022-06-07 at 10-47-14 RelMan Activities Overview.png
  7. Ensure the automated email was sent after the push to release-cdntest
    • Taskcluster email: “firefox xx.0 buildx/mozilla-release has been pushed to cdntest.
  8. Ensure QA have performed the Update test on release-cdntest
    • QA will post a message to the #qa-coordination channel in Slack when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
  9. Ensure Legal have signed off on release notes
    • Please Note: This is only required based on the outcome of Step 3.
  10. Ensure that the security advisories are ready
    • Check with the security team on the #security Slack channel

The following tasks need to be performed on go-live day

  1. Schedule push to release at 25% from Ship-It
  2. Bump Fenix rollout rate in the play store to 25%
  3. Bump Focus/Klar rollout rate in the play store to 25%
  4. Bump Fenix rollout rate in the Samsung Store to 25%
  5. Bump Focus rollout rate in the Samsung Store to 25%
  6. Verify that the Microsoft Store is published at 25% rollout
  7. Make release notes live in Nucleus
    • Please Note: There is a 15-30min lag between making the change in Nucleus and the live website picking up this change, so plan to do this 15-30min prior to go-live.
  8. Upload security advisories to GitHub
  9. Signoff on scheduled rule change in Balrog
    • Set the rollout percentage to 25%
    • Sync with release engineering on #release-duty in Matrix to sign-off on the rule change
  10. Send Slack message to #release-coordination to notify of the release
    • For example “Firefox 102, ESR 91.11, Fenix 102.1.1 and Focus/Klar 102.1.1 are live”
  11. Verify that the new release is live on mozilla.org
  12. Verify that the release notes are live here
  13. Verify LATEST_FIREFOX_VERSION in firefox_versions.json
  14. Verify that security advisories are live here
  15. Ship new Desktop release in Ubuntu Snap Store
  16. Email release-drivers & release-signoff that updates are live at 25%
  17. Ensure QA have performed the Update tests on Release
    • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
  18. Email announce list
  19. Schedule Desktop update rate to 0% in Balrog after 24 hours.
  20. Signoff on scheduled rule change in Balrog.
    • Sync with release engineering on #release-duty in Matrix to sign-off on the rule change.

TO DO: Add ESR steps for go-live day

The following tasks need to be performed 24 hours after go-live

  1. Verify Desktop update rate at 0% in Balrog
  2. Email release-signoff & release-drivers to confirm 0% throttling.

The following tasks need to be performed 48 hours after go-live

  1. Review release crash rates and incoming bugs for new blockers.
  2. Bump Desktop update rate to 100% in Balrog.
  3. Signoff on scheduled rule change in Balrog.
    • Sync with release engineering on #release-duty in Matrix to sign-off.
  4. Bump the Microsoft Store rollout to 100%.
    • Click Finalize package rollout
  5. Bump fenix/focus/klar rollout rate in the play store to 100%.
  6. Bump fenix/focus rollout rate in the Samsung Store to 100%.
  7. Email release-signoff & release-drivers to confirm full rollout.

The following tasks need to be performed daily after go-live

  1. Review release crash rates and incoming bugs for new blockers
  2. Review uplift requests for dot release drivers and ride alongs

The following tasks need to be weekly after go-live

  1. Update the Weekly Digest document, as requested on #fx-weekly-digest.
    • Update the Release Milestones (Details) section to add status on the release, dot release, etc.


Dot Release Uplifts

Similar to the RC Uplifts tab. The primary purpose of this tab is to track any bugs driving a dot release and bugs that are under consideration to ride-along, It is also used to assess which products are affected by any drivers. It is up to the release manager what should go into a dot release, and we try to keep these uplift guidelines in mind. Adding even what looks like a trivial fix can add risk, both to the process (delay, extra testing, work for other teams) and to causing new regressions. For security issues, we may not want to take them in a dot release unless there is particular pressure to do so.

The following information needs to be added for each bug:

  • Bug number
  • Component
  • Is the bug a dot release driver?
  • Does the patch affect Desktop, Mobile, or both?
  • Is the fix ready?
  • Is the patch on mozilla-central?
  • Is there an uplift request on the ticket?
  • Is the uplift approved?
  • Is the patch uplifted to release?
  • Does the bug need a release note?
  • Any additional notes
    • For example, outline why the bug is being taken in a dot release.

Please Note: For each dot release, create a section in the Dot Release Uplifts tab to track the bugs in consideration: For example, “Desktop 101.0.1& Fenix/Focus 101.2.0 - GTB 2022-06-08”


Dot Release Checklist

The following tasks need to be performed at the start of preparing a dot release

  1. Duplicate the Desktop Dot Release Checklist tab and name it appropriately.
    • Example: Desktop 101.0.1
  2. Email release-drivers group to inform them a dot release is being created. #* Include information on the driver, if there will be an Android release, and a link to the sheet used to track the uplifts.
  3. Review any pending release uplift requests
    • Example Filter
    • If they are low risk consider including them in the dot release
  4. Uplift patches to release.
  5. Create a release in Nucleus to add release notes
    • Add a release note with a link to the major release for reference.
    • For bugs that have been identified as needing a release note:
      • If a release note is required and the release note text is clear from the bug content then create it directly in Nucleus.
      • Otherwise, request wording before adding the release note.
  6. Follow the same process as creating an RC build
    • If any uplifts impact mobile
      • Duplicate the Mobile Dot Release Checklist tab and name it appropriately.
      • Follow the same process as producing a Fenxi/Focus RC build, with the exception of bumping the version number before building.
      • Fenix Example Commit
      • Focus Example Commit
  7. Follow the same process as reviewing an RC build has QA sign-off

The following tasks need to be performed during go-live of a dot release

  1. Follow the same steps as an RC go-live, with the following exceptions:
    • The dot release is rolled out at 25% but there is no schedule drop to 0%
      • Please Note: Under chemspill situations, a greater rollout percentage may be required.
    • There may not be any security advisories.

The following tasks need to be performed 24 hours after go-live of a dot release

  1. Follow the same steps as after an RC go-live, with the following exceptions:
    • If there are no blocking issues then bump the rollout to 100% after 24 hours

The following tasks need to be performed daily after go-live of a dot release

  • Follow the same monitoring steps as after an RC go-live.