Release Management/Release Process Checklist Documentation: Difference between revisions

# Add a Release Tracking Sheet link to [https://wiki.mozilla.org/Release_Management/Release_owners Release Owners] wiki

#* Add the link to the date in the Release Date column.  

#* Note: This is performed on the Friday before Merge Day.

## Update the Release Milestones in Bugzilla

##* The products and milesontes are automatically completed

## Click [ Submit ]

## Add the cf_tracking_firefox and cf_status_firefox for the Nightly release

##* Example: If 97 is becoming the Nightly, then add cf_tracking_firefox97 and cf_status_firefox97.

## Edit the current cf_tracking_firefox_esr tracking flag to add the new release as a Value and deactivate the oldest release.

##* Example: If esr is based on 91, 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then edit cf_tracking_firefox_esr91 to add a Value 97+ set to mozilla-next-drivers and deactivate the Value 94+

## Edit the current cf_tracking_thunderbird_esr tracking flag to add the new release as a Value and deactivate the oldest release.

##* Example: If esr is based on 91, 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then edit cf_tracking_firefox_esr91 to add a Value 97+ set to mozilla-next-drivers and deactivate the Value 94+.

# Create a release notes skeleton in [https://nucleus.mozilla.org/ Nucleus].

## Click '''Admin Interface'''

#* Ensure that the tickets are progressing, follow up as required.

# Review '''incoming regression bugs'''

#* Ensure that new tickets are prioritized.  

#* Ensure that carry-over tickets are still on the team’s radar.

#** Check the possible commit in [https://github.com/mozilla/application-services application-services main] that corresponds to the failure.

#** Start a thread on [https://mozilla.slack.com/archives/C0559DDDPQF #application-services-eng] for investigation.

# Review Test Plans for features targeting the release. QA shares test plans via email. For example:

#* Become familiar with how the feature works.

#* Remind developers that the window for landing riskier fixes is coming to a close until after the version bump.

#* [https://groups.google.com/a/mozilla.org/g/dev-platform/c/uoCFNyGJx0c Example email]  

# Prepare the Beta release notes in [https://nucleus.mozilla.org/ Nucleus]

## Click '''Admin Interface'''

## Set the Release Date to the date when the first Beta is released

## Ensure to remove any notes that are only applicable to nightly

= Beta Checklist =

Once a release moves to the Beta cycle, the daily tasks performed during the Nightly cycle will continue to be carried out as new bug reports come in from a wider audience and new features move through the QA cycle towards shipping. There is also an added triage step during Beta - monitoring the “missed uplifts” email or queries to find issues fixed in Nightly that still affect Beta.

# Switch to Application Services in Firefox Android to Release

#* Change the Version and Channel in mobile/android/android-components/plugins/dependencies/src/main/java/ApplicationServices.kt

#** See example [https://hg.mozilla.org/releases/mozilla-beta/rev/4c3f709451f8aa49beaf8d63fc2131c3ac85e82e commit]

#* Push directly mozilla-beta

# Review any pending uplifts that may be required before proceeding with a beta 1 build

#* This should not be a common occurrence, however, an example would be a crash fix that may have landed in central post-merge.

# Turn on [https://support.google.com/googleplay/android-developer/answer/9859654?hl=en managed publishing] in the Play Console for Focus.

# Manually create a Focus release on the closed testing track (Foxfooding) @ 25% rollout and submit for review

#* Please Note: Desktop Build Validation sign-off is usually provided the day after Merge Day.

#* QA will post a message to the [https://mozilla.slack.com/archives/CAC9YSH3P #qa-coordination] channel in Slack when they complete functional testing.   

#* This allows the app to publish automatically once it passes review.

# Turn off managed publishing in the Play Console for Focus

# Verify the [https://product-details.mozilla.org/1.0/mobile_versions.json mobile_versions.json] is correct

#* Verify the beta_version is correct

# Verify all approved bugs landed on mozilla-beta.

#* This step is performed as part of the daily activities, tracked in the checklist for confirmation.

# Confirm the Desktop and DevEdition builds have started.

#* Taskcluster email: “[desktop] Build of firefox xx.0bx build 1”

# Manually create a Fenix release on the Production track @ 50% rollout for b2 and @ 99% rollout for b3+

# Manually create a Focus release on the closed testing track (Foxfooding) @ 50% rollout for b2 and @ 99% rollout for b3+

# Confirm notification sent when the Desktop and DevEditon CDN push finishes.

#* Taskcluster email: “firefox xx.0bx build1/mozilla-beta has been pushed to cdntest”

#* For b3+, once QA has signed off, roll out Focus to the Closed testing - Foxfooding track in Google Play to 100%.

#** Reply to the QA sign-off Slack message and include the rollout percentage.

==As available, the following items should be monitored during the Beta cycle==

# Phabricator patches for remote-settings updates are generated automatically. Typically this is every Monday and Thursday. These patches must be included in the beta uplifts.

#* Monitor email notifications sent by Phabricator: “No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes”

# Use the Mana [https://mozilla-hub.atlassian.net/wiki/spaces/FIREFOX/pages/277284167/Beta+Quality+Metrics Beta Quality Metrics Page] to review the Beta Quality Metrics.

#* The data is embedded from this [https://docs.google.com/spreadsheets/d/1Ev8HSxyAQFB5eLX98r6hmDR7FFNViJD4JWnqxiEtx48/edit#gid=2061648925 wiki page].

==The following needs to be performed at the End of Early Beta==

==The following needs to be performed at the end of the Beta Cycle==

# Disable automated beta builds in Ship-it.

# Please Note: Monitor beta/release uplift requests between the final beta build and the beta to release merge. It is possible that a late uplift request may come in for a regression/stability fix, evaluate if this is a blocker for the RC and if it should be uplifted to beta before the beta to release merge and RC build.

=RC Checklist=

#* Verify that all approved uplift requests were uplifted to beta

# Verify all approved bugs landed on mozilla-release

# Once the merge is complete, verify that the [https://treeherder.mozilla.org/jobs?repo=mozilla-release Treeherder] tests green/starred

# Set up Desktop build in [https://shipit.mozilla-releng.net/ Ship-It].

## Connect to the VPN.

## Log into [https://shipit.mozilla-releng.net/ Ship-It].

## Complete as follows and click Create Release:

##* [[File:Screen Shot 2022-06-07 at 9.26.10 AM.png|thumb|center]]

# Confirm notification sent when Firefox Android builds finish.

# Push Firefox Android (AC, Fenix, Focus) release in Ship-It via Push.

# Push Desktop to Beta at 100% from Ship-It via Ship RC.

#* [[File:Screenshot 2022-06-07 at 09-32-46 RelMan Activities Overview.png|thumb|center]]

## Submit to the store for review

#* Please see [https://docs.microsoft.com/en-us/windows/uwp/publish/app-submissions App submissions] documentation for additional information on the Microsoft Store submission process.

==As available, the following should be monitored/performed during RC week for Desktop/Android==

# Phabricator patches for remote-settings updates are generated automatically. Typically this is every Monday and Thursday. These patches must be included in release/esr uplifts.

#* Monitor email notifications sent by Phabricator: “No Bug, mozilla-release repo-update HSTS HPKP remote-settings tld-suffixes”/“No Bug, mozilla-esr91 repo-update HSTS HPKP remote-settings tld-suffixes”

# Update the Weekly Digest document, as requested on [https://mozilla.slack.com/archives/C01TTHGPSKB #fx-weekly-digest].

#* Update the Release Milestones (Details) section to add Status (On Track, Delayed), Ship Date.

# Verify that QA have signed off update tests on release-localtest

#* QA will post a message to the [https://mozilla.slack.com/archives/CAC9YSH3P #qa-coordination] Slack channel when they complete update testing and another message when they complete functional testing.   

#* Please Note: Build Validation sign-off is usually provided the day after Fenix/Focus builds are produced.

#* Focus and Fenix QA sign-off are sent via [https://mozilla.slack.com/archives/CAC9YSH3P #qa-coordination] Slack Channel.

#* See [https://play.google.com/console/u/0/developers/7083182635971239206/app/4972519468758466290/app-dashboard?timespan=thirtyDays Firefox Fast & Private Browser]

#* Reply to the QA sign-off slack message

#** “Thanks, pushed to Production at 5% rollout.”

#* See [https://play.google.com/console/u/0/developers/7083182635971239206/app/4972436558672701695/app-dashboard?timespan=thirtyDays Firefox Focus: No Fuss Browser] and [https://play.google.com/console/u/0/developers/7083182635971239206/app/4974004938716340515/app-dashboard?timespan=thirtyDays Firefox Klar Browser]

#* Reply to the QA sign-off slack message

#** “Thanks, pushed to Production at 5% rolled out.

=RC Uplifts=

# Ensure that the security advisories are ready

#* Check with the security team on the [https://mozilla.slack.com/archives/C495VS94P #security] Slack channel

==The following tasks need to be performed on go-live day==

# Schedule push to release at 25% from [https://shipit.mozilla-releng.net/ Ship-It]

# Bump Fenix rollout rate in the play store to 25%

# Bump Focus rollout rate in the [https://seller.samsungapps.com/application/main.as?contentId=000003397900&platform=&readOnly=Y#/cert/info Samsung Store]  to 25%

# Ship Firefox Android (AC, Fenix, Focus) from Ship-It via Ship

# Upload security advisories to GitHub

# Signoff on scheduled rule change in [https://balrog.services.mozilla.com/rules?product=Firefox&channel=release Balrog]

#* Set the rollout percentage to 25%

#* Sync with release management for another team member to signoff on the rule change.

# Send Slack message to [https://mozilla.slack.com/archives/C6HD6P4TF #release-coordination] to notify of the release

#* For example “Firefox Desktop 115.0, ESR 115.0.0, Fenix/Focus/Klar Android 115.0, and Firefox/Focus/Klar iOS 115.0 are live”

#* Include links to the release notes.

# Verify LATEST_FIREFOX_VERSION in [https://product-details.mozilla.org/1.0/firefox_versions.json firefox_versions.json]

# Verify that the release is live on Flathub

#* The `release-flatpak-push-firefox` job runs as part of the ship graph and the publication is handled by Flathub.

# Bump the [https://partner.microsoft.com/en-us/dashboard/products/9NZVDKPMR9RD/overview Microsoft Store] rollout to 100%.

#* Click Finalize package rollout

# Review fenix/focus release crash rates, incoming bugs for new blockers, and spike/trend in negative play store reviews.

#* Check to see if there is anything concerning that would prevent completing the rollout.

# Update the Weekly Digest document, as requested on [https://mozilla.slack.com/archives/C01TTHGPSKB #fx-weekly-digest].

#* Update the Release Milestones (Details) section to add status on the release, dot release, etc.

* Is the patch uplifted to release?

* Does the bug need a release note?

* Any additional notes

** For example, outline why the bug is being taken in a dot release.

##* If an app is still pending review on a closed track then you cannot promote it to a production track.

##* See [https://support.google.com/googleplay/android-developer/answer/9859348?hl=en Prepare and roll out a release] for more information.

##* Keep the same release notes as the major release. Add any new notes at the top prefixed by the dot release version.

##* With managed publishing turned on, the app will not automatically release after the store review.

# Follow the same process as monitoring when an RC build has QA sign-off with the following exceptions:

## Turn off [https://support.google.com/googleplay/android-developer/answer/9859654?hl=en managed publishing] in the Play Console.