View source for Release Management/Nightly Respin

{{DISPLAYTITLE:Nightly Usability}}

<p style="font-size: larger; font-weight:bold;">This page details the process to back out a faulty patch and, if needed, get nightlies respun.</p>

Maintaining Firefox Nightly usability as the user’s main browser is a responsibility shared by everyone contributing to Firefox development. Our goal is to keep the Firefox Nightly channel as stable and usable as possible for our community. If the browser crashes or faces severe usability issues, Nightly users may leave, reducing critical feedback during development. Nightly feedback comes through crash reports, bug reports, and various in-product metrics. Getting new users on the nightly channel is hard. Maintaining a sizable active population on this channel is more valuable than avoiding disruptions such as backing out a patch or pausing updates for a short period.
When a severe regression is introduced in Firefox Nightly, the goal is to address the issue as quickly as possible. Depending on the scenario, this may require pausing Nightly updates, backing out the regressing patch, and creating new Nightly builds before resuming updates. The Release Management team ensures the balance between allowing expected bugs on this channel and maintaining the highest possible quality to retain a user base.


== Summarized process ==
# Identify the regression (e.g., crash spike, core functionality issue).
# File/track the bug in Bugzilla, ensuring all necessary flags are set.
# Pause updates if the regression impacts a large number of users.
# Request a backout via the Sheriffs in Matrix.
# Request new builds via the Sheriffs, if necessary.
# Communicate with the community about the bug and its resolution.

==How to track in Bugzilla==
* If it is a crash spike, then file the bug via Socorro.
** If the bug was already reported then ensure the bug has the correct crash signature.
* If the bug was already filed by a community member, then use it to track the regression. Add the nightly-community keyword if missing.
* If it is a functional regression and no bug has been created yet, then create a bug. 

The bug will be used to track a resolution for the regression. Ensure the following are set on the bug, where N is the version number for Nightly:
* ''status-firefoxN'' tracking flag is set as affected
* ''tracking-firefoxN'' set as blocking 
* Target milestone set to ''mozillaN''

==When should we pause automatic background updates and backout a patch?==

If you expect that many users will be impacted by a regression, then we should pause background updates.

We should stop background updates and back out in the following scenarios:
* New crash spike
* Regression in core functionality without an intuitive workaround, for example:
** Usability issue on a Top site 
** Awesome bar usability
** Bookmarks are inaccessible or cannot be created
** History is inaccessible or cannot be created/deleted
** Possible data loss
** Firefox Sync is not sending or receiving data

There can be scenarios where the regression in core functionality is only experienced by users who have enabled an experimental feature via Firefox Labs. Though the workaround for the user is to disable the experimental feature, this may not be intuitive for users. We will still aim to backout the regressing patch. 

==How to pause automatic background updates==

===Desktop===
Blocking automatic updates will not prevent new users from installing Firefox Nightly from [https://www.mozilla.org/firefox/channel/desktop/ mozilla.org] but it will mitigate the impact on our existing user base. Also, blocking automatic updates will not prevent users from checking manually for updates.

Currently, Release Management cannot pause automatic updates only for specific platforms.

Release management can pause Firefox Desktop Nightly automatic updates as follows:
# View the [https://balrog.services.mozilla.com/rules?product=Firefox&channel=nightly Firefox: nightly] rule in Balrog
# Select Disable Updates from the options menu
# Add the regression bug number as the reason comment

When updates are paused the [https://whattrainisitnow.com/release/?version=nightly Nightly] page on [https://whattrainisitnow.com whattrainisitnow.com] will display a message and include a link to the regression bug.

===Mobile===
Pausing Firefox Nightly updates on Mobile is not as flexible as Firefox Desktop due to app store restrictions.

Release Management manually submits Firefox Nightly mobile builds for app review. 
* If the severe regression is in a build that has not yet been pushed for app review, then do not push it.
* If the severe regression is in a build that was pushed but is still pending review, then cancel the submission.
If the issue is specific to a device or OS version, the app can be limited or blocked from those devices in the Google Play Store via the device catalog.

===How to find the patch to back out===
If it is a crash spike, then the stack trace analysis combined with the patches that landed in mozilla-central can help find the bug that introduced the instability.
* If it’s unclear then reach out on [https://mozilla.enterprise.slack.com/archives/C4D3JFF26 #firefox] in Slack or [https://matrix.to/#/#stability:mozilla.org #stability] on Matrix.
If it is a functional regression bug with a reproducible case, then we can use [https://mozilla.github.io/mozregression/quickstart.html mozregression] to identify the regressing bug.

If it is a functional regression bug without a reproducible case but there is an approximate window of when the bug was introduced, then the possible regressor could be found by looking through the patches that recently landed in Mozilla-central.
* If it’s unclear then reach out on [https://mozilla.enterprise.slack.com/archives/C4D3JFF26 #firefox] in Slack.

It could be identified that the regression is caused by an experiment rollout. Check if any experiments were recently enabled and work with the responsible engineering team to disable the experiment.

===How to request a backout===
The Sheriffs manage backing out patches from mozilla-central. You can contact sheriffs in the [https://matrix.to/#/#sheriffs:mozilla.org #sheriffs] channel on Matrix. Let them know the bug you want backed out and the reason for the backout. The Sheriffs will reopen the regressor with a comment that includes a link to the backout commit, the reason for the backout, and a needinfo on the assignee.

Once the backout is done, you can resolve the regression bug as FIXED and change the ''status-firefoxN'' tracking flag from ''affected'' to ''fixed''.

''Note'': Release managers manage backing out patches from mozilla-beta and mozilla-release.

===How to request new Nightly builds===
The Sheriffs can trigger new Nightly builds. You can contact sheriffs in the [https://matrix.to/#/#sheriffs:mozilla.org #sheriffs] channel on Matrix.

New Nightly builds could be requested when requesting the Sheriffs to backout the regressing patches.
* If the next nightly is about to build then it may be better to wait for the scheduled build.
* If the regression is platform specific then you can request new nightlies for that platform only.
device catalog.

== Communicating about the issue ==

We should not hesitate to communicate the issue with a reference to the bug number to our community so as to minimize the number of duplicate bugs. If the issue needs steps to reproduce which are not obvious or a specific hardware/OS combination, having all communications centralized in a single bug helps.

We should also remember communicating about the resolving of the issue and urge people to upgrade to the updated nightly (so as to reduce automatic crash reports and unneeded bugs filed).

Communicating about major regressions in Nightly is also part of the informal social contract we have with our alpha testers, making sure they are informed of major technical issues impacting them helps keeping them engaged.

The main communication channels to communicate a regression are our [https://twitter.com/firefoxnightly @FirefoxNightly] Twitter account, our [https://mozilla.social/@FirefoxNightly @FirefoxNightly@mozilla.social] Mastodon account and our [https://matrix.to/#/#nightly:mozilla.org #Nightly] chatroom on  Matrix/Element.

==Decision Matrix==
{| class="wikitable"
|-
!  Scenario !!Disable Updates !! Backout Regressor !! Request New Nightlies !! Additonal Info
|-
| '''New Top Crash''' || Yes || Yes || Yes || 
|-
| '''High severity''' bug impacting '''core functionality'''  || Yes || Yes || Yes ||
|-
| '''High severity''' bug only impacting '''experimental functionality''' exposed by the Firefox Labs UI  || No || No || No || Expected that engineering will prioritize a fix. If a fix is not expected within the next Nightly build window then consider back out the regressor.
|-
| '''Medium severity''' bug impacting '''core functionality''' || No || No || No || Expected that engineering will prioritize a fix.
May consider backing out in beta if we don’t have a fix during Nightly.
|-
| '''Medium severity''' bug only impacting '''experimental functionality''' exposed by the Firefox Labs UI || No || No || No ||  Expected that engineering will prioritize a fix.
|-
| '''Medium severity''' bug only impacting '''experimental functionality''' exposed by the Firefox Labs UI || No || No || No || Expected that engineering will prioritize a fix.
|-
| '''Low severity''' bug impacting '''core functionality''' || No || No || No || Expected that engineering will prioritize a fix.
|-
| '''Low severity''' bug only impacting '''experimental functionality''' exposed by the Firefox Labs UI || No || No || No || Expected that engineering will prioritize a fix.


|}

[[Category:Release_Management]]
[[Category:Release_Management:Processes|Nightly Respin]]