QA/Desktop Firefox/Walkthroughs/Release Coordination: Difference between revisions

The following document describes the work flows and processes that guide testing of releases of Firefox for PC platforms.

The following describes the processes which deliver a Firefox version from Nightly (mozilla-central) through to Release. Please consult [[Release_Management/Release_Process|Release Management's Release Process]] for more detailed information.

Firefox starts on mozilla-central, this is where the initial code is landed to deliver new features to users. Every six weeks the current Nightly version is uplifted to the Aurora branch then Nightly becomes the next Firefox version. This "uplift" always happens on a Monday, except when people are unavailable due to statutory holidays.

The following describes the general timeline for the entire six week life-cycle of a typical Nightly.

* Day 1:

** Release Management closes the tree and merges mozilla-central to mozilla-aurora

* Ensuring topcrash bugs are being investigated

* Ensuring iteration developed bugs are owned and signed off

* More information: [[QA/Desktop_Firefox/Walkthroughs|walkthroughs]], [[Releases/Firefox_31/Test_Plan#Nightly|example test plan]].

* Day 0 (Monday)

** Release Management will migrate mozilla-central to mozilla-aurora [[https://hg.mozilla.org/releases/mozilla-aurora/rev/f2607a346b7d  example changeset]] and files a bug to throttle updates on the ''aurora'' channel

** Release Management migrates mozilla-aurora to mozilla-beta

On the Monday before every release will be ''Migration Day''. Sometime before the end of the day, Release Management will close the mozilla-aurora tree and merge in the changes from mozilla-central. Once this is complete the tree will be reopened and Release Management will file a bug to disable updates on the ''aurora'' channel. Once this has been completed QA needs to complete the following tasks:

* Comment in the bug to confirm updates have been disabled and mark the bug VERIFIED FIXED

Following the migration, QA has four days to test and sign-off Aurora builds before updates are enabled. The following tasks should be performed before the Friday following the release.

Once you have the confidence of the builds and updates, communicate your sign-off to Release Drivers. This should be completed by Friday morning if at all possible.

On the Friday following every release updates to Aurora builds will be re-enabled on the ''aurora'' channel. After QA has given their OK to enabled the updates, Release Management will file the bug, Release Engineering will push the changes, and QA needs to verify the changes.

* Comment in the bug to confirm updates have been re-enabled and mark the bug VERIFIED FIXED

On the Monday prior to release Aurora will be migrated to Beta. This happens without QA sign-off so make sure the following tasks are completed well in advance:

Once the migration happens, the first Beta build will be generated. Testing of this build can begin on Tuesday after the releases have been shipped and must be completed by Thursday. See the Beta section for more details.

* Day 0 (Monday)

** Release Management will migrate mozilla-aurora to mozilla-beta

** Release Management migrates mozilla-aurora to mozilla-beta

The following defines the tasks and work involved in a typical "sign-off".

All release testplans follow the same basic format:

* Checklist

</pre>

Functional automation can be run as soon as all candidate builds are available (~10 hours from go-to-build). To execute the automation you'll want to upload a manifest to the mozmill-ondemand server. You will want to test en-US plus one random locale per platform.

If you have any failures you'll need to dig into those to see if they are known failures or not. You should be able to tell by doing a bugzilla query agains the Mozmill Tests component. Otherwise, ask someone in #automation for assistance.

Very similar to Functional Automation, you'll want to upload a manifest to test updates. The difference here is that you'll want to indicate the update channel, target build ID and source versions you want to test.

Add a link to this report and update the results to your test matrix in the test plan.

Depending on the update strategy upon release it is reasonable to expect we need to test update throttling. If updates are at 100% you should get a background update, 0% you should NOT get a background update; you can use [https://moztrap.mozilla.org/manage/case/8223/ this Moztrap test] to check that. If you need to test an intermediary throttling percentage use [https://moztrap.mozilla.org/manage/case/8213/ this Moztrap test] instead.

This will only need to be done for updates on ''release'' channel and only at Release Management's request.

Manual testing is typically performed by our Softvision team in Romania. What needs testing in a given build is guided by our [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Beta_Checklist Beta Checklist]. Additionally there may be requirements to check high profile features or high risk bug fixes. There's no real "how-to" here, just use your best judgement.

Bug fix verifications are guided by bug queries and mercurial pushlogs. There's no real "rule" for what should and shouldn't be tested. You'll need to read the bugs and assess whether the change is significant enough to warrant testing. The basic litmus test is if the landed patch carries high risk (usually indicated in the uplift approval request) or if the bug is specifically called out in the release notes.

Any bugs you think warrant verification before a release is shipped should be given the ''verifyme'' keyword. Include a query to these verifyme marked bugs and Softvision will test them as part of their sign-off work.

QAWANTED bug investigations are ongoing, high priority bugs which need QA testing. For the most part these are not fixed and need regression windows, steps to reproduce, minimized testcases, or URLs so developers can investigate a fix in a timely manner.  

It is recommended that you track these bugs in the testplan by query and by providing a list of the bugs with their current status and assignee. This list should be revisited on a daily basis.

While not necessarily a required part of signing off a release it is good practice to make sure some of the reported bugs have been triaged. Softvision runs a weekly triage session with the community which will typically go through some but not all of these bugs. It is important to triage these bugs to A) move them into the right component so developers are aware, and B) to become aware of serious regressions before we release thus mitigating a chemspill release.

Basically you'll want to triage for UNCONFIRMED bugs with either the version matching your release or the tracking/status flag set to ?. Providing a query to these bugs in your testplan is a good way to get community involved. Any bugs confirmed should be listed in your testplan and escalated to Release Management either by flagging it tracking?, emailing Release Management, or raising it in the Tuesday/Thursday Release Coordination meetings.

Stability analysis is fairly straightforward. Simply ask one of the Stability team to review crash-stats and flag any stability concerns related to whichever version you are trying to sign-off. This should be done last, before sending the sign-off to push updates live, to allow for enough time to pass to collect data from early adopters.

Once you have confidence in the quality of the product you need to send an email to Release Drivers. In general you will need to send two sign-off emails: one to signal pushing updates to the release channel, another to sign-off those live updates. The exception is that Release builds need a conditional sign-off before pushing updates to mirrors (ie. releasetest).  

</pre>

* Day 0 (Tuesday)

** Release Management migrates the mozilla-beta branch to mozilla-release

** Release Management migrates mozilla-beta to mozilla-release and sends the go to build the next Release

* Day 0..36

** QA focuses on verifying most critical and security fixes using the nightly builds

** QA runs Mozmill-ondemand update automation on the ''release'' channel then sends a sign-off email to Release Drivers when complete