QA/Desktop Firefox/Walkthroughs/Release Coordination: Difference between revisions

Jump to navigation Jump to search

QA/Desktop Firefox/Walkthroughs/Release Coordination (view source)

Revision as of 22:58, 15 April 2014

12,942 bytes added ,  15 April 2014
→‎Priorities
Line 403: Line 403:


== Priorities ==
== Priorities ==
The following defines the tasks and work involved in a typical "sign-off".
=== Developing a Test Plan ===
All release testplans follow the same basic format:
* Checklist
* Functional Automation
* Manual Testing
* Update Automation
* Bug fix verifications
Once Release Management has sent the go to build you should prepare the test plan. You can see an example of a test plan [https://wiki.mozilla.org/Releases/Firefox_21/Test_Plan#Beta_4 here]. Once you've generated the test plan send an email to Softvision to inform them of the testing required and the deadline to sign-off. Be sure to indicate any high-profile/high-risk areas to focus on.
Here is an example email:
<pre>
Subject: Firefox 21.0b4 Testing
Message: Release Management has sent the go to build our next Beta.
        You should start seeing builds on the FTP server in a few hours.
        Once the builds are available please begin testing as indicated
        by the test plan[1]. Please pay particular attention to Plugins
        as there were several changes to the plugins backend which carry
        a moderate level of risk for regression. Our plan is to sign-off
        and ship these builds by Thursday morning.
        Thank you
        1. https://wiki.mozilla.org/Releases/Firefox_21/Test_Plan#Beta_4
</pre>
=== Functional Automation ===
Functional automation can be run as soon as all candidate builds are available (~10 hours from go-to-build). To execute the automation you'll want to upload a manifest to the mozmill-ondemand server. You will want to test en-US plus one random locale per platform.
Here is an example manifest:
<pre>
[testrun]
script=functional
report=http://mozauto.iriscouch.com/mozmill-release/
[linux ubuntu 12.04 32bit]
platform=linux
21.0#1=en-US pt-BR
[linux ubuntu 12.04 64bit]
platform=linux64
21.0#1=en-US fi
[linux ubuntu 12.10 32bit]
platform=linux
21.0#1=zh-TW pt-BR
[linux ubuntu 12.10 64bit]
platform=linux64
21.0#1=es-MX fi
[windows 7 32bit]
platform=win32
21.0#1=fi da
[windows 7 64bit]
platform=win32
21.0#1=da it
[windows 8 32bit]
platform=win32
21.0#1=it ru
[windows 8 64bit]
platform=win32
21.0#1=ru es-ES
[windows vista 32bit]
platform=win32
21.0#1=ro ko
[windows xp 32bit]
platform=win32
21.0#1=ko ro
[mac 10.6 64bit]
platform=mac
21.0#1=es-ES de
[mac 10.7 64bit]
platform=mac
21.0#1=fr es-ES
[mac 10.8 64bit]
platform=mac
21.0#1=de fr
</pre>
To execute the testrun you'll want to follow these steps:
* Connect to the MTP VPN
* Load http://mm-ci-master.qa.scl3.mozilla.com:8080/
* Once the Jenkins interface loads click the admin tab
* Click the green arrow button for ''trigger-ondemand''
* Upload the config file you've created (document must be in plain text)
Once uploaded go back to the main dashboard, you should see a bunch of ondemand functional testruns queue up. The testruns will take ~40 minutes to execute. Once they finish, check your mozmill-ci email for any failures.
If something goes wrong, please get in touch with ashughes, davehunt, or whimboo in the #automation channel on IRC.
If everything went okay you can look up the results on the [http://mozmill-ondemand.blargon7.com mozmill-release dashboard]. Filter by ''Functional Tests > Reports'', ''Branch'' matching the version tested, ''OS'' matching ''All'', and a date range matching the last 24-hours. You should see a list of results matching the testruns. Check for 0 failures in all rows.
If you have any failures you'll need to dig into those to see if they are known failures or not. You should be able to tell by doing a bugzilla query agains the Mozmill Tests component. Otherwise, ask someone in #automation for assistance.
=== Update Automation ===
Very similar to Functional Automation, you'll want to upload a manifest to test updates. The difference here is that you'll want to indicate the update channel, target build ID and source versions you want to test.
Here are the channels you'll need to test:
* Beta: betatest & releasetest before sign-off, beta after sign-off
* ESR: esrtest & releasetest before sign-off, esr after sign-off
* Release: betatest before conditional sign-off, releasetest before final sign-off, release after sign-off
Here is an example manifest:
<pre>
[testrun]
script=update
report=http://mozauto.iriscouch.com/mozmill-release/
target-build-id=20130423212553
channel=betatest
[mac 10.6 64bit]
platform=mac
19.0b1=en-US de
20.0b1=en-US ja-JP-mac
21.0b1=en-US it
21.0b3=en-US es-ES
[mac 10.7 64bit]
platform=mac
19.0b2=en-US pt-BR
20.0b2=en-US fr
21.0b2=en-US ru
21.0b3=en-US zh-TW
[mac 10.8 64bit]
platform=mac
19.0b3=en-US it
20.0b3=en-US ro
21.0b2=en-US de
21.0b3=en-US ko
[linux ubuntu 32bit]
platform=linux
19.0b4=en-US de
20.0b4=en-US ko
21.0b1=en-US zh-TW
21.0b3=en-US ru
[linux ubuntu 64bit]
platform=linux64
19.0b5=en-US es-ES
20.0b5=en-US pt-BR
21.0b2=en-US de
21.0b3=en-US it
[windows 8 32bit]
platform=win32
19.0b6=en-US es-ES
20.0b6=en-US zh-TW
21.0b1=en-US ro
21.0b3=en-US da
[windows 8 64bit]
platform=win32
19.0b1=en-US fr
20.0b1=en-US it
21.0b2=en-US es-ES
21.0b3=en-US zh-TW
[windows 7 32bit]
platform=win32
19.0b2=en-US es-ES
20.0b2=en-US zh-TW
21.0b1=en-US fr
21.0b3=en-US da
[windows 7 64bit]
platform=win32
19.0b3=en-US es-ES
20.0b3=en-US de
21.0b2=en-US ko
21.0b3=en-US ro
[windows vista 32bit]
platform=win32
19.0b4=en-US de
20.0b4=en-US pt-BR
21.0b1=en-US it
21.0b3=en-US es-ES
[windows xp 32bit]
platform=win32
19.0b5=en-US fr
20.0b5=en-US ko
21.0b2=en-US de
21.0b3=en-US ru
</pre>
To execute the testrun you'll want to follow these steps:
* Connect to the MTP VPN
* Load http://mm-ci-master.qa.scl3.mozilla.com:8080/
* Once the Jenkins interface loads click the admin tab
* Click the green arrow button for ''trigger-ondemand''
* Upload the config file you've created
Once uploaded go back to the main dashboard, you should see a bunch of ondemand update testruns queue up. The testruns will take ~40 minutes to execute. Once they finish, check your mozmill-ci email for any failures.
If something goes wrong, please get in touch with ashughes, davehunt, or whimboo in the #automation channel on IRC.
If everything went okay you can look up the results on the [http://mozmill-ondemand.blargon7.com mozmill-release dashboard]. Filter by ''Update Tests > Overview'', ''Branch'' matching the version tested, ''OS'' matching ''All'', ''Channel'' matching the channel you tested, and a date range matching the last 24-hours. You should see a list of results matching the testruns. Check for 0 failures in all rows.
Add a link to this report and update the results to your test matrix in the test plan.
==== Update Throttling ====
Depending on the update strategy upon release it is reasonable to expect we need to test update throttling. If updates are at 100% you should get a background update, 0% you should NOT get a background update; you can use [https://moztrap.mozilla.org/manage/case/8223/ this Moztrap test] to check that. If you need to test an intermediary throttling percentage use [https://moztrap.mozilla.org/manage/case/8213/ this Moztrap test] instead.
This will only need to be done for updates on ''release'' channel and only at Release Management's request.
=== Manual Testing ===
Manual testing is typically performed by our Softvision team in Romania. What needs testing in a given build is guided by our [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Beta_Checklist Beta Checklist]. Additionally there may be requirements to check high profile features or high risk bug fixes. There's no real "how-to" here, just use your best judgement.
=== Bug Fix Verifications ===
Bug fix verifications are guided by bug queries and mercurial pushlogs. There's no real "rule" for what should and shouldn't be tested. You'll need to read the bugs and assess whether the change is significant enough to warrant testing. The basic litmus test is if the landed patch carries high risk (usually indicated in the uplift approval request) or if the bug is specifically called out in the release notes.
When searching for bugs by query you'll want to search against Target Milestone, the tracking-firefoxN:+ flag, and the status-firefoxN:fixed flag, where ''N'' is the version of Firefox you are testing.
When searching for bugs by pushlog you'll want to query between the changeset ID of the current build and the changeset ID of the previous build. For example, the pushlog for Firefox 21 Beta 4 searches from the Beta 3 changeset (e845a10035f2) to the Beta 4 changeset (04aba2e6927f): http://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=e845a10035f2&tochange=04aba2e6927f
Any bugs you think warrant verification before a release is shipped should be given the ''verifyme'' keyword. Include a query to these verifyme marked bugs and Softvision will test them as part of their sign-off work.
=== QAWANTED Bug Investigations ===
QAWANTED bug investigations are ongoing, high priority bugs which need QA testing. For the most part these are not fixed and need regression windows, steps to reproduce, minimized testcases, or URLs so developers can investigate a fix in a timely manner.
Simply querying bugzilla for bugs with the ''qawanted'' keyword and the tracking-firefoxN:+ flag set for the current version you are testing should catch most of these. For the most part Release Management will raise these in the Release Coordination meetings or via email, especially if it's critical.
The majority of your effort should be spent trying to nail down these bugs and get them resolved before release. As the release day approaches a judgement call will need to be made to either ship with the known bug or to block the release. While this decision is ultimately Release Management's decision, your judgement of the impact on "quality" is essential.
It is recommended that you track these bugs in the testplan by query and by providing a list of the bugs with their current status and assignee. This list should be revisited on a daily basis.
=== UNCONFIRMED Bug Investigations ===
While not necessarily a required part of signing off a release it is good practice to make sure some of the reported bugs have been triaged. Softvision runs a weekly triage session with the community which will typically go through some but not all of these bugs. It is important to triage these bugs to A) move them into the right component so developers are aware, and B) to become aware of serious regressions before we release thus mitigating a chemspill release.
Basically you'll want to triage for UNCONFIRMED bugs with either the version matching your release or the tracking/status flag set to ?. Providing a query to these bugs in your testplan is a good way to get community involved. Any bugs confirmed should be listed in your testplan and escalated to Release Management either by flagging it tracking?, emailing Release Management, or raising it in the Tuesday/Thursday Release Coordination meetings.
=== Stability Analysis ===
Stability analysis is fairly straightforward. Simply ask one of the Stability team to review crash-stats and flag any stability concerns related to whichever version you are trying to sign-off. This should be done last, before sending the sign-off to push updates live, to allow for enough time to pass to collect data from early adopters.
=== Sending the Sign-off ===
Once you have confidence in the quality of the product you need to send an email to Release Drivers. In general you will need to send two sign-off emails: one to signal pushing updates to the release channel, another to sign-off those live updates. The exception is that Release builds need a conditional sign-off before pushing updates to mirrors (ie. releasetest).
Here are some example emails to illustrate.
; Conditional sign-off
''Only applies for Release Candidate builds -- ESR and Beta releases are pushed to mirrors automatically''
<pre>
Subject: [Desktop] QA signs off Firefox 22 for push to mirrors
Message: QA has finished all functional testing and bug fix verifications.
        Having found no serious regressions or blockers for release we feel
        it is safe to push the snippets to mirrors. Please push snippets at
        Release Management's go.
</pre>
; Release Sign-off
''Applies to Release, Beta, and ESR builds''
<pre>
Subject: [Desktop] QA signs off Firefox 22 for release
Message: QA has finished all functional testing, bug fix verifications,
        and update tests on releasetest. Having found no serious regressions
        nor blockers we feel it is safe to release these builds. Please push
        snippets live at Release Management's go.
</pre>
; Live Update Sign-off
''Applies to Release, Beta, and ESR builds''
<pre>
Subject: [Desktop] QA signs off Firefox 22 updates on release channel
Message: QA has finished testing updates to Firefox 22 on the release
        channel and everything is working as expected.
</pre>
Ashughes
Confirmed users
14,525

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/965211"

Navigation menu