Releases/Firefox 30/Test Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(175 intermediate revisions by 8 users not shown)
Line 2: Line 2:


= Firefox 30 Test Plan =
= Firefox 30 Test Plan =
''QA Lead: Tracy Walker''
== Summary ==
{| class="fullwidth-table"
|-
| style="width:15%" | '''Milestone'''
| [https://wiki.mozilla.org/Firefox/Flight_Tracking#Firefox_30 Firefox 30]
|-
| '''Status'''
| Currently in '''Beta'''
|-
| '''QA-Lead'''
| [mailto:twalker@mozilla.com Tracy Walker] (irc: tracy)
|-
| '''QA-2nd'''
| [mailto:kairo@mozilla.com Robert Kaiser] (irc: KaiRo)
|-
| '''Rel-Eng contact'''
| [mailto:bhearsum@mozilla.com Ben Hearsum] (irc: bhearsum)
|-
| '''Rel-Man contact'''
| [mailto:lsblakk@mozilla.com Lukas Blakk] (irc: lsblakk)
|}
 
The following is the test plan for Firefox 30 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 30. After the release this document will be used as an archive of what was done to validate this release.
 
''If you have some free time, please pick a task below and get in touch with one of the leads.''


== Schedule ==
== Schedule ==
* 2014.02.03: {{ok|}} Nightly
* 2014.02.03: {{done|}} Nightly
* 2014.03.17: Aurora
* 2014.03.17: {{done|}} Aurora
* 2014.04.25: Pre-Beta feature sign-offs
* 2014.04.25: {{done|}} Pre-Beta feature sign-offs
* 2014.04.29: Beta 1
* 2014.04.29: {{done|}} Beta 1
* 2014.05.06: Beta 2
* 2014.05.06: {{done|}} Beta 2
* 2014.05.08: Beta 3
* 2014.05.09: {{done|}} Beta 3
* 2014.05.13: Beta 4
* 2014.05.14: {{done|}} Beta 4
* 2014.05.15: Beta 5
* 2014.05.16: {{done|}} Beta 5
* 2014.05.20: Beta 6
* 2014.05.21: {{done|}} Beta 6
* 2014.05.22: Beta 7
* 2014.05.23: {{done|}} Beta 7
* 2014.05.27: Beta 8
* 2014.05.28: {{done|}} Beta 8
* 2014.05.29: Beta 9
* 2014.05.30: {{done|}} Beta 9
* 2014.06.03: Beta 10
* 2014.06.04: {{done|}} Beta 10 (RC build 1)
* 2014.06.05: Release Candidate
* 2014.06.06: {{done|}} RC (build 2)
* 2014.06.10: Final
* 2014.06.10: {{done|}} Final
 
== Nightly ==
=== Features ===
''[https://wiki.mozilla.org/Features/Release_Tracking#Targeting_Firefox_30 Feature Tracking]''


''Tentative - we'll be switching to 2 week sprint model for a portion of desktop Fx work and will link to those sprint tracks here''
== Features ==
=== Disabled ===
The following features have been deemed not ready for release and will be delayed to Firefox 31 or later.
* [https://bugzilla.mozilla.org/show_bug.cgi?id=967028 Session Store Changes] (Alexandra Lucinet)
* [https://bugzilla.mozilla.org/buglist.cgi?bug_id=976552%2C%20976545%2C%20979424%2C%20974460&bug_id_type=anyexact&query_format=advanced&list_id=9750193 Translation] (Bogdan Maris)
* [https://bugzilla.mozilla.org/buglist.cgi?bug_id=959571%2C%20959576%2C%20959579%2C%20959558%2C%20557665%2C%20959559%2C%20959572%2C%20959573%2C%20962490&bug_id_type=anyexact&query_format=advanced&list_id=9750192 Search] (Petruta Rasa)
* [https://bugzilla.mozilla.org/buglist.cgi?bug_id=951627%20%2C%20629911%2C%20616853%2C%20613785&bug_id_type=anyexact&query_format=advanced&list_id=9729222 Modal/non-Modal Dialogs] (Manuela Muntean)
* [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=979280%2C988070%2C992901%2C993391%2C984904%2C984903%2C984902%2C982115%2C818584%2C951651%2C937560%2C894331%2C937560%2C885241%2C519514%2C625778%2C834545&list_id=10333062 Places] (Andrei Vaida)


; Sign-off Criteria
=== On Track ===
* All user stories covered by manual or automated tests
* Manual testing complete with no blocking regressions
* Automated testing complete with no blocking regressions
* 0 untriaged unconfirmed bugs
* 0 untriaged unverified fixes


; Status
{| class="querytable"
| class="header" | Feature
| class="header" | Owner
| class="header" | Documentation
| class="header" | Tracked Bugs
| class="header" | Aurora Merge Sign-off
| class="header" | Pre-beta Sign-off
|-
|}


;Notes.
=== Signed Off ===
The following features have no blockers and have been deemed ready for release.
* [https://bugzilla.mozilla.org/show_bug.cgi?id=806917 GStreamer 1.0 Support] (Bogdan Maris)
* [https://bugzilla.mozilla.org/show_bug.cgi?id=910670 Preference UI for proxy autologin] (Bogdan Maris)
* [https://bugzilla.mozilla.org/show_bug.cgi?id=970600 Background-blend-mode] (Catalin Varga)


=== Fixed Bugs ===
= Nightly =
== Fixed Bugs ==
; How to Help
; How to Help
* [https://nightly.mozilla.org Download] and install the latest Firefox Nightly
* [https://nightly.mozilla.org Download] and install the latest Firefox Aurora build
* [http://mzl.la/1kRtjOM Load] this list of bug reports
* [http://mzl.la/1loH5vd Load] this list of bug reports
* Read each bug report until you find one you understand and want to test
* Read each bug report until you find one you understand and want to test
* Download and install the version of Firefox used to report the bug
* Download and install the version of Firefox used to report the bug
Line 64: Line 81:
*
*


=== Reported Bugs ===
== Reported Bugs ==
Triage unconfirmed bugs.  
Triage unconfirmed bugs.  


Line 79: Line 96:
*
*


=== Stability ===
== Stability ==
; How to Help
; How to Help
* [https://crash-analysis.mozilla.com/rkaiser/0000.overview.html Review] the Firefox 30 Explosiveness Report
* [https://crash-analysis.mozilla.com/rkaiser/0000.overview.html Review] the Firefox 30 Explosiveness Report
Line 101: Line 118:
; Topcrash Bugs
; Topcrash Bugs


=== QA Wanted ===
== QA Wanted ==
; How to Help
; How to Help
* [https://nightly.mozilla.org Download] and install the latest Firefox Nightly
* [https://nightly.mozilla.org Download] and install the latest Firefox Nightly
Line 115: Line 132:
*
*


=== Testdays ===
== Testdays ==
 
= Aurora =
== Branch Migration ==
=== Pre Sign-off Tasks ===
* {{done|}} Firefox branch migrated (Release Management)
* {{done|}} Mozmill Tests branch migrated (Automation Development)
* {{done|}} Updates throttled on ''aurora'' channel (Release Engineering)
* {{done|}} Updates verified throttled on ''aurora'' channel (QA)
 
=== Sign-off Tasks ===
* {{done|}} Mozmill Daily functional testruns pass with no failures ([report])
* {{done|}} Mozmill Daily ''auroratest'' update testruns pass with no failures ([report])
* {{done|}} Scope features tested and signed off for Aurora uplift
 
=== Post Sign-off Tasks ===
* {{done|}} Aurora sign-off email sent (QA)
* {{done|}} Updates unthrottled on ''aurora'' channel (Release Engineering)
* {{done|}} Updates verified unthrottled on ''aurora'' channel (QA)
 
= Beta =
=== Regression Testing ===
* Run [https://moztrap.mozilla.org/runtests/environment/3975/ these tests] for whatever platform(s) you have available.
* Report a bug if you encounter one.
* If you report a bug check to see if it's a regression by testing a previous Firefox version (if it doesn't reproduce in earlier versions it's a regression).
* Add any reported bugs to the Bugs Found section below.
* All results will appear in [https://moztrap.mozilla.org/results/cases/?filter-run=3975 this dashboard]
 
== Beta 1 ==
=== Build Info ===
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0b1-candidates/ build1]
* Build ID: 20140428174145
* Changes: [https://hg.mozilla.org/releases/mozilla-beta/log/192165 Changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30b1 Mozmill Configs]
 
=== Check List ===
* {{done|}} Regression testing ([https://moztrap.mozilla.org/results/cases/?filter-run=3983 report]) - FAIL
** regression found: {{bug|1003839}} - Live bookmark is still created despite cancelling the subscription
*** shipped to beta1 with this regression. it is tracked by Rel-Man to ensure it is fixed prior to final 30 release.
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?branch=30.0&platform=All&from=2014-04-28&to=2014-04-29 report]) - PASS
* {{done|}} Updates on ''betatest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=betatest&from=2014-04-29&to=2014-04-29&target=30.0 report]) - PASS
* {{done|}} Updates on ''releasetest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-04-29&to=2014-04-29&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-05-01&to=2014-05-01&target=30.0 report])
** {{Bug|1004575}} - ondemand_update test runs stalling out during recording results.
 
=== Regression Testing ===
* Please run a brief smoketest for the [https://wiki.mozilla.org/Releases/Firefox_30/Test_Plan#Signed_Off features] signed off on prior to merge to ensure they continue to be ready for our beta audience.
** GStreamer 1.0 Support (Alexandra Lucinet)
** Preference UI for proxy autologin (Bogdan Maris)
** Places (Andrei Vaida)
** Background-blend-mode (Catalin Varga)
 
== Beta 2 ==
=== Build Info ===
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0b2-candidates/ build1]
* Build ID: 20140505140302
* Changes: [https://hg.mozilla.org/releases/mozilla-beta/log/192201 Changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30b2 Mozmill Configs]
 
=== Check List ===
* {{done|}} Regression testing ([https://moztrap.mozilla.org/results/cases/?filter-run=3983 moztrap report], [https://etherpad.mozilla.org/Fx-30b2 Exploratory testing]) - PASS
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=30.0&platform=All&from=2014-05-05&to=2014-05-06 report]) - PASS
* {{done|}} Updates on ''betatest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=betatest&from=2014-05-06&to=2014-05-06&target=30.0 report]) - PASS
* {{done|}} Updates on ''releasetest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-05-06&to=2014-05-06&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-05-06&to=2014-05-06&target=30.0 report]) - PASS
 
=== Regression Testing ===
* [http://mzl.la/1seSSNo Fix Verifications], focusing on those most recently fixed during this cycle.
* [https://etherpad.mozilla.org/Fx-30b2 Exploratory testing] around Session Restore, Australis.
 
== Beta 3 ==
* shipped with tested failures note in regression testing reports.  Further investigation is needed, but we'll ensure bugs are filed and tracked.
 
=== Build Info ===
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0b3-candidates/ build1]
* Build ID: 20140508121358
* Changes: [http://hg.mozilla.org/releases/mozilla-beta/log/192228 Changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30b3 Mozmill Configs]
 
=== Check List ===
* {{done|}} Regression testing ([https://moztrap.mozilla.org/results/cases/?filter-run=3983 moztrap report], [https://etherpad.mozilla.org/Fx-30b3 exploratory testing]) - FAIL
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=30.0&platform=All&from=2014-05-08&to=2014-05-09 report]) - PASS
* {{done|}} Updates on ''betatest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=betatest&from=2014-05-09&to=2014-05-09&target=30.0 report]) - PASS
* {{done|}} Updates on ''releasetest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-05-09&to=2014-05-09&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-05-09&to=2014-05-09&target=30.0 report]) - PASS
 
=== Regression Testing ===
* [http://mzl.la/1seSSNo Fix Verifications], focusing on those most recently fixed during this cycle.
* [https://etherpad.mozilla.org/Fx-30b3 Exploratory testing] around Safe Browsing, Themes.
 
== Beta 4 ==
 
=== Build Info ===
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0b4-candidates/ build1]
* Build ID: 20140512231802
* Changes: [https://hg.mozilla.org/releases/mozilla-beta/log/192256 Changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30b4 Mozmill Configs]
 
=== Check List ===
* {{done|}} Regression testing ([https://moztrap.mozilla.org/results/cases/?filter-run=3983 moztrap report], [https://etherpad.mozilla.org/Fx-30b4 exploratory testing]) - PASS
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=30.0&platform=All&from=2014-05-13&to=2014-05-13 report]) - PASS
* {{done|}} Updates on ''betatest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=betatest&from=2014-05-13&to=2014-05-13&target=30.0 report]) - PASS
* {{done|}} Updates on ''releasetest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-05-13&to=2014-05-13&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-05-14&to=2014-05-14&target=30.0 report]) - PASS
 
=== Regression Testing ===
* [http://mzl.la/1seSSNo Fix Verifications], focusing on those most recently fixed during this cycle.
* [https://etherpad.mozilla.org/Fx-30b4 Exploratory testing] around Social API, FxA & Sync, Toolbar customization.
 
== Beta 5 ==
=== Build Info ===
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0b5-candidates/ build1]
* Build ID: 20140515140857
* Changes: [https://hg.mozilla.org/releases/mozilla-beta/log/192297 Changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30b5 Mozmill Configs]
 
=== Check List ===
* {{done|}} Regression testing ([https://moztrap.mozilla.org/results/cases/?filter-run=3983 moztrap report]) - PASS
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=30.0&platform=All&from=2014-05-15&to=2014-05-16 report]) - PASS
* {{done|}} Updates on ''betatest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=betatest&from=2014-05-16&to=2014-05-16&target=30.0 report]) - PASS
* {{done|}} Updates on ''releasetest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-05-16&to=2014-05-16&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-05-16&to=2014-05-16&target=30.0 report]) - PASS
 
=== Regression Testing ===
* [http://mzl.la/1seSSNo Fix Verifications], focusing on those most recently fixed during this cycle.
 
== Beta 6 ==
=== Build Info ===
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0b6-candidates/build2/ build2]
* Build ID: 20140520115057
* Changes: [https://hg.mozilla.org/releases/mozilla-beta/log/192333 changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30b6 Mozmill Configs]
 
=== Check List ===
* {{done|}} Regression testing ([http://mzl.la/1seSSNo Fix Verifications]) - PASS
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=30.0&platform=All&from=2014-05-20&to=2014-05-21 report]) - PASS
* {{done|}} Updates on ''betatest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=betatest&from=2014-05-21&to=2014-05-21&target=30.0 report]) - PASS
* {{done|}} Updates on ''releasetest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-05-21&to=2014-05-21&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-05-21&to=2014-05-21&target=30.0 report]) - PASS
 
=== Regression Testing ===
* [http://mzl.la/1seSSNo Fix Verifications], focusing on those most recently fixed during this cycle.
 
== Beta 7 ==
=== Build Info ===
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0b7-candidates/build1/ build1]
* Build ID: 20140522105902
* Changes: [https://hg.mozilla.org/releases/mozilla-beta/log/192368 changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30b7 Mozmill Configs]
 
=== Check List ===
* {{done|}} Regression testing ([http://mzl.la/1seSSNo Fix Verifications]) - PASS
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=30.0&platform=All&from=2014-05-22&to=2014-05-23 report]) - PASS
* {{done|}} Updates on ''betatest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=betatest&from=2014-05-23&to=2014-05-23&target=30.0 report]) - PASS
* {{done|}} Updates on ''releasetest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-05-23&to=2014-05-23&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-05-23&to=2014-05-23&target=30.0 report]) - PASS
 
=== Regression Testing ===
* [http://mzl.la/1seSSNo Fix Verifications], focusing on those most recently fixed during this cycle.
** {{Bug|980714}} - Blinking caret in panel text - VERIFIED
** {{Bug|1003839}} - Live bookmark is still created despite cancelling the subscription. - VERIFIED
 
== Beta 8 ==
=== Build Info ===
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0b8-candidates/build1/ build1]
* Build ID: 20140527133511
* Changes: [https://hg.mozilla.org/releases/mozilla-beta/log/192393 changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30b8 Mozmill Configs]
 
=== Check List ===
* {{done|}} Regression testing ([https://moztrap.mozilla.org/runtests/environment/3975/ Beta Regression tests], [http://mzl.la/1seSSNo Fix Verifications]) - PASS
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=30.0&platform=All&from=2014-05-27&to=2014-05-28 report]) - PASS
* {{done|}} Updates on ''betatest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=betatest&from=2014-05-28&to=2014-05-28&target=30.0 report]) - PASS
* {{done|}} Updates on ''releasetest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-05-28&to=2014-05-28&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-05-28&to=2014-05-28&target=30.0 report]) - PASS
 
=== Regression Testing ===
* [https://moztrap.mozilla.org/runtests/environment/3975/ Beta Regression tests]
* [http://mzl.la/1seSSNo Fix Verifications], focusing on those most recently fixed during this cycle.
 
== Beta 9 ==
=== Build Info ===
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0b9-candidates/build1/ build1]
* Build ID: 20140529161749
* Changes: [https://hg.mozilla.org/releases/mozilla-beta/log/4985ad6eb2e1 changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30b9 Mozmill Configs]
 
=== Check List ===
* {{done|}} Regression testing ([https://moztrap.mozilla.org/runtests/environment/3975/ Beta Regression tests], [https://moztrap.mozilla.org/runtests/environment/4203/ PDF tests], [https://etherpad.mozilla.org/Fx-30b9 Shutdown hangs Exploratory testing], [http://mzl.la/1seSSNo Fix Verifications]) - PASS
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=30.0&platform=All&from=2014-05-29&to=2014-05-30 report]) - PASS
* {{done|}} Updates on ''betatest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=betatest&from=2014-05-30&to=2014-05-30&target=30.0 report]) - PASS
* {{done|}} Updates on ''releasetest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-05-30&to=2014-05-30&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-05-30&to=2014-05-30&target=30.0 report]) - PASS
 
=== Regression Testing ===
* [https://moztrap.mozilla.org/runtests/environment/3975/ Beta Regression tests]
* [https://moztrap.mozilla.org/runtests/environment/4203/ PDF tests] - issues found so far are documented [https://etherpad.mozilla.org/Fx-30b9 here].
* [https://etherpad.mozilla.org/Fx-30b9 Shutdown hangs Exploratory testing]
* [http://mzl.la/1seSSNo Fix Verifications], focusing on those most recently fixed during this cycle.
 
= Release =
== Release Candidate Build 1 ==
=== Build Info ===
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0-candidates/build1/ build1]
* Build ID: 20140603140158
* Changes: [https://hg.mozilla.org/releases/mozilla-release/log/193459 changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30RC Mozmill Configs]
 
=== Check List ===
* {{done|}} Regression testing ([https://moztrap.mozilla.org/runtests/environment/3975/ Beta Regression tests], [https://moztrap.mozilla.org/runtests/environment/4203/ PDF tests], [http://mzl.la/1seSSNo Fix Verifications]) - PASS
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=30.0&platform=All&from=2014-06-03&to=2014-06-04 report]) - PASS
* {{done|}} Updates on ''releasetest'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-06-04&to=2014-06-04&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-06-04&to=2014-06-04&target=30.0 report]) - PASS


== Aurora ==
=== Regression Testing ===
=== Branch Migration ===
* [https://moztrap.mozilla.org/runtests/environment/3975/ Beta Regression tests]
==== Pre Sign-off Tasks ====
* [https://moztrap.mozilla.org/runtests/environment/4203/ PDF tests]
* {{|}} Firefox branch migrated (Release Management)
* [http://mzl.la/1seSSNo Fix Verifications], focusing on the [https://bugzilla.mozilla.org/show_bug.cgi?id=1003707 fix for PDF printing]
* {{|}} Mozmill Tests branch migrated (Automation Development)
* {{|}} Updates throttled on ''aurora'' channel (Release Engineering)
* {{|}} Updates verified throttled on ''aurora'' channel (QA)


==== Sign-off Tasks ====
== Release Candidate Build 2 ==
* {{|}} Mozmill Daily functional testruns pass with no failures ([report])
=== Build Info ===
* {{|}} Mozmill Daily ''auroratest'' update testruns pass with no failures ([report])
* Builds: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/30.0-candidates/build2/ build2]
* {{|}} Scope features tested and signed off for Aurora uplift
* Build ID: 20140605174243
** feature...
* Changes: [https://hg.mozilla.org/releases/mozilla-release/log/193462 changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx30RC Mozmill Configs]


==== Post Sign-off Tasks ====
=== Check List ===
* Aurora sign-off email sent (QA)
* {{done|}} [https://moztrap.mozilla.org/results/cases/?filter-run=4262 Sanity Check] - PASS
* Updates unthrottled on ''aurora'' channel (Release Engineering)
* {{done|}} Automation Functional tests ([http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=30.0&platform=All&from=2014-06-05&to=2014-06-06 report]) - PASS
* Updates verified unthrottled on ''aurora'' channel (QA)
* {{done|}} Updates on ''releasetest'' channel beta builds ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-06-05&to=2014-06-06&target=30.0 report]) - PASS
* {{done|}} Updates on ''beta'' channel ([http://mozmill-release.blargon7.com/#/update/detail?&channel=beta&from=2014-06-05&to=2014-06-06&target=30.0 report]) - PASS


== Beta ==
=== Regression Testing ===
* [https://moztrap.mozilla.org/results/cases/?filter-run=4262 Sanity Check]


== Bugs by Status ==
== Final Release ==
* {{done|}} Updates on ''releasetest'' channel release builds ([http://mozmill-release.blargon7.com/#/update/detail?&channel=releasetest&from=2014-06-10&to=2014-06-10&target=30.0 report]) - PASS
** {{done|}}Spot checks for {{bug|1009893}} - Show /whatsnew tour URL to users updating to 30.0 and 31.0 (from all past versions of Firefox) - PASS
* {{done|}} Updates on ''release'' channel release builds ([http://mozmill-release.blargon7.com/#/update/detail?&channel=release&from=2014-06-10&to=2014-06-10&target=30.0 report]) - PASS

Latest revision as of 16:13, 10 June 2014

« Firefox 29 | Firefox 30 | Firefox 31 »

Firefox 30 Test Plan

Summary

Milestone Firefox 30
Status Currently in Beta
QA-Lead Tracy Walker (irc: tracy)
QA-2nd Robert Kaiser (irc: KaiRo)
Rel-Eng contact Ben Hearsum (irc: bhearsum)
Rel-Man contact Lukas Blakk (irc: lsblakk)

The following is the test plan for Firefox 30 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 30. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

  • 2014.02.03: [DONE] Nightly
  • 2014.03.17: [DONE] Aurora
  • 2014.04.25: [DONE] Pre-Beta feature sign-offs
  • 2014.04.29: [DONE] Beta 1
  • 2014.05.06: [DONE] Beta 2
  • 2014.05.09: [DONE] Beta 3
  • 2014.05.14: [DONE] Beta 4
  • 2014.05.16: [DONE] Beta 5
  • 2014.05.21: [DONE] Beta 6
  • 2014.05.23: [DONE] Beta 7
  • 2014.05.28: [DONE] Beta 8
  • 2014.05.30: [DONE] Beta 9
  • 2014.06.04: [DONE] Beta 10 (RC build 1)
  • 2014.06.06: [DONE] RC (build 2)
  • 2014.06.10: [DONE] Final

Features

Disabled

The following features have been deemed not ready for release and will be delayed to Firefox 31 or later.

On Track

Signed Off

The following features have no blockers and have been deemed ready for release.

Nightly

Fixed Bugs

How to Help
  • Download and install the latest Firefox Aurora build
  • Load this list of bug reports
  • Read each bug report until you find one you understand and want to test
  • Download and install the version of Firefox used to report the bug
  • Using the older build, test the claim made in the bug report to see if you can reproduce it
    • If you cannot reproduce the bug move on to another bug or ask the reporter to verify the fix
  • Using the latest Nightly, test to see if you can reproduce the bug
    • If you cannot reproduce the bug set the STATUS to VERIFIED FIXED and add a comment describing your testing
    • If you can still reproduce the bug set the STATUS to REOPENED and add a comment describing your testing
  • Add any bug reopened to the list below using the bug number template (verified fixes will be tracked elsewhere)
  • If you need help, send email to Tracy Walker with the bug summary in the Subject field
Reopened Bugs

Reported Bugs

Triage unconfirmed bugs.

How to Help
  • Download and install the latest Firefox Nightly
  • Load this list of bug reports
  • Read each bug report until you find one you understand and want to test
  • Test the claim made in the bug report and see if you can reproduce it
  • If you can reproduce the bug, set the STATUS to NEW, add a comment describing your testing, and try to find a regression window
  • Any confirmed bugs should be added to the list below using the bug number template
  • If you need help, send email to Tracy Walker with the bug summary in the Subject field
Confirmed Bugs

Stability

How to Help
  • Review the Firefox 30 Explosiveness Report
    • Any signatures rising significantly (>= 2.0 explosiveness rating) without a bug number should be reported to Bugzilla and nominated for tracking
    • If there is already a bug report, update the report with the latest numbers
    • Ensure the bug report has the topcrash-platform keyword added (where platform is the platform where crashes occur -- ie. topcrash-win for Windows)
    • Ensure the bug is listed below using the bug number template
  • Review the Firefox 30 Topcrash Report
    • Any signature in the top 10 without a bug number should be reported to Bugzilla and nominated for tracking
    • If there is already a bug report, update the report with the latest ranking information
    • Ensure the bug report has the topcrash-platform keyword added (where platform is the platform where crashes occur -- ie. topcrash-win for Windows)
    • Ensure the bug is listed below using the bug number template
  • Review the Firefox 30 tracked bugs that aren't tracked in 29 or 28.
    • Each bug listed needs to have its rank updated once per week
    • Use the topcrash reports from crash-stats to get the rank for each branch
    • Add this information to the bug report
    • Ensure the bug report has the topcrash-platform keyword added (where platform is the platform where crashes occur -- ie. topcrash-win for Windows)
    • Conversely, remove the topcrash keyword if the crash is no longer in the top-10
  • For each of these tasks, ensure the etherpad is updated with your name and the date checked
Topcrash Bugs

QA Wanted

How to Help
  • Download and install the latest Firefox Nightly
  • Review this list of bug reports to find a bug you both understand and want to test (each should contain a comment detailing what's needed from QA)
    • If the bug needs steps to reproduce, try to find a reproducible set of steps
    • If the bug needs a regression window, try to find a regression window
    • If the bug is fixed and needs verification, test to confirm you can reproduce the bug on an older build but not the latest build
  • Any tested bugs should be added to the list below using the bug number template
  • Remove the qawanted keyword from any fulfilled request
  • If you need help, send email to Tracy Walker with the bug summary in the Subject field
Tested Bugs

Testdays

Aurora

Branch Migration

Pre Sign-off Tasks

  • [DONE] Firefox branch migrated (Release Management)
  • [DONE] Mozmill Tests branch migrated (Automation Development)
  • [DONE] Updates throttled on aurora channel (Release Engineering)
  • [DONE] Updates verified throttled on aurora channel (QA)

Sign-off Tasks

  • [DONE] Mozmill Daily functional testruns pass with no failures ([report])
  • [DONE] Mozmill Daily auroratest update testruns pass with no failures ([report])
  • [DONE] Scope features tested and signed off for Aurora uplift

Post Sign-off Tasks

  • [DONE] Aurora sign-off email sent (QA)
  • [DONE] Updates unthrottled on aurora channel (Release Engineering)
  • [DONE] Updates verified unthrottled on aurora channel (QA)

Beta

Regression Testing

  • Run these tests for whatever platform(s) you have available.
  • Report a bug if you encounter one.
  • If you report a bug check to see if it's a regression by testing a previous Firefox version (if it doesn't reproduce in earlier versions it's a regression).
  • Add any reported bugs to the Bugs Found section below.
  • All results will appear in this dashboard

Beta 1

Build Info

Check List

  • [DONE] Regression testing (report) - FAIL
    • regression found: bug 1003839 - Live bookmark is still created despite cancelling the subscription
      • shipped to beta1 with this regression. it is tracked by Rel-Man to ensure it is fixed prior to final 30 release.
  • [DONE] Automation Functional tests (report) - PASS
  • [DONE] Updates on betatest channel (report) - PASS
  • [DONE] Updates on releasetest channel (report) - PASS
  • [DONE] Updates on beta channel (report)
    • bug 1004575 - ondemand_update test runs stalling out during recording results.

Regression Testing

  • Please run a brief smoketest for the features signed off on prior to merge to ensure they continue to be ready for our beta audience.
    • GStreamer 1.0 Support (Alexandra Lucinet)
    • Preference UI for proxy autologin (Bogdan Maris)
    • Places (Andrei Vaida)
    • Background-blend-mode (Catalin Varga)

Beta 2

Build Info

Check List

Regression Testing

Beta 3

  • shipped with tested failures note in regression testing reports. Further investigation is needed, but we'll ensure bugs are filed and tracked.

Build Info

Check List

Regression Testing

Beta 4

Build Info

Check List

Regression Testing

Beta 5

Build Info

Check List

  • [DONE] Regression testing (moztrap report) - PASS
  • [DONE] Automation Functional tests (report) - PASS
  • [DONE] Updates on betatest channel (report) - PASS
  • [DONE] Updates on releasetest channel (report) - PASS
  • [DONE] Updates on beta channel (report) - PASS

Regression Testing

Beta 6

Build Info

Check List

  • [DONE] Regression testing (Fix Verifications) - PASS
  • [DONE] Automation Functional tests (report) - PASS
  • [DONE] Updates on betatest channel (report) - PASS
  • [DONE] Updates on releasetest channel (report) - PASS
  • [DONE] Updates on beta channel (report) - PASS

Regression Testing

Beta 7

Build Info

Check List

  • [DONE] Regression testing (Fix Verifications) - PASS
  • [DONE] Automation Functional tests (report) - PASS
  • [DONE] Updates on betatest channel (report) - PASS
  • [DONE] Updates on releasetest channel (report) - PASS
  • [DONE] Updates on beta channel (report) - PASS

Regression Testing

  • Fix Verifications, focusing on those most recently fixed during this cycle.
    • bug 980714 - Blinking caret in panel text - VERIFIED
    • bug 1003839 - Live bookmark is still created despite cancelling the subscription. - VERIFIED

Beta 8

Build Info

Check List

Regression Testing

Beta 9

Build Info

Check List

Regression Testing

Release

Release Candidate Build 1

Build Info

Check List

Regression Testing

Release Candidate Build 2

Build Info

Check List

  • [DONE] Sanity Check - PASS
  • [DONE] Automation Functional tests (report) - PASS
  • [DONE] Updates on releasetest channel beta builds (report) - PASS
  • [DONE] Updates on beta channel (report) - PASS

Regression Testing

Final Release

  • [DONE] Updates on releasetest channel release builds (report) - PASS
    • [DONE] Spot checks for bug 1009893 - Show /whatsnew tour URL to users updating to 30.0 and 31.0 (from all past versions of Firefox) - PASS
  • [DONE] Updates on release channel release builds (report) - PASS
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_30/Test_Plan&oldid=987622"