MozillaWiki

Releases/Firefox 31/Test Plan: Difference between revisions

Page Discussion

Releases/Firefox 31/Test Plan (view source)

Revision as of 19:54, 22 July 2014

8,173 bytes added ,  22 July 2014
m
→‎Noteable Issues
 
(46 intermediate revisions by 3 users not shown)
Line 241: Line 241:
* Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures
* Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures


;Untriaged Bugs (tracked)
; Untriaged Bugs
<bugzilla>
* [http://mzl.la/1wcpmbe Product fixes landed in Firefox 31] (2 bugs remaining as of 2014-07-11)
{
* [http://mzl.la/1nmurx0 Platform fixes landed in Firefox 31] (759 bugs remaining as of 2014-07-11)
"f1":"product",
* [http://mzl.la/1r4jxwY Product fixes uplifted in Firefox 31] (1 bugs remaining as of 2014-07-16)
"o1":"nowordssubstr",
* [http://mzl.la/1qfL2G2 Platform fixes uplifted in Firefox 31] (59 bugs remaining as of 2014-07-16)
"v1":"Android OS",
"f2":"keywords",
"o2":"nowordssubstr",
"v2":"verifyme intermittent assertion",
"f3":"status_whiteboard",
"o3":"nowordssubstr",
"v3":"qa asan",
"f4":"flagtypes.name",
"o4":"notequals",
"v4":"in-testsuite+",
"f5":"op_sys",
"o5":"nowordssubstr",
"v5":"Gonk Android",
"f6":"cf_qa_whiteboard",
"o6":"nowordssubstr",
"v6":"qa good first verify",
"f7":"flagtypes.name",
"o7":"notequals",
"v7":"in-testsuite?",
"f8":"product",
"o8":"anywordssubstr",
"v8":"Firefox Core Toolkit",
"f9":"component",
"o9":"nowordssubstr",
"v9":"build config gaia simulator",
"cf_status_firefox31":"fixed"
}
</bugzilla>


== Reported Bugs ==
== Reported Bugs ==
Line 305: Line 277:
}
}
</bugzilla>
</bugzilla>
= Regression Testing =
* [https://moztrap.mozilla.org/runtests/environment/4367/ Moztrap testrun] to cover past regressions, at-risk components, and feature integration tests
* WebRTC regression test
* PDF.js and printing
* Migration on first-run
Top-5 components by fixes
# JavaScript Engine - Jason Orendorff
# DOM - Johnny Stenback
# Graphics/WebGL - Jeff Muizelaar
# Layout - David Baron
# SVG - Jonathan Watt
* Next Action: Florin to find owners to work with Engineers indicated on creating a minimum-viable regression test(s) to stress the above areas - tests to be run throughout the beta cycle


= Milestones =
= Milestones =
Line 627: Line 585:


; Manual Testing
; Manual Testing
* {{ok|}} [https://moztrap.mozilla.org/results/cases/?filter-run=4367 Regression testing] -  
* {{ok|}} [https://moztrap.mozilla.org/results/cases/?filter-run=4367 Regression testing] - PASS
 
; Bugs
<bugzilla>
{
"id":"1032770",
"include_fields":"id,summary,status,resolution"
}
</bugzilla>
 
== Beta 7 ==
''[https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Walkthrough/Beta Read the Beta Coordination Guide for more info]''
 
; Build1 (20140703154127)
* [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/31.0b7-candidates/build1/ builds]
* [https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=a2a5087e5a6a&tochange=56ec82ed9283 changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx31b7 mozmill configs]
 
; Automated Testing
* [http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=31.0&platform=All&from=2014-07-04&to=2014-07-04 Functional Tests] ([http://mozmill-release.blargon7.com/#/functional/top?app=All&branch=31.0&platform=All&from=2014-07-04&to=2014-07-04 top fails])
* [http://mozmill-release.blargon7.com/#/update/detail?app=All&branch=All&channel=releasetest&from=2014-07-04&to=2014-07-04&target=31.0 Update Tests on releasetest] 
* [http://mozmill-release.blargon7.com/#/update/detail?app=All&branch=All&channel=beta&from=2014-07-04&to=2014-07-04&target=31.0 Update Tests on beta]
 
; Manual Testing
* [https://docs.google.com/spreadsheets/d/12yEBqNZqaR6_2XbqFrVwRVxt_URjNN3iMmu-GX60Z0Q/edit#gid=0 D2D blocklist regression testing]
* [https://etherpad.mozilla.org/Fx-31b7SocialAPI Social API regression testing]
* [https://moztrap.mozilla.org/results/cases/?filter-run=4367 Firefox 31 regression tests]
 
; Bugs
<bugzilla>
{
"id":"1034592",
"include_fields":"id,summary,status,resolution"
}
</bugzilla>
 
== Beta 8 ==
''[https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Walkthrough/Beta Read the Beta Coordination Guide for more info]''
 
; Build1 (20140707160635)
* [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/31.0b8-candidates/build1/ builds]
* [https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=56ec82ed9283&tochange=0fbe2d3fdf2b changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx31b8 mozmill configs]
 
; Automated Testing
* [http://mozmill-release.blargon7.com/#/functional/reports?app=All&branch=31.0&platform=All&from=2014-07-08&to=2014-07-08 Functional Tests] ([http://mozmill-release.blargon7.com/#/functional/top?app=All&branch=31.0&platform=All&from=2014-07-08&to=2014-07-08 top fails])
* [http://mozmill-release.blargon7.com/#/update/detail?app=All&branch=All&channel=releasetest&from=2014-07-08&to=2014-07-08&target=31.0 Update Tests on releasetest] 
* [http://mozmill-release.blargon7.com/#/update/detail?app=All&branch=All&channel=beta&from=2014-07-08&to=2014-07-08&target=31.0 Update Tests on beta]
 
; Manual Testing
* [https://moztrap.mozilla.org/runtests/environment/4590/ Social Panel Regression Tests] - ''run these on Windows XP, Windows 8.1, Mac OSX 10.9, and Ubuntu 13.10''
* WebGL Demos - [https://developer.mozilla.org/en-US/demos/tag/tech:webgl MDN], [http://www.webgl.com/category/webgl-games/ WebGL.com] - ''run these on Windows 7, Windows 8.1, Mac OSX 10.9, and Ubuntu 13.10 with at least one AMD, one NVidia, and one Intel GPU''
* [https://moztrap.mozilla.org/results/cases/?filter-run=4367 Firefox 31 regression tests]
 
; Bugs
<bugzilla>
{
"id":"1035776,1035803,1035835",
"include_fields":"id,summary,status,resolution"
}
</bugzilla>
 
== Beta 9 ==
''[https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Walkthrough/Beta Read the Beta Coordination Guide for more info]''
 
; Build1 (20140710141843)
* [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/31.0b9-candidates/build1/ builds]
* [https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=0fbe2d3fdf2b&tochange=f05aff296114 changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx31b9 mozmill configs]
 
; Automated Testing
* [http://mozmill-release.blargon7.com/#/functional/top?app=All&branch=31.0&platform=All&from=2014-07-11&to=2014-07-11 Functional Tests]
* [http://mozmill-release.blargon7.com/#/update/detail?app=All&branch=All&channel=releasetest&from=2014-07-11&to=2014-07-11&target=31.0 Update Tests on releasetest] 
* [http://mozmill-release.blargon7.com/#/update/detail?app=All&branch=All&channel=beta&from=2014-07-11&to=2014-07-11&target=31.0 Update Tests on beta]
 
; Manual Testing
* Exploratory testing for stability issues - [https://etherpad.mozilla.org/Fx31b9 PASS]


; Bugs
; Bugs
Line 636: Line 670:
}
}
</bugzilla>
</bugzilla>
== Release Candidate 1 ==
''[https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Walkthrough/Beta Read the Beta Coordination Guide for more info]''
; Build1 (20140714151536)
* [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/31.0-candidates/build1/ builds]
* [https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=0fbe2d3fdf2b&tochange=f05aff296114 changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx31RC1 mozmill configs]
; Automated Testing
* [http://mozmill-release.blargon7.com/#/functional/top?app=All&branch=31.0&platform=All&from=2014-07-14&to=2014-07-15 Functional Tests]
* [http://mozmill-release.blargon7.com/#/update/detail?app=All&branch=All&channel=releasetest&from=2014-07-14&to=2014-07-15&target=31.0 Update Tests on releasetest] 
* [http://mozmill-release.blargon7.com/#/update/detail?app=All&branch=All&channel=beta&from=2014-07-14&to=2014-07-15&target=31.0 Update Tests on beta]
Note: keep an eye on addon hotfix bugs to verify them asap: https://bugzilla.mozilla.org/buglist.cgi?f1=blocked&list_id=10737327&o1=equals&resolution=---&resolution=FIXED&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&v1=928173 
=== Sign-off for Beta ===
; Verifications
* {{bug|1038282}} Remove sw during beta-release migration for Firefox 31, keep hsb - PASS
=== Sign-off for Release ===
; Verifications
* {{bug|1028972}} crash in xul.dll@0x8e13cd | xul.dll@0x1ab44c | xul.dll@0x126047 | xul.dll@0x19effe | xul.dll@0x1c8403 | _MD_CURRENT_THREAD | PR_Unlock | xul.dll@0x11d2d4 | xul.dll@0x1c844d | xul.dll@0x34a70d
* {{bug|988549}} startup crash in _VEC_memzero | _VEC_memzero - N/A (latest change is NOT in Firefox 31 RC build 1)
Fixes landed on RC build (still need verification)
- https://bugzilla.mozilla.org/show_bug.cgi?id=812695 - [Read comment #414] [D2D] Text Rendering Issues on Windows 7 with Platform Update KB2670838 (MSIE 10 Prerequisite) or on Windows 8.1 ==> this is a rather old rendering issue affecting some Radeon cards. Going through the bug I could not find any Radeon model that we have available. There is only one machine with an AMD Radeon HD 3000, which may have been affected, but we haven't tried it yet.
- https://bugzilla.mozilla.org/show_bug.cgi?id=1034327 - Memory leak with TURN relay ==> according to Nils: "This was caught in the WebRTC QA lab and can be verified with that as well. Additionally manual verification has been done and can be done as needed."
== Release Candidate 2 ==
''[https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Walkthrough/Beta Read the Beta Coordination Guide for more info]''
; Build1 (20140716183446)
* [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/31.0-candidates/build2/ builds]
* [https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=f05aff296114&tochange=TBD changelog]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx31RC2 mozmill configs]
; Automated Testing
* [http://mozmill-release.blargon7.com/#/functional/top?app=All&branch=31.0&platform=All&from=2014-07-17&to=2014-07-17 Functional Tests]
* [http://mozmill-release.blargon7.com/#/update/detail?app=All&branch=All&channel=releasetest&from=2014-07-17&to=2014-07-17&target=31.0 Update Tests on releasetest] 
* [http://mozmill-release.blargon7.com/#/update/detail?app=All&branch=All&channel=beta&from=2014-07-17&to=2014-07-17&target=31.0 Update Tests on beta]
; Manual Testing
* Conduct testing of the New Tab page to ensure nothing regressed due to {{bug|1038997}} - FAIL ([https://etherpad.mozilla.org/Fx-31rc2 test results])
=== Noteable Issues ===
* bug 1015341, bug 1035534, bug 1035537 and {{bug|1035529}} all related to Movie Toolbar (bandoo) and combine to be our #7 topcrash
** Bandoo has been contacted and we'll be tracking this in Release (They are working on a fix)
* bug 984033 is an OOM that is now #4, correlated with Crossrider framework add-ons
** 0 crashes in Nightly since the patch landed in mozilla-central on July 17
** too late for Release but it should be uplifted to 32 and might be good to consider as a ridealong if we do a .point release
* bug 1012924 (experiments shutddown crash) is #2/#3 (depending on specific report) after OOM|small, competing with processMarkStackTop
** this should be gone when we release (just like Fx30) but raises concerns about running experiments through the entire Beta cycle (perhaps disable it in the last Beta until stability settles down)
* New tab page issues ([https://etherpad.mozilla.org/Fx-31rc2 etherpad]) -- do not block as they are also reproducible in Fx 30 (ie. not regressions from backing out New Tab Page Tiles)
Lizhenry
Confirmed users
2,816

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/993765...999265"