« Firefox 22 | Firefox 23 | Firefox 24 »

Firefox 23 Test Plan

Summary

Milestone	Firefox 23
Status	Currently in Aurora -- merging to Beta on 2013-06-24
Lead	Anthony Hughes (irc: ashughes)
2nd	Juan Becerra (irc: juanb)
Builds	[ftp] [http] [web]

The following is the test plan for Firefox 23 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 22. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

[DONE] 2013.04.01: Nightly Merge
[DONE] 2013.05.13: Aurora Merge
[DONE] 2013.06.21: Pre-Beta feature sign-offs
[DONE] 2013.06.27: Beta 1
[DONE] 2013.07.02: Beta 2
[DONE] 2013.07.05: Beta 3
[DONE] 2013.07.09: Beta 4
[DONE] 2013.07.12: Beta 5
[DONE] 2013.07.16: Beta 6
[DONE] 2013.07.19: Beta 7
[DONE] 2013.07.23: Beta 8
2013.07.26: Beta 9
2013.07.31: Beta 10
2013.08.05: Release Candidate
2013.08.06: Final

Nightly

Merge

Functional Automation - result

Update Spotchecks - result

Features

See the Feature Release Tracking wiki for a general overview

Aurora

Migration Sign-off


Builds	nightlies
Changeset	tip (pushlog)

Checklist

[DONE] Automated functional tests
[DONE] Update tests on auroratest
[DONE] Update tests on aurora

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform	Results [dashboard]
Windows XP	[DONE] en-US de fr it: 0 fails
Windows Vista	[DONE] en-US de fr it: 0 fails
Windows 7 32-bit	[DONE] en-US de fr it: 0 fails
Windows 7 64-bit	[DONE] en-US de fr it: 0 fails
Windows 8 32-bit	[DONE] en-US de fr it: 0 fails
Windows 8 64-bit	[DONE] en-US de fr it: 0 fails
Mac OS X 10.6	[DONE] en-US de fr it: 0 fails
Mac OS X 10.7	[DONE] en-US de fr it: 0 fails
Mac OS X 10.8	[DONE] en-US de fr it: 0 fails
Ubuntu 12.04 32-bit	[DONE] en-US de fr it: 0 fails
Ubuntu 12.04 64-bit	[DONE] en-US de fr it: 0 fails

Test Failures (known bugs)

Updates

Pre-migration Update Throttling

Platform	Tested Version	Target Version	Result
Windows XP	Firefox 21.0a2 en-US	no update	PASS
Windows XP	Firefox 22.0a2 ar	no update	PASS
Mac OS X 10.8	Firefox 20.0a2 en-US	no update	PASS
Mac OS X 10.8	Firefox 22.0a2 ja-JP-mac	no update	PASS
Ubuntu 12.04 64-bit	Firefox 19.0a2 en-US	no update	PASS
Ubuntu 12.04 64-bit	Firefox 22.0a2 de	no update	PASS

Auroratest

Platform	Tested Version	Target Version	Result
Windows XP	Firefox 22.0a2 2013-04-26 en-US	Firefox 23.0a2 2013-05-15 en-US	PASS
Windows XP	Firefox 22.0a2 2013-05-08 de	Firefox 23.0a2 2013-05-15 de	PASS
Mac OS X 10.8	Firefox 14.0a2 2012-06-02 en-US	Firefox 23.0a2 2013-05-15 en-US	PASS
Mac OS X 10.8	Firefox 22.0a2 2013-05-10 fr	Firefox 23.0a2 2013-05-15 fr	PASS
Ubuntu 12.04 64-bit	Firefox 21.0a2 2013-03-07 en-US	Firefox 23.0a2 2013-05-15 en-US	PASS
Ubuntu 12.04 64-bit	Firefox 22.0a2 2013-05-09 it	Firefox 23.0a2 2013-05-15 it	PASS

Issues

Aurora

Platform	Tested Version	Target Version	Result
Windows XP	Firefox 22.0a2 2013-04-26 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
Windows XP	Firefox 22.0a2 2013-05-08 de	Firefox 23.0a2 2013-05-17 de	PASS
Mac OS X 10.8	Firefox 14.0a2 2012-06-02 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
Mac OS X 10.8	Firefox 22.0a2 2013-05-10 fr	Firefox 23.0a2 2013-05-17 fr	PASS
Ubuntu 12.04 64-bit	Firefox 21.0a2 2013-03-07 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
Ubuntu 12.04 64-bit	Firefox 22.0a2 2013-05-09 it	Firefox 23.0a2 2013-05-17 it	PASS

Issues

Post-migration Update Unthrottling

Platform	Tested Version	Target Version	Result
Windows XP	Firefox 22.0a2 2013-04-26 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
Windows XP	Firefox 22.0a2 2013-05-08 de	Firefox 23.0a2 2013-05-17 de	PASS
Mac OS X 10.8	Firefox 14.0a2 2012-06-02 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
Mac OS X 10.8	Firefox 22.0a2 2013-05-10 fr	Firefox 23.0a2 2013-05-17 fr	PASS
Ubuntu 12.04 64-bit	Firefox 21.0a2 2013-03-07 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
Ubuntu 12.04 64-bit	Firefox 22.0a2 2013-05-09 it	Firefox 23.0a2 2013-05-17 it	PASS

Beta

Beta 1

Build Info


Builds	FTP, HTTP
Build ID	20130625125232
Changes	77a8241ae55d
Configs	mozmill

Checklist

Task	Status
Feature Sign-off	[DONE]
Automated Tests	[DONE]
Bug Fix Verifications	[DONE]
Update Tests (betatest)	[DONE]
Update Tests (releasetest)	[DONE]
Stability & Support sign-off	[DONE]
Update Tests (beta)	[ON TRACK]

Automated Tests

Platform	Status (results)
Windows XP	[DONE] 0 fails
Windows Vista	[DONE] 0 fails
Windows 7 32-bit	[DONE] 0 fails
Windows 7 64-bit	[DONE] 0 fails
Windows 8 32-bit	[DONE] 0 fails
Windows 8 64-bit	[DONE] 3 fails
Mac OSX 10.6	[DONE] 0 fails
Mac OSX 10.7	[DONE] 0 fails
Mac OSX 10.8	[DONE] 0 fails
Ubuntu Linux 32-bit	[DONE] 1 fails
Ubuntu Linux 64-bit	[DONE] 0 fails

Failures (known)

Feature Sign Off

Feature	QA Contact	Status
Mixed content blocking	Matt Wobensmith, Mihai Morar	[DONE] Ready for Beta
Add click-to-start disabled plugins	Paul Silaghi	[DONE] Ready for Beta
SocialAPI Share Panel	Anthony Hughes, Samvedana Gohil	[DONE] Ready for Beta
SocialAPI SocialMark	Anthony Hughes, Samvedana Gohil	[DONE] Ready for Beta
SocialAPI Web Install	Anthony Hughes, Samvedana Gohil	[DONE] Ready for Beta
SocialAPI Fullscreen Video Chat	Anthony Hughes, Samvedana Gohil	[DONE] Ready for Beta
Consolidate search preferences	Manuela Muntean	[DONE] Ready for Beta
New scrollbar style on OSX 10.7+	Marc Schifer	[DONE] Ready for Beta

Regression Tests

Moztrap Testrun #1519

Platform	Status (results)
Windows 7 32-bit	[DONE] 2 fails
Windows 8 64-bit	[DONE] 1 fails
Mac OSX 10.8	[DONE] 0 fails
Ubuntu Linux 32-bit	[DONE] 0 fails

Bug Fix Verifications

Bugs [more]	Status
bug 881991 [FHR] Duplicate aborted sessions being inserted	need-info
bug 881370 Crash when using JSD2 Debugger API for profiling	[BLOCKED]
bug 880042 Multi-file form field nsIFilePicker crash if overriding component has not implemented a domfiles getter	[DONE]
bug 878449 Crash in mozilla::widget::JumpListBuilder::AddListToBuild	[DONE]
bug 877612 Cannot enter into fullscreen mode after print dialog is opened	[DONE]
bug 877097 crash in mozilla::layout::ScrollbarActivity::UpdateOpacity	REOPENED
bug 876856 Crash in mozilla::dom::MediaStreamList::Length after closing a peer connection	[DONE]
bug 876721 crash in mozilla::gl::SharedSurface_ANGLEShareHandle::Create	[DONE]
bug 874090 Crash in mozilla::dom::Notification::GetPermissionInternal	[DONE]
bug 873963 Arbitrary code execution from Font Inspector	[DONE]
bug 872758 [FHR] Multi delete documents	[BLOCKED]
bug 871942 crash in nsWebShellWindow::RequestWindowClose	[DONE]
bug 869825 crash in mozilla::dom::ReparentWrapper	[DONE]
bug 868904 Too-much-recursion crash with CSS animated transform, SVG	[DONE]
bug 868337 crash in js::ion::DoCallFallback	[DONE]
bug 868257 watchPosition only sends one location	need-info
bug 867445 Remove test pilot from Firefox	[DONE]
bug 867226 Incomplete framebuffer abort in mozilla::layers::LayerManagerOGL::CreateFBOWithTexture	[DONE]
bug 866339 crash in mozilla::dom::DocumentBinding::createDocumentFragment	[DONE]
bug 865584 crash in js::ion::IonBuilder::createThisScriptedSingleton	[DROPPED]
bug 865260 crash in js::IsStandardClassResolved	[DONE]
bug 864505 [SocialAPI] active chat titlebar fails to change	[DONE]
bug 863646 Start up crash in nsContentUtils::GetCurrentJSContext in profile manager	[DONE]
bug 862610 Crash in mozilla::dom::HTMLImageElementBinding::DefineDOMInterface	[DONE]
bug 862084 Crash in mozilla::dom::HTMLSelectElementBinding::item	[DONE]
bug 860560 Search Box should respond to the changing of the selectedEngine/defaultenginename preferences	[DONE]
bug 860149 crash in mozilla::image::Decoder::Write	[DONE]
bug 859718 crash in mozilla::image::RasterImage::DecodePool::DecodeJob::Run	[DONE]
bug 859714 File open dialog buttons not working	[DONE]
bug 858946 Download panel doesn't display date correctly due to toLocaleFormat()	[DONE]
bug 858800 crash in mozilla::plugins::MiniShmBase::GetWritePtr	[DONE]
bug 855370 Remove the ability to not "Always show the tab bar"	[DONE]
bug 854319 crash in mozilla::MediaResource::`.* deleting destructor	[DONE]
bug 851934 crash in js::mjit::stubs::ValueToBoolean on Outlook.com	[DONE]
bug 851702 Remove "Enable JavaScript" checkbox from Prefs	[DONE]
bug 851701 Remove "Load images automatically" checkbox from Prefs	[DONE]
bug 847267 Enable DXVA2 on Windows Vista+ to accelerate H.264 video decoding	[DONE]
bug 839206 Replace plugin installation notification bar with door hanger	[DONE]
bug 821073 full screen support for video chats	[DONE]
bug 704063 Add unprefixed requestAnimationFrame	[DONE]
bug 331105 Allow users to set a default search engine	[DONE]
bug 52500 Make regular CSS properties apply on <input type='file'>	[DONE]
Metrics: 36/42 verified (86%)

Update Tests

Platform	Betatest [report]	Releasetest [report]	Beta [report]
Windows XP	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Windows Vista	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Windows 7 32-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Windows 7 64-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Windows 8 32-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Windows 8 64-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Mac OSX 10.6	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Mac OSX 10.7	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Mac OSX 10.8	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Ubuntu Linux 32-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Ubuntu Linux 64-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails

New Issues

Bugs	Status
bug 887298 New created profile doesn't appear in the Profile Manager	[NEW]
bug 887276 Firefox 23 icon is changed	[DONE]
bug 887230 Add the ability to perform searches after a number from the URL bar	[DUPE]
bug 887228 Sound stops when video is played at 0.25x rate	[NEW]

Beta 2

Build Info

Builds: FTP, HTTP
Build ID: 20130701144430
Changes: eb3499a7575d
Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	PASS (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW
Update Tests (betatest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW Versions: 19.0b, 20.0b, 21.0b, 22.0b 23.0b1
Update Tests (releasetest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW Versions: 19.0b, 20.0b, 21.0b, 22.0b 23.0b1
Update Tests (beta)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW Versions: 19.0b, 20.0b, 21.0b, 22.0b 23.0b1
Regression Tests	PASS (details)	All tests in MozTrap Testrun #1638 pass without any serious regressions Tests: Breakpad, MD5 hash algorithm, and Plug-in compatibility exploratory smoketests Platforms: Windows 7 32-bit, Mac OSX 10.7 Locales: en-US Versions: 23.0b2#1

Issues Found

bug 889519 [mozmill] Lots of ondemand_update failures with various Java IO Exceptions

Beta 3

Build Info

Builds: FTP, HTTP
Build ID: 20130703181823
Changes: 4be469df6d23
Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	PASS (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 23.0b3#1 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (betatest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (releasetest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (beta)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Regression Tests	PASS (details)	All Moztrap tests pass with no new regressions Platforms: Windows 8.1 32-bit, Windows 8.1 64-bit, Mac OSX 10.9 Builds: 23.0b3#1 en-US

Issues Found

Full Query

ID	Summary	Status	Resolution
890247	The Post Privacy Setting dropdown of the Share panel is not fully displayed	RESOLVED	WONTFIX
890258	Share login panel partly white	RESOLVED	WONTFIX
890260	Login errors in the Share panel are displayed partially over the sign in form	RESOLVED	WONTFIX
890264	An option for logging out of Facebook should be added to the Facebook Messenger menu	RESOLVED	WONTFIX
890283	browser.search.defaultenginename should not be case sensitive	RESOLVED	INVALID

5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);

Beta 4

Build Info

Builds: FTP, HTTP
Build ID: 20130708202947
Changes: d318fea569d6
Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	PASS (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 23.0b4#1 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (betatest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b3 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (releasetest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b3 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (beta)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b3 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Regression Tests	PASS (details)	All Moztrap tests pass with no new regressions Platforms: Windows Vista, Windows 7 32-bit, Mac OSX 10.6, Ubuntu Linux 32-bit Builds: 23.0b4#1 en-US

Issues Found

Bugs	Status
No bugs found...

Beta 5

Build Info

Builds: FTP, HTTP
Build ID: 20130711122148
Changes: ce7fb11233cd
Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	PASS (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 23.0b5#1 da de en-US es-ES fr it ja ko pt-BR ro ru
Update Tests (betatest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b4 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (releasetest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b4 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (beta)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b4 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

Issues Found

Beta 6

Build Info

Builds: FTP, HTTP
Build ID: 20130715155216
Changes: 0a7642297703
Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	PASS (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 23.0b6#1 Locales: af da de en-US es-ES es-MX fi fr hr hu it ja ko ku ml or pl pt-BR pt-PT ro ru sr zh-CN zh-TW zu
Update Tests (betatest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b5 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (releasetest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b5 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (beta)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b5 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Regression Tests	PASS (details)	All Moztrap tests pass with no new regressions Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit Builds: 19.0b1-23.0b5 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW

Note: On Ubuntu 12.10 32bit, the Copy & Paste from plug-in content to another application, in the case of quitting Firefox before pasting, doesn't work (same behavior on Chromium and Opera)

Issues Found

bug 894397 Norton toolbar is not displayed in the Add-ons Manager
bug 894412 Text selected from Gmail is removed from clipboard after Firefox restart
bug 894509 [mozmill] l10n failures in testTabbedBrowser/testNewTab.js
bug 894510 [mozmill] Failure - domain name mismatch in testSearchSelection.js
bug 894513 [mozmill] Location bar should contain pasted clipboard content - got , expected ipsum in testPasteLocationBar.js

Beta 7

Build Info

Builds: FTP, HTTP
Build ID: 20130718163513
Changes: 1897bd316ab6
Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	PASS (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 23.0b7#1 Locales: af da de en-US es-ES es-MX fi fr hr hu it ja ko ku ml or pl pt-BR pt-PT ro ru sr zh-CN zh-TW zu
Update Tests (betatest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b6 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (releasetest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b6 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (beta)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b6 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Regression Tests	PASS (details)	All Moztrap tests pass with no new regressions Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit Builds: 23.0b7#1 Locales: en-US

Issues Found

bug 856541 [mozmill] testEnableCookies.js fails with "Disconnect Error: Application unexpectedly closed"
bug 894509 [mozmill] testNewTab.js fails with "Correct tab title - 'Pestaña nueva' should equal 'Nueva pestaña'"
bug 894513 [mozmill] testPasteLocationbar.js fails with "Location bar should contain pasted clipboard content - got , expected ipsum"
bug 895996 [mozmill] testAddMozSearchProvider.js fails with "Disconnect Error: Application unexpectedly closed"
bug 835132 [mozmill] testSearchSelection.js fails with "Current domain name matches target domain name"
bug 818128 [mozmill] testVisibleItemsMax.js fails with "Number of visible rows should equal 6"

Beta 8

Build Info

Builds: FTP, HTTP
Build ID: 20130722172257
Changes: 8b103274f2d1
Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	PASS (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 23.0b8#1 Locales: af da de en-US es-ES es-MX fi fr hr hu it ja ko ku ml or pl pt-BR pt-PT ro ru sr zh-CN zh-TW zu
Update Tests (betatest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b7 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (releasetest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b7 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (beta)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b7 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Regression Tests	PASS (details)	Regression testing around beta plugins/drivers Platforms: Windows 8.1, Mac OSX 10.9 Builds: 23.0b8#1 Locales: en-US

Issues Found

[ON TRACK] bug 894513 [mozmill] Failure in testPasteLocationBar.js
[ON TRACK] bug 895996 [mozmill] Failure in testAddMozSearchProvider.js
[ON TRACK] bug 835132 [mozmill] Failure in testSearchSelection.js
[ON TRACK] bug 818128 [mozmill] Failure in testVisibleItemsMax.js
[ON TRACK] bug 866599 [mozmill] Failure in testPasswordSavedAndDeleted.js
[DROPPED] bug 897182 [mozmill] Failure in testDefaultBookmarks.js
[DROPPED] bug 897183 [mozmill] Failure in testPreferences_masterPassword
[DROPPED] bug 897177 [mozmill] Multiple failures with Firefox 23 en-US on Ubuntu

Beta 9

Build Info

Builds: FTP, HTTP
Build ID: 20130725195523
Changes: c1541a6ecf41
Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	PASS (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 23.0b9#1 Locales: af da de en-US es-ES es-MX fi fr hr hu it ja ko ku ml or pl pt-BR pt-PT ro ru sr zh-CN zh-TW zu
Update Tests (betatest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b8 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (releasetest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b8 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (beta)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b8 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Regression Tests	PASS (details)	Regression testing around mixed content blocking (ex. apps.facebook.com, Koding.com Apps, Firstgroup.com Purchases) Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.7, Ubuntu 13.04 32-bit Builds: 23.0b9#1 Locales: en-US

Issues Found

FIXED ~~[BLOCKED] bug 898587 No updates available to Firefox 23.0b9~~
FIXED ~~[BLOCKED] bug 898574 Mozmill nodes in *.qa.scl3.mozilla.com cannot reach external machines via the proxy~~
bug 898391 Bubble Adventures game keeps on reloading
bug 898402 Inconsistent behavior on some of the Facebook apps
bug 898399 Doorhanger is displayed only at first loading of https://koding.com/ using new profile
bug 894513 [mozmill] testPasteLocationbar.js fails with "Location bar should contain pasted clipboard content - got , expected ipsum"
bug 895996 [mozmill] testAddMozSearchProvider.js fails with "Disconnect Error: Application unexpectedly closed"
bug 835132 [mozmill] testSearchSelection.js fails with "Current domain name matches target domain name"
bug 818128 [mozmill] testVisibleItemsMax.js fails with "Number of visible rows should equal 6"
bug 826693 [mozmill] testGreenLarry.js fails with "controller.waitForPageLoad(): Timeout waiting for page loaded."
bug 898568 [mozmill] testSearchSuggestions.js fails with "Suggestions Mozilla from Google and SAPO search providers are different - got 'false' "

Beta 10

Build Info

Builds: FTP, HTTP
Build ID: 20130729175331
Changes: TBD
Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	PASS (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 23.0b10#1 Locales: af da de en-US es-ES es-MX fi fr hr hu it ja ko ku ml or pl pt-BR pt-PT ro ru sr zh-CN zh-TW zu
Update Tests (betatest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b9 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (releasetest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b9 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (beta)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b9 Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Regression Tests	PASS (details)	All Moztrap regression tests PASS with no regressions found

Issues Found

bug 894513 [mozmill] testPasteLocationbar.js fails with "Location bar should contain pasted clipboard content - got , expected ipsum"
bug 895996 [mozmill] testAddMozSearchProvider.js fails with "Disconnect Error: Application unexpectedly closed"
bug 835132 [mozmill] testSearchSelection.js fails with "Current domain name matches target domain name"
bug 818128 [mozmill] testVisibleItemsMax.js fails with "Number of visible rows should equal 6"
bug 899565 Linkedin search box slightly misplaced
bug 899579 Unable to select only a part of the link from html popup on Youtube HTML5 Videos

Release Candidate

Build Info

	Build 1	Build 2
ID	20130730113002	20130801200732
Builds	FTP, HTTP	FTP, HTTP
Pushlog	2c423c4c6d0b	5efffaf39814
Configs	ondemand.cfg	ondemand.cfg

Feature Sign Off

Feature	Owner	Status
Mixed content blocking	Matt Wobensmith	DONE - Ready for Release
Share, SocialMark, WebInstall, Tear-off Chat	Samvedana Gohil	DONE - Ready for Release
Consolidated Search Preferences	Matt Wobensmith	DONE - Ready for Release
Max OSX Scrollbars	Marc Schifer	Delayed to Firefox 24

Regression Testing

Criteria	Description	Result	Reports
Mozmill Tests	Automated functional smoketests utilizing Mozmill	PASS	build1, build2
Moztrap Tests	Manual tests covering areas at high risk to regression	PASS	build1
AMD Radeon Stability Tests	Manual dogfooding to shake out intermittent instability	PASS	n/a
Partner Repack Update Tests	Manual spotchecking of updates for partner repack builds	PASS	n/a

Update Testing

Criteria	Description	Result	Reports
Betatest	Automated software update tests on the betatest channel	PASS	build1, build2
Releasetest	Automated software update tests on the releasetest channel	PASS	build1
Release	Automated software update tests on the release channel	PASS	build1

Issues Found

Bug	Status
bug 900520 Yahoo Repack Build - the installation can't be customized	RESOLVED INVALID
bug 900516 Yahoo Repack Build - Installer keeps prompting to close Firefox before installing	RESOLVED INVALID
bug 900463 Share panel on Facebook doesn't initially load content	NEW
bug 900454 The Share button doesn't change colour to indicate the page has been shared	RESOLVED WONTFIX
bug 900414 Dragged and dropped text from a HTML page to webmail is pasted in Chinese	NEW
bug 900373 Callout missing on YouTube's Statistics	NEW

Bug Triage

Query	Description	Remaining
QAWANTED	Bugs requiring immediate investigative assistance	0
VERIFYME	Bugs with fixes requiring immediate verification	15
FIXED	Bugs with fixes which should be verified as soon as possible	146
REGRESSION	Bugs requiring identification of the first affected build	222
UNCONFIRMED	Bugs requiring confirmation if they are reproducible in a current build	73

Known Issues

Full Query

ID	Summary	Status	Resolution
449392	Running the Java Applet Video Player together with scrolling the Site causes paint error	RESOLVED	INVALID
582097	Java error, ClassNotFoundException: Tcpbw100.class fails with 4.0b3, works w/ 4.0b1	RESOLVED	WORKSFORME
693808	Entering numbers or single words and then pressing Enter in the location bar should bring search results	VERIFIED	FIXED
806116	touchpad TAP only triggers ":active" with double-tap on element with tabindex	NEW
811085	Can't use Adobe Reader for PDF files	RESOLVED	WONTFIX
842566	EXC_BAD_ACCESS (SIGSEGV) after building with DMD	NEW
881027	100% cpu when on new maps.google.com (may be unable to kill, requiring reboot)	RESOLVED	WORKSFORME
885351	Google is always used in about:home and the URL bar when setting the default search engine to an invalid one	VERIFIED	FIXED
887228	Sound stops when video is played at 0.25x rate	RESOLVED	WORKSFORME
887298	New created profile doesn't appear in the Profile Manager	RESOLVED	WORKSFORME
888885	Weird performance differences in new google maps	RESOLVED	INACTIVE
888914	Bottom of page hidden with Dev Tools open	RESOLVED	WONTFIX
889320	Firefox does not show PDF when Content Security Policy is enabled	VERIFIED	FIXED
889790	Need to be able to add Video to an existing audio-only webrtc call or remove Video	RESOLVED	FIXED
890355	Attachment P7M not showing	NEW
890843	Missing control capabilities on animated GIFs for a user (GIF Player capabilities: pause, stopping, per frame forward etc)	RESOLVED	WONTFIX
891424	Modifying XUL panel while open causes the browser to hide	VERIFIED	FIXED
892375	DevTools Console Net doesn't allow opening URLs on first click	RESOLVED	WORKSFORME
892431	Cant leave a webpage	UNCONFIRMED
892506	Autocomplete/Passwordmanager automatically fills in first field on website with a wrong value	RESOLVED	INVALID
894397	Norton toolbar is not displayed in the Add-ons Manager	RESOLVED	INVALID
894412	Text selected from Gmail is removed from clipboard after Firefox restart	RESOLVED	DUPLICATE
898399	Doorhanger is displayed only at first loading of https://koding.com/ using new profile	RESOLVED	INVALID
900373	Callout missing on YouTube's Statistics	RESOLVED	INVALID
900414	Dragged and dropped text from a HTML page to webmail is pasted in Chinese	RESOLVED	DUPLICATE
900463	Share panel on Facebook doesn't initially load content	RESOLVED	WONTFIX

26 Total; 4 Open (15.38%); 18 Resolved (69.23%); 4 Verified (15.38%);

Potential Explosive Concerns

Bug	Mitigation
bug 860641 startup crash in ffi_call with RelevantKnowledge 1.0.0.2	Should be RelNoted to educate workaround
bug 767343 crash in nsSupportsStringImpl::SetData	Year old crash trending up slightly, need to investigate further

Firefox 23.0.1

Build Info

	Build 1	Build 2
ID	20130814063812	20130815181913
Builds	FTP, HTTP	FTP, HTTP
Changelog	a55c55edf302	0576ba82616e
Configs	ondemand.cfg	n/a

Automated Testing

Criteria	Description	Result	Testruns	Runtime (h:mm)	Reports
Functional	Automated functional tests via Mozmill-CI and ondemand	PASS	192	1:06	mozmill-release (top-fails)
Betatest	Automated software update tests on the betatest channel	PASS	305	0:34	mozmill-release
Releasetest	Automated software update tests on the releasetest channel	PASS	309	0:42	mozmill-release
Release	Automated software update tests on the release channel	PASS	306	0:35	mozmill-release

Top Failures

Test	Status	Bug	Count
testSecurity/testSafeBrowsingNotificationBar.js	KNOWN TEST FAILURE (ASSIGNED)	bug 905033	89
testSearch/testSearchSuggestions.js	KNOWN TEST FAILURE (REOPENED)	bug 807251	10
testAwesomeBar/testVisibleItemsMax.js	KNOWN TEST FAILURE (ASSIGNED)	bug 818128	10

Manual Testing

WebRTC

Using the Moztrap test as a guideline, verify that Firefox 23.0.1 is no worse than Firefox 22.0 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:

Have only one browser open on each machine at any given time
Have only one call running at any given time on a particular machine
Make sure the caller and callee are always on *different* machines.
Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
If you find regressions, report a bug and CC Randell and Maire; they can help track down if the regressions are real or not
- Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.

Caller	Callee	Result	Notes
Firefox 23.0.1 on Windows 7	Firefox 23.0.1 on Windows 7	[AT RISK]	Caller: 4s delay immediately Callee: 4s delay after 5 min
Firefox 22.0 on Windows 7	Firefox 23.0.1 on Windows XP	[AT RISK]	Caller: 4s delay immediately Callee: 4s delay after 5 min
Chrome Stable on Windows 7	Firefox 23.0.1 on Windows 8	[AT RISK]	Caller: 2s delay immediately Callee: PASS
Firefox 23.0.1 on Windows 7	Firefox 22.0 on Mac OSX 10.8	[AT RISK]	Caller: PASS Callee: 1.5s delay immediately
Firefox 23.0.1 on Windows 7	Chrome Stable on Ubuntu 12.04	[AT RISK]	Caller: PASS Callee: 1.5s delay immediately, 3s delay after 5 min
Firefox 22.0 on Windows XP	Firefox 23.0.1 on Windows 7	[AT RISK]	Caller: 5s delay immediately Caller: 5s delay immediately
Firefox 23.0.1 on Windows XP	Firefox 22.0 on Windows XP	[AT RISK]	Caller: 4s delay immediately Callee: PASS
Firefox 23.0.1 on Windows XP	Chrome Stable on Windows 8	[AT RISK]	Caller: 8s delay immediately Callee: PASS
Chrome Stable on Windows XP	Firefox 23.0.1 on Mac OSX 10.8	[DONE]	Caller: PASS Callee: PASS
Firefox 22.0 on Windows XP	Firefox 23.0.1 on Ubuntu 13.04	[AT RISK]	Caller: 2s delay immediately Callee: PASS
Chrome Stable on Windows 8	Firefox 23.0.1 on Windows 7	[DONE]	Caller: PASS Callee: PASS
Firefox 23.0.1 on Windows 8	Chrome Stable on Windows XP	[AT RISK]	Caller: 6s delay immediately Callee: PASS
Firefox 22.0 on Windows 8	Firefox 23.0.1 on Windows 8	[AT RISK]	Caller: 6s delay immediately Callee: PASS
Firefox 23.0.1 on Windows 8	Firefox 22.0 on Windows 8	[AT RISK]	Caller: 4s delay immediately Callee: PASS
Firefox 22.0 on Windows 8	Firefox 23.0.1 on Mac OSX 10.8	[AT RISK]	Caller: 2s delay immediately Callee: PASS
Chrome Stable on Windows 8	Firefox 23.0.1 on Ubuntu 13.04	[DONE]	Caller: PASS Callee: PASS
Firefox 23.0.1 on Mac OSX 10.8	Firefox 22.0 on Windows 7	[AT RISK]	Caller: 4s delay after 5 min with jerky sound Callee: audio okay but video is poor
Chrome Stable on Mac OSX 10.8	Firefox 23.0.1 on Windows XP	[AT RISK]	Caller: 1s delay after 5 min growing to 2s after 10 min Callee: PASS
Firefox 22.0 on Mac OSX 10.8	Firefox 23.0.1 on Windows 8	[AT RISK]	Caller: 1s delay after 5 min Callee: PASS
Firefox 23.0.1 on Mac OSX 10.8	Chrome Stable on Mac OSX 10.8	[AT RISK]	Caller: 1s delay after 5 min Callee: PASS
Firefox 23.0.1 on Mac OSX 10.8	Firefox 22.0 on Ubuntu 13.04	[AT RISK]	Caller: 2s delay after 5 min with jerky sound Callee: audio okay but video quality is poor
Firefox 23.0.1 on Ubuntu 12.04	Chrome Stable on Windows 7	[DONE]	Caller: PASS Callee: PASS
Firefox 22.0 on Ubuntu 12.04	Firefox 23.0.1 on Windows XP	[AT RISK]	Caller: 1s delay after 5 min Callee: PASS
Chrome Stable on Ubuntu 12.04	Firefox 23.0.1 on Windows 8	[DONE]	Caller: PASS Callee: PASS
Firefox 23.0.1 on Ubuntu 12.04	Firefox 22.0 on Mac OSX 10.8	[AT RISK]	Caller: 2s delay after 5 min Callee: PASS
Firefox 23.0.1 on Ubuntu 12.04	Chrome Stable on Ubuntu 13.04	[DONE]	Caller: PASS Callee: PASS

Basic Installation and Startup

Using the Moztrap test as a guideline, test that en-US and l10n builds install and function correctly.

Platform	Locales	Result	Notes
Windows XP	en-US & de	[DONE]	Stub Installer offered 23.0, not 23.0.1 (expected)
Windows Vista	en-US & fr	[DONE]	Stub Installer offered 23.0, not 23.0.1 (expected)
Windows 7	en-US & it	[DONE]	Stub Installer offered 23.0, not 23.0.1 (expected)
Windows 8	en-US & pt-BR	[DONE]	Stub Installer offered 23.0, not 23.0.1 (expected)
Windows 8.1	en-US & zh-TW	[DONE]	zh-TW font not rendering in installer, reported bug 906015

Bug Fix Verifications

Bug	Result	Notes
bug 901944 Rendering glitches on H.264 video only in FF23 on Vista	[DONE]	Verified by Samvedana Gohil
bug 902173 "[fr] Outils" wrongly translated to "Outils de développement" in menu since FF 23	[DONE]	Verified by Ioana Budnar
bug 902532 Spellchecking broken with non-ASCII characters in profile path	[DONE]	Verified by Ioana Budnar
bug 901527 Audio static/"burble"/breakup in mozilla to mozilla webrtc calls	[DONE]	Verified via WebRTC testing above
bug 902349 crash in nsStyleSet::FileRules with AMD Radeon HD 6310	[DROPPED]	Not verifiable by QA
bug 904001 Block rlnx.dll, pmnx.dll, opnx.dll, prnx.dll 1.3.334.9 (Relevant Knowledge 1.0.2)	[DROPPED]	Not verifiable by QA

New Issues

Bug	Status
bug 906057 Delay on WebRTC calls	NOT BLOCKING - will be investigated for resolution in Firefox 24

Post-Mortem

Successes

The following is a list of things which did go well with Firefox 23 from Nightly through to Release.

Developer Tanvi Vyas went above and beyond, working with QA (and many other parties) to make sure her feature (Mixed Content Blocking) was well-understood and well-tested.
- affected sites tracked
- a lot of community documentation
- for the future, highlight this as a good model for working with developers
- go to someone and ask how you can help
- be proactive in reaching out to developers
Coordination of release (emails, tasks, etc) went really well, it's clear what needs to get done when

Failures

The following is a list of things which did not go well with Firefox 23 from Nightly through to Release.

bug 901527 Audio static/"burble"/breakup in mozilla to mozilla webrtc calls
- Was not reported until nearly the last Beta
- WebRTC testing was not a high priority in our Beta testing
  - come up with a better way to track high priority areas that aren't necessarily in the scope of feature testing and the existing Beta checklist
  - work closer with Asa and Developers to get greater visibility, not just in features but high-risk landings
  - add features from current release to regression suite for next release or two

bug 901944 Rendering glitches on H.264 video only in FF23 on Vista
- High-risk areas need coverage on a broader range of platforms/hardware
- What happened here proves a quite serious communication problem between QA and dev. The QA working on the H264... support feature was told it's only for Windows 7 and 8 and never got further updates. bug 847267 (which caused bug 901944) wasn't added as a dependency to the tracking bug either bug 799318. The feature information tracking should get improved. Otherwise we might get left with the only option of spamming devs with emails requesting for updates every week (a situation unpleasant for everyone).
- I think we might be getting to a point where visibility of engineering work to QA (or lack thereof) is negatively impacting the quality of the product
  - status emails are sent prior to uplift but that might miss certain uplifts -- should increase frequency of these emails
  - use need-info request on tracking bugs to get the information we need, and block on it if necessary

bug 904001 Block rlnx.dll, pmnx.dll, opnx.dll, prnx.dll 1.3.334.9 (Relevant Knowledge 1.0.2)
- We were unable to verify this due to old versions of the installer not being available
  - Part of a bigger problem, don't have an extensible mechanism to block libraries before we load Firefox
  - Matt to connect Security Assurance team, Brian Bondy with the Stability team

bug 902532 Spellchecking broken with non-ASCII characters in profile path
- Discovered following release
- Can/should this be covered by automation?
  - Not in Mozmill, possibly in Marionette -- carry this discussion forward
- Can/should we be doing more edge-case testing than common-case testing?
  - look at our coverage and where our blind spots are
  - any automated test involving strings should cover strings with non-ASCII characters

bug 902173 "[fr] Outils" wrongly translated to "Outils de développement" in menu since FF 23
- Locale-specific issue reported to us in Release, was not in any other branch
- Can/should we be engaging with localizers to be involved in RC sign-off?
  - run an l10n testday for RC
  - reach out to Axel to see if we can get localizer feedback during the RC cycle

bug 902349 crash in nsStyleSet::FileRules with AMD Radeon HD 6310
- We were not able to reproduce this issue despite having hardware previously known to reproduce these types of issues
  - We are doomed unless we can get engineers engaged in fixing this
  - QA is doing everything we can, Marc to talk to Bob about dev involvement
  - Tracy: Is it the same machine in RelEng farm building the "broken" builds?
    - No, it's an issue with how optimization works at build time.
- Question whether doing twin RC builds is worth the QA/RelEng resources as compared to just respinning on occasion
  - QA needs to run full automation on both builds, regardless if we don't ship build2
  - RelEng needs to generate both builds, clean up build2 if we decide to ship build1
  - Stability rarely, if ever, has enough data to say conclusively if a build is good or not
  - Having multiple builds has not prevented us from having to ship a follow-up build in this instance
  - After 4 releases (24 weeks) dealing with this, can we do some statistical, risk, and cost-benefit analysis to justify doing this further?
- For further consideration, what's our strategy if we have no more leads and have to just live with this issue?

Features lists for this release were not centralized anywhere, so at least one feature did not get to QA: Network Monitor.
The beta scope was not adapted to the new 2 betas per week process, which created QA overhead.
Invalid/unclear tests in MozTrap.
- Unclear tests should be updated. If their owner can't do that, the QA team is willing to take on that. Some of these tests can lead to invalid bugs (e.g. bug 900457).
- Some of these tests are present in the ESR smoke tests too (e.g. save and open test, play videos test etc - we took the liberty of updating where it was only neccesary to change the links in order to be able to run the tests).
  - Otilia to get SV to roll up unclear/failed tests into the Release/Beta testing emails

Mitigating Strategy

The following is a list of actions we can/should take to build on our successes and prevent similar failures.

[ashughes] come up with a better way to track high priority areas that aren't necessarily in the scope of feature testing and the existing Beta checklist
[mschifer] work closer with Asa and Developers to get greater visibility, not just in features but high-risk landings
[ashughes] add features from current release to regression suite for next release or two
[otilia] increase frequency of feature status emails during Aurora/Beta
[QA] use need-info flag on tracking bugs for outstanding/blocking status request
[mwobensmith] connect those working on providing a more extensible system for blocking libraries at startup
[ashughes] Investigate with Henrik whether bug 902532 can be covered by Marionette, also non-ASCII string testing, failure of Mozmill if DTD is wrong
[QA] Audit current coverage areas and identify blindspots, particularly edge-case coverage
[ashughes] run an l10n testday for RC periodically, talk to Axel about support and getting localizers more engaged in RC testing
[ashughes] continue discussing viability of "twin build" strategy
[softvision] roll up failed/unclear Moztrap tests into the "Firefox N Testing" emails so we can fix broken tests
[ashughes] raise discussion about running a weekly QA post-mortem (modeled from RelEng's meetings)

All-hands Post-mortem

Discussion

is generating build2 for RC worth it?
need a single source of truth for features
should we make Beta 10 RC1, offer it to Beta users, to get more stability soak time?
relationship building between QA, Dev, and other teams

Actions

joduinn to investigate desktop repack - channel changing
bsmedberg to run his script on build#1 crash data
- if we can find it, then we'll continue making build #2 automatically
- if we can't find it, then we'll stop making build #2 and make more dot releases :(
- result: found no crashes with build1 so we need to discuss dropping build2 from our standard process; this will mean potentially doing more .1 releases and some weekend work; a risk and trade-off we are willing to make I think based on prevalence
lsblakk to create meeting about RC instead of beta 10
- QA supports this; stop doing Beta 10 and offer Beta/RC users the same RC1 build; grants us a couple more days of soak time for stability
lsblakk to speak to Laura about finding out about putting in a tab for users about a specific change - timing, finding out sooner - what causes/guidelines about when to do this
lsblakk - check with Axel if having a hook on l10n repos for key/visible string would help protect localization teams from errors like [fr] tools in the future; especially for string changes
akeybl to start a discussion about a single source of truth for feature tracking; potentially reuse wiki portal for bug queries; ex. https://wiki.mozilla.org/User:Akeybl/Release_Dash
ashughes/tracy to start discussion in a channel meeting about getting more value out of Aurora; current value is low because it's our lowest audience; Tracy suggests rebranding might help; Alex suggested scrapping Aurora; Jonath advocates for three branches (1 dev, 2 testing); Brendan does not see a need nor urgency due to B2G

Meeting Details

QA Post-mortem

When: Monday August 19, 0800h PT
Where: QA_Desktop Vidyo Room
Notes: https://mozqa.etherpad.mozilla.org/fx23-desktop-post-mortem

All-hands Post-mortem

When: Tuesday August 20, 0900h PT
Where: ReleaseCoordination Vidyo Room
Notes: https://etherpad.mozilla.org/23-0-Post-Mortem

Releases/Firefox 23/Test Plan

Contents

Firefox 23 Test Plan

Summary

Schedule

Nightly

Merge

Features

Aurora

Migration Sign-off

Automated Testing

Updates

Pre-migration Update Throttling

Auroratest

Aurora

Post-migration Update Unthrottling

Beta

Beta 1

Build Info

Checklist

Automated Tests

Feature Sign Off

Regression Tests

Bug Fix Verifications

Update Tests

New Issues

Beta 2

Build Info

Testing Info

Issues Found

Beta 3

Build Info

Testing Info

Issues Found

Beta 4

Build Info

Testing Info

Issues Found

Beta 5

Build Info

Testing Info

Issues Found

Beta 6

Build Info

Testing Info

Issues Found

Beta 7

Build Info

Testing Info

Issues Found

Beta 8

Build Info

Testing Info

Issues Found

Beta 9

Build Info

Testing Info

Issues Found

Beta 10

Build Info

Testing Info

Issues Found

Release Candidate

Build Info

Feature Sign Off

Regression Testing

Update Testing

Issues Found

Bug Triage

Known Issues

Potential Explosive Concerns

Firefox 23.0.1

Build Info

Automated Testing

Manual Testing

WebRTC

Basic Installation and Startup

Bug Fix Verifications

New Issues

Post-Mortem