Releases/Firefox 23/Test Plan

« Firefox 22 | Firefox 23 | Firefox 24 »

Firefox 23 Test Plan

Summary

Milestone	Firefox 23
Status	Currently in Aurora -- merging to Beta on 2013-06-24
Lead	Anthony Hughes (irc: ashughes)
2nd	Juan Becerra (irc: juanb)
Builds	[ftp] [http] [web]

The following is the test plan for Firefox 23 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 22. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

• [DONE] 2013.04.01: Nightly Merge
• [DONE] 2013.05.13: Aurora Merge
• [DONE] 2013.06.21: Pre-Beta feature sign-offs
• [ON TRACK] 2013.06.27: Beta 1
• 2013.07.05: Beta 2
• 2013.07.12: Beta 3
• 2013.07.19: Beta 4
• 2013.07.26: Beta 5
• 2013.07.31: Beta 6
• 2013.08.01: Release Candidate
• 2013.08.06: Final

Nightly

Merge

Functional Automation - result

Update Spotchecks - result

Features

See the Feature Release Tracking wiki for a general overview

Aurora

Migration Sign-off

Builds	nightlies
Changeset	tip (pushlog)

Checklist

• [DONE] Automated functional tests
• [DONE] Update tests on auroratest
• [DONE] Update tests on aurora

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform	Results [dashboard]
Windows XP	[DONE] en-US de fr it: 0 fails
Windows Vista	[DONE] en-US de fr it: 0 fails
Windows 7 32-bit	[DONE] en-US de fr it: 0 fails
Windows 7 64-bit	[DONE] en-US de fr it: 0 fails
Windows 8 32-bit	[DONE] en-US de fr it: 0 fails
Windows 8 64-bit	[DONE] en-US de fr it: 0 fails
Mac OS X 10.6	[DONE] en-US de fr it: 0 fails
Mac OS X 10.7	[DONE] en-US de fr it: 0 fails
Mac OS X 10.8	[DONE] en-US de fr it: 0 fails
Ubuntu 12.04 32-bit	[DONE] en-US de fr it: 0 fails
Ubuntu 12.04 64-bit	[DONE] en-US de fr it: 0 fails

Test Failures (known bugs)

Updates

Pre-migration Update Throttling

Platform	Tested Version	Target Version	Result
Windows XP	Firefox 21.0a2 en-US	no update	PASS
	Firefox 22.0a2 ar	no update	PASS
Mac OS X 10.8	Firefox 20.0a2 en-US	no update	PASS
	Firefox 22.0a2 ja-JP-mac	no update	PASS
Ubuntu 12.04 64-bit	Firefox 19.0a2 en-US	no update	PASS
	Firefox 22.0a2 de	no update	PASS

Auroratest

Platform	Tested Version	Target Version	Result
Windows XP	Firefox 22.0a2 2013-04-26 en-US	Firefox 23.0a2 2013-05-15 en-US	PASS
	Firefox 22.0a2 2013-05-08 de	Firefox 23.0a2 2013-05-15 de	PASS
Mac OS X 10.8	Firefox 14.0a2 2012-06-02 en-US	Firefox 23.0a2 2013-05-15 en-US	PASS
	Firefox 22.0a2 2013-05-10 fr	Firefox 23.0a2 2013-05-15 fr	PASS
Ubuntu 12.04 64-bit	Firefox 21.0a2 2013-03-07 en-US	Firefox 23.0a2 2013-05-15 en-US	PASS
	Firefox 22.0a2 2013-05-09 it	Firefox 23.0a2 2013-05-15 it	PASS

Issues

Aurora

Platform	Tested Version	Target Version	Result
Windows XP	Firefox 22.0a2 2013-04-26 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
	Firefox 22.0a2 2013-05-08 de	Firefox 23.0a2 2013-05-17 de	PASS
Mac OS X 10.8	Firefox 14.0a2 2012-06-02 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
	Firefox 22.0a2 2013-05-10 fr	Firefox 23.0a2 2013-05-17 fr	PASS
Ubuntu 12.04 64-bit	Firefox 21.0a2 2013-03-07 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
	Firefox 22.0a2 2013-05-09 it	Firefox 23.0a2 2013-05-17 it	PASS

Issues

Post-migration Update Unthrottling

Platform	Tested Version	Target Version	Result
Windows XP	Firefox 22.0a2 2013-04-26 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
	Firefox 22.0a2 2013-05-08 de	Firefox 23.0a2 2013-05-17 de	PASS
Mac OS X 10.8	Firefox 14.0a2 2012-06-02 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
	Firefox 22.0a2 2013-05-10 fr	Firefox 23.0a2 2013-05-17 fr	PASS
Ubuntu 12.04 64-bit	Firefox 21.0a2 2013-03-07 en-US	Firefox 23.0a2 2013-05-17 en-US	PASS
	Firefox 22.0a2 2013-05-09 it	Firefox 23.0a2 2013-05-17 it	PASS

Beta

Beta 1

Build Info

Builds	FTP, HTTP
Build ID	20130625125232
Changes	77a8241ae55d
Configs	mozmill

Checklist

Task	Status
Feature Sign-off	[DONE]
Automated Tests	[DONE]
Bug Fix Verifications	[DONE]
Update Tests (betatest)	[DONE]
Update Tests (releasetest)	[DONE]
Stability & Support sign-off	[DONE]
Update Tests (beta)	[ON TRACK]

Automated Tests

Platform	Status (results)
Windows XP	[DONE] 0 fails
Windows Vista	[DONE] 0 fails
Windows 7 32-bit	[DONE] 0 fails
Windows 7 64-bit	[DONE] 0 fails
Windows 8 32-bit	[DONE] 0 fails
Windows 8 64-bit	[DONE] 3 fails
Mac OSX 10.6	[DONE] 0 fails
Mac OSX 10.7	[DONE] 0 fails
Mac OSX 10.8	[DONE] 0 fails
Ubuntu Linux 32-bit	[DONE] 1 fails
Ubuntu Linux 64-bit	[DONE] 0 fails

Failures (known)

• bug 783484 - testAddons_ UninstallExtension
• bug 835132 - testSearchSelection

Feature Sign Off

Feature	QA Contact	Status
Mixed content blocking	Matt Wobensmith	[DONE] Ready for Beta
Add click-to-start disabled plugins	Paul Silaghi	[DONE] Ready for Beta
SocialAPI Share Panel	Anthony Hughes, Samvedana Gohil	[DONE] Ready for Beta
SocialAPI SocialMark	Anthony Hughes, Samvedana Gohil	[DONE] Ready for Beta
SocialAPI Web Install	Anthony Hughes, Samvedana Gohil	[DONE] Ready for Beta
SocialAPI Fullscreen Video Chat	Anthony Hughes, Samvedana Gohil	[DONE] Ready for Beta
Consolidate search preferences	Manuela Muntean	[DONE] Ready for Beta
New scrollbar style on OSX 10.7+	Marc Schifer	[DONE] Ready for Beta

Regression Tests

Moztrap Testrun #1519

Platform	Status (results)
Windows 7 32-bit	[DONE] 2 fails
Windows 8 64-bit	[DONE] 1 fails
Mac OSX 10.8	[DONE] 0 fails
Ubuntu Linux 32-bit	[DONE] 0 fails

Bug Fix Verifications

Bugs [more]	Status
bug 881991 [FHR] Duplicate aborted sessions being inserted	need-info
bug 881370 Crash when using JSD2 Debugger API for profiling	[BLOCKED]
bug 880042 Multi-file form field nsIFilePicker crash if overriding component has not implemented a domfiles getter	[DONE]
bug 878449 Crash in mozilla::widget::JumpListBuilder::AddListToBuild	[DONE]
bug 877612 Cannot enter into fullscreen mode after print dialog is opened	[DONE]
bug 877097 crash in mozilla::layout::ScrollbarActivity::UpdateOpacity	REOPENED
bug 876856 Crash in mozilla::dom::MediaStreamList::Length after closing a peer connection	[DONE]
bug 876721 crash in mozilla::gl::SharedSurface_ANGLEShareHandle::Create	[DONE]
bug 874090 Crash in mozilla::dom::Notification::GetPermissionInternal	[DONE]
bug 873963 Arbitrary code execution from Font Inspector	[DONE]
bug 872758 [FHR] Multi delete documents	[BLOCKED]
bug 871942 crash in nsWebShellWindow::RequestWindowClose	[DONE]
bug 869825 crash in mozilla::dom::ReparentWrapper	[DONE]
bug 868904 Too-much-recursion crash with CSS animated transform, SVG	[DONE]
bug 868337 crash in js::ion::DoCallFallback	[DONE]
bug 868257 watchPosition only sends one location	need-info
bug 867445 Remove test pilot from Firefox	[DONE]
bug 867226 Incomplete framebuffer abort in mozilla::layers::LayerManagerOGL::CreateFBOWithTexture	[DONE]
bug 866339 crash in mozilla::dom::DocumentBinding::createDocumentFragment	[DONE]
bug 865584 crash in js::ion::IonBuilder::createThisScriptedSingleton	[DROPPED]
bug 865260 crash in js::IsStandardClassResolved	[DONE]
bug 864505 [SocialAPI] active chat titlebar fails to change	[DONE]
bug 863646 Start up crash in nsContentUtils::GetCurrentJSContext in profile manager	[DONE]
bug 862610 Crash in mozilla::dom::HTMLImageElementBinding::DefineDOMInterface	[DONE]
bug 862084 Crash in mozilla::dom::HTMLSelectElementBinding::item	[DONE]
bug 860560 Search Box should respond to the changing of the selectedEngine/defaultenginename preferences	[DONE]
bug 860149 crash in mozilla::image::Decoder::Write	[DONE]
bug 859718 crash in mozilla::image::RasterImage::DecodePool::DecodeJob::Run	[DONE]
bug 859714 File open dialog buttons not working	[DONE]
bug 858946 Download panel doesn't display date correctly due to toLocaleFormat()	[DONE]
bug 858800 crash in mozilla::plugins::MiniShmBase::GetWritePtr	[DONE]
bug 855370 Remove the ability to not "Always show the tab bar"	[DONE]
bug 854319 crash in mozilla::MediaResource::`.* deleting destructor	[DONE]
bug 851934 crash in js::mjit::stubs::ValueToBoolean on Outlook.com	[DONE]
bug 851702 Remove "Enable JavaScript" checkbox from Prefs	[DONE]
bug 851701 Remove "Load images automatically" checkbox from Prefs	[DONE]
bug 847267 Enable DXVA2 on Windows Vista+ to accelerate H.264 video decoding	[DONE]
bug 839206 Replace plugin installation notification bar with door hanger	[DONE]
bug 821073 full screen support for video chats	[DONE]
bug 704063 Add unprefixed requestAnimationFrame	[DONE]
bug 331105 Allow users to set a default search engine	[DONE]
bug 52500 Make regular CSS properties apply on <input type='file'>	[DONE]
Metrics: 36/42 verified (86%)

Update Tests

Platform	Betatest [report]	Releasetest [report]	Beta [report]
Windows XP	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Windows Vista	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Windows 7 32-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Windows 7 64-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Windows 8 32-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Windows 8 64-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Mac OSX 10.6	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Mac OSX 10.7	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Mac OSX 10.8	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Ubuntu Linux 32-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails
Ubuntu Linux 64-bit	[DONE] 0 fails	[DONE] 0 fails	[DONE] 0 fails

New Issues

Bugs	Status
bug 887298 New created profile doesn't appear in the Profile Manager	[NEW]
bug 887276 Firefox 23 icon is changed	[DONE]
bug 887230 Add the ability to perform searches after a number from the URL bar	[DUPE]
bug 887228 Sound stops when video is played at 0.25x rate	[NEW]

Beta 2

Build Info

• Builds: FTP, HTTP
• Build ID: 20130701144430
• Changes: eb3499a7575d
• Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	PASS (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW
Update Tests (betatest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW Versions: 19.0b*, 20.0b*, 21.0b*, 22.0b* 23.0b1
Update Tests (releasetest)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW Versions: 19.0b*, 20.0b*, 21.0b*, 22.0b* 23.0b1
Update Tests (beta)	PASS (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW Versions: 19.0b*, 20.0b*, 21.0b*, 22.0b* 23.0b1
Regression Tests	PASS (details)	All tests in MozTrap Testrun #1638 pass without any serious regressions Tests: Breakpad, MD5 hash algorithm, and Plug-in compatibility exploratory smoketests Platforms: Windows 7 32-bit, Mac OSX 10.7 Locales: en-US Versions: 23.0b2#1

Issues Found

• [DROPPED] bug 889519 [Mozmill-CI] Lots of ondemand_update failures with various Java IO Exceptions -- RESOLVED WORKSFORME

Beta 3

Build Info

• Builds: FTP, HTTP
• Build ID: TBD
• Changes: TBD
• Configs: mozmill

Testing Info

Test	Result	Criteria
Automated Tests	result (details)	All Mozmill ondemand_functional testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 23.0b3#1 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (betatest)	result (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (releasetest)	result (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests (beta)	result (details)	All Mozmill ondemand_update testruns PASS with no new failures Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10 Builds: 19.0b1-23.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Regression Tests	PASS (details)	All Moztrap tests pass with no new regressions Platforms: Windows 8.1 32-bit, Windows 8.1 64-bit, Mac OSX 10.9 Builds: 23.0b3#1 en-US

Issues Found

• bug 890247 The Post Privacy Setting dropdown of the Share panel is not fully displayed
• bug 890258 Share login panel partly white
• bug 890260 Login errors in the Share panel are displayed partially over the sign in form
• bug 890264 An option for logging out of Facebook should be added to the Facebook Messenger menu
• bug 890283 browser.search.defaultenginename should not be case sensitive

VERIFYME Bugs

• [DONE] bug 885351 Google is always used in about:home and the URL bar when setting the default search engine to an invalid one
• [DONE] bug 865702 Weirdness with comboboxes pushed aside by Bookmarks sidebar
• [DONE] bug 872479 Toolbox tab icons are misplaced
• [DONE] bug 860213 Saving a canvas to disk makes firefox hang
• [DONE] bug 882050 Dynamically inserted SVG <image>s don't always animate
• [NEEDINFO] bug 881991 [FHR] Duplicate aborted sessions being inserted
• [BLOCKED] bug 881370 Crash when using JSD2 Debugger API for profiling
• [DONE] bug 880042 Multi-file form field nsIFilePicker crash if overriding component has not implemented a domfiles getter
• [DONE] bug 878449 Crash in mozilla::widget::JumpListBuilder::AddListToBuild
• [DONE] bug 877612 Cannot enter into fullscreen mode after print dialog is opened
• [REOPENED] bug 877097 crash in mozilla::layout::ScrollbarActivity::UpdateOpacity
• [DONE] bug 876856 Crash in mozilla::dom::MediaStreamList::Length after closing a peer connection
• [DONE] bug 876721 crash in mozilla::gl::SharedSurface_ANGLEShareHandle::Create
• [DONE] bug 874090 Crash in mozilla::dom::Notification::GetPermissionInternal
• [DONE] bug 873963 Arbitrary code execution from Font Inspector
• [BLOCKED] bug 872758 [FHR] Multi delete documents
• [DONE] bug 871942 crash in nsWebShellWindow::RequestWindowClose
• [DONE] bug 869825 crash in mozilla::dom::ReparentWrapper
• [DONE] bug 868904 Too-much-recursion crash with CSS animated transform, SVG
• [DONE] bug 868337 crash in js::ion::DoCallFallback
• [NEEDINFO] bug 868257 watchPosition only sends one location
• [DONE] bug 867758 Coordinate start of animation for same-FPS images
• [DONE] bug 867445 Remove test pilot from Firefox
• [DONE] bug 867226 Incomplete framebuffer abort in mozilla::layers::LayerManagerOGL::CreateFBOWithTexture
• [DONE] bug 866444 No Site favicon in History submenus – Recently Closed Tabs, Windows
• [DONE] bug 866339 crash in mozilla::dom::DocumentBinding::createDocumentFragment
• [DROPPED] bug 865584 crash in js::ion::IonBuilder::createThisScriptedSingleton
• [DONE] bug 865260 crash in js::IsStandardClassResolved
• [DONE] bug 864505 [SocialAPI] active chat titlebar fails to change
• [DONE] bug 863646 Start up crash in nsContentUtils::GetCurrentJSContext in profile manager
• [DONE] bug 862610 Crash in mozilla::dom::HTMLImageElementBinding::DefineDOMInterface
• [DONE] bug 862084 Crash in mozilla::dom::HTMLSelectElementBinding::item
• [DONE] bug 860560 Search Box should respond to the changing of the selectedEngine/defaultenginename preferences
• [DONE] bug 860149 crash in mozilla::image::Decoder::Write
• [DONE] bug 859718 crash in mozilla::image::RasterImage::DecodePool::DecodeJob::Run
• [DONE] bug 859714 File open dialog buttons not working
• [DONE] bug 858946 Download panel doesn't display date correctly due to toLocaleFormat()
• [DONE] bug 858800 crash in mozilla::plugins::MiniShmBase::GetWritePtr
• [DONE] bug 857817 [NBV] Consumes all client resources when load is slightly increased
• [DONE] bug 855370 Remove the ability to not "Always show the tab bar"
• [DONE] bug 854319 crash in mozilla::MediaResource::`.* deleting destructor
• [DONE] bug 851934 crash in js::mjit::stubs::ValueToBoolean on Outlook.com
• [DONE] bug 851702 Remove "Enable JavaScript" checkbox from Prefs
• [DONE] bug 851701 Remove "Load images automatically" checkbox from Prefs
• [DONE] bug 850571 crash in nsView::`.* deleting destructor'
• [DONE] bug 847267 Enable DXVA2 on Windows Vista+ to accelerate H.264 video decoding
• [DONE] bug 839206 Replace plugin installation notification bar with door hanger
• [DONE] bug 821073 full screen support for video chats
• [DONE] bug 799172 "Back" after "Options" just returns to the first dialog
• [DONE] bug 769348 URLs auto-completed in address bar might unexpectedly use ftp:// instead of http://
• [DONE] bug 704063 Add unprefixed requestAnimationFrame
• [DONE] bug 331105 Allow users to set a default search engine
• [DONE] bug 52500 Make regular CSS properties apply on <input type='file'>
• [DONE] bug 856344 Crash [@ js::Proxy::has] with adopted <form>
• [DONE] bug 878037 Crash on print if list-style-image (or :after|:before content) is animated gif
• [DONE] bug 860378 Crash with splitText, svg.text.css-frames.enabled
• [DONE] bug 860551 (SVGZoomAndPan instanceof SVGZoomAndPan) crashes
• [DONE] bug 863950 Crash with outdated CaretPosition object
• [DONE] bug 886067 Netmonitor displays request sizes as "0 KB" after opening Console
• more

QAWANTED Bugs

• [DONE] bug 887228 Sound stops when video is played at 0.25x rate
• [AT RISK] bug 883222 Ember.js calls on an event throws 'Permission denied to access property'
• [DONE] bug 878449 crash in mozilla::widget::JumpListBuilder::AddListToBuild
• [ON TRACK] bug 860641 startup crash in ffi_call with RelevantKnowledge 1.0.0.2
• more

UNCONFIRMED Bugs

• [DROPPED] bug 873887 security.mixed_content.block_display_content set to true breaks Zynga games
• [DROPPED] bug 866475 can't make a new tab, back button can't use
• [DROPPED] bug 865350 on about:newtab page, the bookmark star button does nothing
• [NEW] bug 864635 Star in About Nightly dialog overlaps some text
• [DONE] bug 864001 Flash player -interrupt
• [NEW] bug 863679 Jank on page. Profile highlights js_Date
• [DROPPED] bug 861781 Very slow startup with "automatic proxy configuration URL" (*.pac)
• [DROPPED] bug 859117 Fullscreen on Nascar raceview in nightly and aurora
• [NEW] bug 857938 Page flickers while scrolling using MATE desktop manager
• [BLOCKED] bug 801379 "Clear recent history" does not clear recently closed tabs & windows
• more

Known Issues

Full Query

ID	Summary	Status
693808	Entering numbers or single words and then pressing Enter in the location bar should bring search results	VERIFIED
842566	EXC_BAD_ACCESS (SIGSEGV) after building with DMD	NEW
885351	Google is always used in about:home and the URL bar when setting the default search engine to an invalid one	VERIFIED
887228	Sound stops when video is played at 0.25x rate	RESOLVED
887298	New created profile doesn't appear in the Profile Manager	RESOLVED

5 Total; 1 Open (20%); 2 Resolved (40%); 2 Verified (40%);