« Firefox 8 | Firefox 9 | Firefox 10 »

Firefox 9 Test Plan

Summary

Milestone	Firefox 9
Status	Currently in Beta -- Beta 2 on November 18th, 2011
Lead	Anthony Hughes (irc: ashughes)
2nd	Juan Becerra (irc: juanb)

The following is the test plan for Firefox 9 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 9. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

Task	Owner	Status
Automation Triage	Softvision	ON TRACK - 2011-11-14
Bug Triage	Softvision	ON TRACK - 2011-11-14
Feature Sign-offs	Softvision	ON TRACK - 2011-11-14
Full Functional Tests (Aurora)	iBeta	MISSED? - 2011-11-14
Website Compatibility (Aurora)	uTest	MISSED? - 2011-11-14
Milestones
Beta 2	ashughes	Builds expected 2011-11-16
Beta 1	ashughes	DONE - 2011-11-11
Aurora Merge	ashughes	DONE - 2011-09-27
Nightly Merge	ashughes	DONE - 2011-08-16

Automation

BFTs

Task	Triage the Mozmill dashboard for test failures and regressions
Lead	Vlad Maniac (Softvision)
Peer	Alex Lakatos (Softvision)
Status	NEW (12), ASSIGNED (9), FIXED (3), REGRESSIONS (0)
Instructions
Check the Mozmill dashboard for failing tests for the day Cross reference any failure to the list of NEW and ASSIGNED bugs If the bug is NEW, assign it to yourself and try to fix it If the bug is ASSIGNED and there has been no activity on the bug in over a week, offer your assistance to fix the bug If there is NO bug on file, file it using this template If you need assistance, ask ashughes in the #mozmill channel
Bugs Found

Endurance

Task	Triage the Mozmill Endurance dashboard for test failures and performance regressions
Lead	Mihaela Velimiroviciu (Softvision)
Peer
Status	Spreadsheet (Nightly)
Instructions
Check the dashboard for the performance numbers from the most recent nightly Add the data to the spreadsheet Report any performance spikes or test failures to Dave Hunt and Anthony Hughes
Bugs Found

Bug Triage

Lead	Vlad Maniac
Peer	Alex Lakatos
Queries	QA+, FIXED, UNCONFIRMED
Bugs Found

Features

Track features which have landed in Firefox 9 and are ready to be tested (owners)

The following is a statement of feature readiness for release.

Feature	Owner	Status
JS Type Inference	N/A	No QA Needed - 2011-11-14 (ashughes)
Do Not Track Signal	Ioana Budnar	SIGNED OFF - 2011-12-19
Multi-process IndexDB	N/A	Unscoped as per e10s status - 2011-11-11
PFS2 Datasource	Virgil Dicu	No release target specified. Nothing implemented yet. - 2011-12-14
Links
Scoped Features, Unscoped Features

Full Functional Tests

The following tracks testing done by iBeta for full functional test coverage. This task is performed once during Aurora only.

Lead: Anthony Hughes
[ link to test suite]
[ link to results of testing]
[ link to bugs filed by iBeta]

Website Compatibility

The following tracks testing done by uTest for website compatibility. This task is performed after every merge.

Lead: Anthony Hughes
Link to test suite
Link to report for Nightly Web Compatibility
Link to report for Aurora Web Compatibility
Link to report for Beta Web Compatibility
Link to bugs filed

Nightly Merge

Verify that the builds generated from the merge to Nightly have updated properly.

Nightly

[DONE] [ashughes] Windows XP: PASS
- Source: Mozilla/5.0 (Windows NT 5.1; rv:8.0a1) Gecko/20110813 Firefox/8.0a1
- Target: Mozilla/5.0 (Windows NT 5.1; rv:9.0a1) Gecko/20110817 Firefox/9.0a1
[DONE] [ashughes] Windows 7: PASS
- Source: Mozilla/5.0 (Windows NT 6.1; rv:8.0a1) Gecko/20110814 Firefox/8.0a1
- Target: Mozilla/5.0 (Windows NT 6.1; rv:9.0a1) Gecko/20110817 Firefox/9.0a1
[DONE] [ashughes] Mac OS X 10.6: PASS
- Source: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0a1) Gecko/20110812 Firefox/8.0a1
- Target: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0a1) Gecko/20110817 Firefox/9.0a1
[DONE] [ashughes] Ubuntu 11.04: PASS
- Source: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:8.0a1) Gecko/20110815 Firefox/8.0a1
- Target: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:9.0a1) Gecko/20110817 Firefox/9.0a1

Issues Found

Aurora Merge

The following are the detailed results of the automated BFT test runs.

[DONE] [ashughes] Windows 2000: PASS
[DONE] [ashughes] Windows XP: PASS
[DONE] [ashughes] Windows Vista: PASS
[DONE] [ashughes] Windows 7: PASS
[DONE] [ashughes] Windows 7 64-bit: PASS
[DONE] [ashughes] Mac OS X 10.6: PASS
[DONE] [ashughes] Mac OS X 10.7: PASS
[DONE] [ashughes] Ubuntu 11.04: PASS
[DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
Reports: Mozmill (8.0a2)

Issues Found

Two tests are constantly failing and which need to be fixed as soon as possible.

Updates

The following are details of update verifications for Firefox 9 merge to Aurora

Auroratest

[DONE] [ashughes] Windows XP: PASS
- Source: Mozilla/5.0 (Windows NT 5.1; rv:8.0a2) Gecko/20110920 Firefox/8.0a2
- Target: Mozilla/5.0 (Windows NT 5.1; rv:9.0a2) Gecko/20110929 Firefox/9.0a2
[DONE] [ashughes] Windows 7: PASS
- Source: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0a2) Gecko/20110926 Firefox/8.0a2
- Target: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0a2) Gecko/20110929 Firefox/9.0a2
[DONE] [ashughes] Mac OS X 10.6: PASS
- Source: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0a2) Gecko/20110926 Firefox/8.0a2
- Target: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0a2) Gecko/20110929 Firefox/9.0a2
[DONE] [ashughes] Ubuntu 11.04: PASS
- Source: Mozilla/5.0 (X11; Linux i686; rv:8.0a2) Gecko/20110923 Firefox/8.0a2
- Target: Mozilla/5.0 (X11; Linux i686; rv:9.0a2) Gecko/20110929 Firefox/9.0a2

Aurora

[DONE] [ashughes] Windows XP: PASS
- Source: Mozilla/5.0 (Windows NT 5.1; rv:8.0a2) Gecko/20110920 Firefox/8.0a2
- Target: Mozilla/5.0 (Windows NT 5.1; rv:9.0a2) Gecko/20110930 Firefox/9.0a2
[DONE] [ashughes] Windows 7: PASS
- Source: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0a2) Gecko/20110926 Firefox/8.0a2
- Target: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0a2) Gecko/20110930 Firefox/9.0a2
[DONE] [ashughes] Mac OS X 10.6: PASS
- Source: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0a2) Gecko/20110926 Firefox/8.0a2
- Target: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0a2) Gecko/20110930 Firefox/9.0a2
[DONE] [ashughes] Ubuntu 11.04: PASS
- Source: Mozilla/5.0 (X11; Linux i686; rv:8.0a2) Gecko/20110923 Firefox/8.0a2
- Target: Mozilla/5.0 (X11; Linux i686; rv:9.0a2) Gecko/20110930 Firefox/9.0a2

Issues Found

Beta 1

The following tasks need to be completed to sign-off the Beta merge

Task	Assignee	Status
Automated Tests	ashughes	SIGNED-OFF - 2011-11-10
Manual Tests	Softvision	SIGNED-OFF - 2011-11-10
Update Tests	ashughes	SIGNED-OFF - 2011-11-10 (releasetest)
Bug Fix Verifications	ashughes, Softvision	SIGNED-OFF - 2011-11-10
Feature Sign-offs	ashughes	SIGNED-OFF - 2011-11-10
Issues Found	ashughes, Softvision	SIGNED-OFF - 2011-11-10
Known Issues	ashughes, Softvision	SIGNED-OFF - 2011-11-10
Links
Builds, Litmus Testrun, Mozmill Dashboard

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Assignee	Result
Windows 2000	ashughes	PASS - 2011-11-10
Windows XP	ashughes	PASS - 2011-11-10
Windows Vista	ashughes	PASS - 2011-11-10
Windows 7 32-bit	ashughes	PASS - 2011-11-10
Windows 7 64-bit	ashughes	PASS - 2011-11-10
Mac OS X 10.6	ashughes	PASS - 2011-11-10
Mac OS X 10.7	ashughes	PASS - 2011-11-10
Ubuntu 11.10 32-bit	ashughes	PASS - 2011-11-10
Ubuntu 11.10 64-bit	ashughes	PASS - 2011-11-10
Reports
Mozmill, Litmus

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

Platform	Assignee	Result
Windows 2000	Simona	PASS- 2011-11-10
Windows XP	Ioana	PASS- 2011-11-10
Windows Vista	Simona	PASS- 2011-11-10
Windows 7 32-bit	Mihaela	PASS- 2011-11-10
Windows 7 64-bit	Paul	PASS- 2011-11-10
Mac OS X 10.6	VladG	PASS- 2011-11-10
Mac OS X 10.7	VladG	PASS - 2011-11-10
Ubuntu 11.10 32-bit	Ioana	PASS- 2011-11-10
Ubuntu 11.10 64-bit	Virgil	PASS- 2011-11-10
Reports
Litmus

Focused Tests

The following are the detailed results of the focused feature testing

Feature	Assignee	Result	Platforms Tested
Web App Compatibility	Softvision	PASS- 2011-11-10	MacOS X 10.6, MacOS X 10.7, Win 7 x64, Win 7 x32, Ubuntu 11.10 x86, Windows XP, Windows Vista, Ubuntu 11.10 x64, Win 2000
Audio/Video	Softvision	PASS- 2011-11-10	Win 7 64, MacOS X 10.6, Windows XP, Ubuntu 11.10 x64
Plugins	Softvision	PASS- 2011-11-10	Win 7 64, MacOS X 10.6, Windows XP, Ubuntu 11.10 x64
Add-ons	Softvision	PASS- 2011-11-10	Win 7 64, MacOS X 10.6, Windows XP, Ubuntu 11.10 x64
Session Store	Softvision	PASS- 2011-11-10	Win 7 64, MacOS X 10.6, Windows XP, Ubuntu 11.10 x64
Rich Profile Updates	Softvision	PASS- 2011-11-10	Win 7 64, MacOS X 10.6, Windows XP, Ubuntu 11.10 x64
Reports
Litmus, Exploratory work spreadsheet, Links to any spreadsheets, etc

Update Tests

The following are details of update verifications

Platform	Assignee	Result: Releasetest	Result: Beta
Windows 2000	ashughes	PASS - 2011-11-10	PASS - 2011-11-10
Windows XP	ashughes	PASS - 2011-11-10	PASS - 2011-11-10
Windows Vista	ashughes	PASS - 2011-11-10	PASS - 2011-11-10
Windows 7 32-bit	ashughes	PASS - 2011-11-10	PASS - 2011-11-10
Windos 7 64-bit	ashughes	PASS - 2011-11-10	PASS - 2011-11-10
Mac OS X10.6	ashughes	PASS - 2011-11-10	PASS - 2011-11-10
Mac OS 10.7	ashughes	PASS - 2011-11-10	PASS - 2011-11-10
Ubuntu1.10 32-bit	ashughes	PASS - 2011-11-10	PASS - 2011-11-10
Ubuntu1.10 64-bit	ashughes	PASS - 2011-11-10	PASS - 2011-11-10
Reports
Releasetest, Beta

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

Query	Assignees	Status
QA+		VERIFIED 6 / 6 - 2011-11-17
FIXED		VERIFIED 1 / 165 - 2011-11-09
Reports
Fixed bugs verification spreadsheet

Issues Found

Bug ID	Summary	Reporter	Status
701404	Silverlight plugin keeps on crashing on MAC OS X 10.7	VladG	NEW

Known Issues

Bug ID	Summary	Status	Blocker?
bug 700835	[Mac] Firefox 8 with Java 6 build 1.6.0_29-b11-402 crashes after tab close	ASSIGNED	No
bug 604215	The zooming transition to the browser isn't smooth with a large number of tabitems	VERIFIED FIXED
bug 697696	Highlight issue in amazon's password box	NEW
bug 697725	Amazon tab favicon flickers when page is loading	NEW
bug 681922	In google docs Upload -> Files has no highlighting	NEW
bug 694690	Youtube HTML5 Video's Fullscreen Button does not work.	RESOLVED WORKSFORME
bug 692446	Firefox prompts to install windows media player plugin even though it is already installed	NEW
bug 679701	[Mac] Embedded window media player plugin is not displayed properly (Flip4Mac)	NEW
bug 382124	Crash Reporter (= Breakpad) doesn't write minidumps on Windows 2000 (old DbgHelp.dll / MiniDumpWriteDump())	RESOLVED WONTFIX
bug 585816	no thumbnail when dragging tabs	NEW
bug 671535	Videos fail to play on Windows 2000	NEW
bug 694288	Vimeo.com - embedded video player is not displayed on Windows 2000	NEW
bug 638741	right click on flash content displays two context menus sometimes	NEW
bug 696096	panic.com - Items are not shown correctly in shopping cart	REOPENED

Beta 2

The following tasks need to be completed to sign-off Beta 2

Task	Assignee	Status
Automated Tests	ashughes	SIGNED-OFF - 2011-11-16
Manual Tests	Softvision	SIGNED-OFF - 2011-11-16
Update Tests	ashughes	SIGNED-OFF - 2011-11-17
Bug Fix Verifications	ashughes, Softvision	SIGNED-OFF - 2011-11-17
Issues Found	ashughes, Softvision	SIGNED-OFF - 2011-11-17
Known Issues	ashughes, Softvision	SIGNED-OFF - 2011-11-17
Links
Builds, Litmus Testrun, Mozmill Dashboard

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Assignee	Result
Windows 2000	ashughes	PASS - 2011-11-16
Windows XP	ashughes	PASS - 2011-11-16
Windows Vista	ashughes	PASS - 2011-11-16
Windows 7 32-bit	ashughes	PASS - 2011-11-16
Windows 7 64-bit	ashughes	PASS - 2011-11-16
Mac OS X 10.6	ashughes	PASS - 2011-11-16
Mac OS X 10.7	ashughes	PASS - 2011-11-16
Ubuntu 11.10 32-bit	ashughes	PASS - 2011-11-16
Ubuntu 11.10 64-bit	ashughes	PASS - 2011-11-16
Reports
Mozmill, Litmus

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

Platform	Assignee	Result
Windows 2000	Simona	PASS - 2011-11-17
Windows XP	Ioana	PASS - 2011-11-17
Windows Vista	Simona	PASS - 2011-11-17
Windows 7 32-bit	Mihaela	PASS - 2011-11-17
Windows 7 64-bit	Paul	PASS - 2011-11-17
Mac OS X 10.6	Vlad	PASS - 2011-11-17
Mac OS X 10.7	Vlad	PASS - 2011-11-17
Ubuntu 11.10 32-bit	Ioana	PASS - 2011-11-17
Ubuntu 11.10 64-bit	Virgil	PASS - 2011-11-17
Reports
[ Litmus]

Focused Tests

The following are the detailed results of the focused feature testing. While testing, keep an eye out for inconsistency, incompatibility, crashes, and hangs. Any issues should be reported in the Issues Found section below.

Feature	Assignee	Status	OS
Plugins Verify stability and compatibility of Flash, Java, Silverlight, QuickTime, etc Report plug-in versions and sites tested	Softvision	PASS - 2011-11-17	Ubuntu 11.10 x86_64, Win7 x86, Win7 64, Mac OS 10.6, Windows XP
Performance Do some stress testing of Firefox 8, monitoring CPU and memory usage, then compare the same testing in Firefox 9 (keep an eye out for crashes/hangs) Report testing performed and comparative CPU/memory usage	Softvision	PASS - 2011-11-17	Ubuntu 11.10 x86_64, Win7 x86, Win7 64, Mac OS 10.6, Windows XP
Rich Profile Updates Populate a profile with bookmarks, history, session data, form data, saved passwords, app-tabs, panorama groups, themes, add-ons, etc on a previous version of Firefox and update to Firefox 9.0b2 on betatest channel Report any lost data, crashes, and inconsistencies	Softvision	PASS - 2011-11-17	Ubuntu 11.10 x86_64, Win7 x86, Win7 64, Mac OS 10.6, Windows XP
Anti-virus Integration Check that updating from a previous version of Firefox with some popular Anti-virus software installed, including add-ons (Norton, McAfee, AVG, ZoneAlarm, etc) Report AV Software and version used during testing and any incompatibilities	Softvision	PASS - 2011-11-17	Windows XP, Win7 x86, Win7 64 (N/A for Ubuntu 11.10 x86_64, Mac OS 10.6)
Reports
Litmus, Exploratory work spreadsheet

Update Tests

The following are details of update verifications

Platform	Assignee	Result: Releasetest	Result: Beta
Windows 2000	ashughes	SIGNED-OFF - 2011-11-17
Windows XP	ashughes	SIGNED-OFF - 2011-11-17
Windows Vista	ashughes	SIGNED-OFF - 2011-11-17
Windows 7 32-bit	ashughes	SIGNED-OFF - 2011-11-17
Windos 7 64-bit	ashughes	SIGNED-OFF - 2011-11-17
Mac OS X10.6	ashughes	SIGNED-OFF - 2011-11-17
Mac OS 10.7	ashughes	SIGNED-OFF - 2011-11-17
Ubuntu1.10 32-bit	ashughes	SIGNED-OFF - 2011-11-17
Ubuntu1.10 64-bit	ashughes	SIGNED-OFF - 2011-11-17
Reports
~~Releasetest~~ (brasstacks outage), [ Beta]

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

Query	VERIFIED	Status
QA+	VERIFIED 1 / 3	PASS - 2011-11-17
FIXED	VERIFIED 6 / 184	RESULT - 2011-11-17
Reports
Fixed bugs verification spreadsheet

Issues Found

Bug ID	Summary	Reporter	Status
bug 703216	Html5 youtube video not rendered smoothly when using expand/shrink button	Virgil	UNCO

Known Issues

Bug ID	Summary	Status	Blocker?
bug 701404	Silverlight plugin keeps on crashing on MAC OS X 10.7	NEW	No
bug 700835	[Mac] Java 6 build 1.6.0_29-b11-402 crashes after tab close	ASSIGNED	No
bug 697696	Highlight issue in amazon's password box	NEW	No
bug 697725	Amazon tab favicon flickers when page is loading	NEW	No
bug 681922	In google docs Upload -> Files has no highlighting	NEW	No
bug 692446	Firefox prompts to install windows media player plugin even though it is already installed	NEW	No
bug 679701	[Mac] Embedded window media player plugin is not displayed properly (Flip4Mac)	NEW	No
bug 585816	no thumbnail when dragging tabs	NEW	No
bug 671535	Videos fail to play on Windows 2000	NEW	No
bug 694288	Vimeo.com - embedded video player is not displayed on Windows 2000	NEW	No
bug 638741	right click on flash content displays two context menus sometimes	NEW	No
bug 694690	Youtube HTML5 Video's Fullscreen Button does not work.	REOPENED	No
bug 699175	The current theme is not active when updating from Firefox 7.0b6 to Firefox 8.0b6	NEW	No

Beta 3

The following tasks need to be completed to sign-off Beta 3

Task	Assignee	Status
Automated Tests	ashughes	SIGNED-OFF - 2011-11-23
Manual Tests	Softvision	SIGNED-OFF - 2011-11-24
Update Tests	ashughes	IN PROGRESS - 2011-11-23
Bug Fix Verifications	ashughes, Softvision	IN PROGRESS - 2011-11-24
Issues Found	ashughes, Softvision	SIGNED-OFF - 2011-11-24
Known Issues	ashughes, Softvision	SIGNED-OFF - 2011-11-24
Links
Builds, Litmus Testrun, Mozmill Dashboard

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Assignee	BFTs
Windows 2000	ashughes	PASS - 2011-11-23
Windows XP	ashughes	PASS - 2011-11-23
Windows Vista	ashughes	PASS - 2011-11-23
Windows 7 32-bit	ashughes	PASS - 2011-11-23
Windows 7 64-bit	ashughes	PASS - 2011-11-23
Mac OS X 10.6	ashughes	PASS - 2011-11-23
Mac OS X 10.7	ashughes	PASS - 2011-11-23
Ubuntu 11.10 32-bit	ashughes	PASS - 2011-11-23
Ubuntu 11.10 64-bit	ashughes	PASS - 2011-11-23
Reports
Mozmill, Litmus

l10n Testruns (Optional)

Platform	Assignee	BFTs: report	Updates: report
Windows XP	ashughes	PASS - 2011-11-25
Mac OS X 10.7	ashughes	HUNG @ pl - 2011-11-25
Ubuntu 9.10 32-bit	ashughes	HUNG @ af - 2011-11-25

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

Platform	Assignee	Result
Windows 2000	Simona	PASS - 2011-11-24
Windows XP	Mihaela	PASS - 2011-11-24
Windows Vista	Simona	PASS - 2011-11-24
Windows 7 32-bit	Mihaela	PASS - 2011-11-24
Windows 7 64-bit	Paul	PASS - 2011-11-24
Mac OS X 10.6	VladG	PASS - 2011-11-24
Mac OS X 10.7	VladG	PASS - 2011-11-24
Ubuntu 11.10 32-bit	Virgil	PASS - 2011-11-24
Ubuntu 11.10 64-bit	Virgil	PASS - 2011-11-24
Reports
Litmus

Focused Tests

The following are the detailed results of the focused feature testing. While testing, keep an eye out for inconsistency, incompatibility, crashes, and hangs. Any issues should be reported in the Issues Found section below.

Feature	Assignee	Status
Plugins Verify stability and compatibility of Flash, Java, Silverlight, QuickTime, etc Report plug-in versions and sites tested	Softvision	PASS-2011-11-24
Performance Do some stress testing of Firefox 8, monitoring CPU and memory usage, then compare the same testing in Firefox 9 (keep an eye out for crashes/hangs) Report testing performed and comparative CPU/memory usage	Softvision	PASS-2011-11-24
Rich Profile Updates Populate a profile with bookmarks, history, session data, form data, saved passwords, app-tabs, panorama groups, themes, add-ons, etc on a previous version of Firefox and update to Firefox 9.0b2 on betatest channel Report any lost data, crashes, and inconsistencies	Softvision	PASS-2011-11-24
Anti-virus Integration Check that updating from a previous version of Firefox with some popular Anti-virus software installed, including add-ons (Norton, McAfee, AVG, ZoneAlarm, etc) Report AV Software and version used during testing and any incompatibilities	Softvision	PASS-2011-11-24
Reports
[ Litmus], Exploratory work spreadsheet

Update Tests

The following are details of update verifications.

Tested By: ashughes

Platform	Releasetest: report	Beta: report
Windows 2000	PASS - 2011-11-23	PASS - 2011-11-25
Windows XP	PASS - 2011-11-23	PASS - 2011-11-25
Windows Vista	PASS - 2011-11-23	PASS - 2011-11-25
Windows 7 32-bit	PASS - 2011-11-23	PASS - 2011-11-25
Windos 7 64-bit	PASS - 2011-11-23	PASS - 2011-11-25
Mac OS X 10.6	PASS - 2011-11-23	PASS - 2011-11-25
Mac OS X 10.7	PASS - 2011-11-23	PASS - 2011-11-25
Ubuntu 11.10 32-bit	PASS - 2011-11-23	PASS - 2011-11-25
Ubuntu 11.10 64-bit	PASS - 2011-11-23	PASS - 2011-11-25

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

Query	VERIFIED	Status	Report
QA+	VERIFIED 26 / 57		spreadsheet

Issues Found

Bug ID	Summary	Reporter	Status
bug 705132	NoScript affects the Check for Updates User Interface and does not update after upgrading Firefox	Simona	INVALID
bug 705174	Scrolling is sluggish on ftp.mozilla.org	ashughes	NEW (not a regression)

Known Issues

Bug ID	Summary	Status	Blocker?
bug 703216	Html5 youtube video not rendered smoothly when using expand/shrink button	INVALID	NO
bug 701404	Silverlight plugin keeps on crashing on MAC OS X 10.7	NEW	NOT YET
bug 700835	[Mac] Java 6 build 1.6.0_29-b11-402 crashes after tab close	FIXED	NO
bug 697696	Highlight issue in amazon's password box	NEW	NO
bug 697725	Amazon tab favicon flickers when page is loading	UNCONFIRMED	NO
bug 681922	In google docs Upload -> Files has no highlighting	UNCONFIRMED	NO
bug 692446	Firefox prompts to install windows media player plugin even though it is already installed	NEW	NOT YET
bug 679701	[Mac] Embedded window media player plugin is not displayed properly (Flip4Mac)	INVALID?	NO
bug 585816	no thumbnail when dragging tabs	WONTFIX	NO
bug 671535	Videos fail to play on Windows 2000	WONTFIX	NO
bug 694288	Vimeo.com - embedded video player is not displayed on Windows 2000	WONTFIX	NO
bug 638741	right click on flash content displays two context menus sometimes	DUPE?	NO
bug 694690	Youtube HTML5 Video's Fullscreen Button does not work.	WORKSFORME	NO
bug 699175	The current theme is not active when updating from Firefox 7.0b6 to Firefox 8.0b6	FIXED	NO

Beta 4

The following tasks need to be completed to sign-off Beta 4

Task	Assignee	Status
Automated Tests	ashughes	SIGNED-OFF - 2011-11-30
Manual Tests	Softvision	SIGNED-OFF - 2011-12-02
Update Tests	ashughes	SIGNED-OFF - 2011-12-02
Bug Fix Verifications	ashughes, Softvision	SIGNED-OFF - 2011-12-02
Issues Found	ashughes, Softvision	SIGNED-OFF - 2011-12-02
Known Issues	ashughes, Softvision	SIGNED-OFF - 2011-12-02
Links
Builds, Litmus Testrun, Mozmill Dashboard

Automated Tests

The following are the detailed results of the automated BFT test runs.

BFTs (tested by ashughes)

Platform	Result: report
Windows 2000	PASS - 2011-11-30
Windows XP	PASS - 2011-11-30
Windows Vista	PASS - 2011-11-30
Windows 7 32-bit	PASS - 2011-11-30
Windows 7 64-bit	PASS - 2011-11-30
Mac OS X 10.6	PASS - 2011-11-30
Mac OS X 10.7	PASS - 2011-11-30
Ubuntu 11.10 32-bit	PASS - 2011-11-30
Ubuntu 11.10 64-bit	PASS - 2011-11-30

l10n Testruns (tested by ashughes) [optional]

Platform	BFTs: [ report]	Updates: [ report]
Windows XP
Mac OS X 10.7
Ubuntu 9.10 32-bit

NOTE: These testruns are optional due to stability issues in the tests / Mozmill

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

Platform	Assignee	Result
Windows 2000	Simona	PASS - 2011-12-02
Windows XP	Paul	PASS - 2011-12-02
Windows Vista	Simona	PASS - 2011-12-02
Windows 7 32-bit	Ioana	PASS - 2011-12-02
Windows 7 64-bit	Paul	PASS - 2011-12-02
Mac OS X 10.6	VlagG	PASS - 2011-12-02
Mac OS X 10.7	VladG	PASS - 2011-12-02
Ubuntu 11.10 32-bit	Ioana	PASS - 2011-12-02
Ubuntu 11.10 64-bit	Virgil	PASS - 2011-12-02
Reports
Litmus

Focused Tests

The following are the detailed results of the focused feature testing. While testing, keep an eye out for inconsistency, incompatibility, crashes, and hangs. Any issues should be reported in the Issues Found section below.

Feature	Assignee	Status
Plugins Verify stability and compatibility of Flash, Java, Silverlight, QuickTime, etc Report plug-in versions and sites tested	Virgil, VladG, Simona, Paul	PASS - 2011-12-02
Performance Do some stress testing of Firefox 8, monitoring CPU and memory usage, then compare the same testing in Firefox 9 (keep an eye out for crashes/hangs) Report testing performed and comparative CPU/memory usage	Virgil, VladG, Simona, Paul	PASS - 2011-12-02
Rich Profile Updates Populate a profile with bookmarks, history, session data, form data, saved passwords, app-tabs, panorama groups, themes, add-ons, etc on a previous version of Firefox and update to Firefox 9.0b2 on betatest channel Report any lost data, crashes, and inconsistencies	Virgil, VladG, Simona, Paul	PASS - 2011-12-02
Anti-virus Integration Check that updating from a previous version of Firefox with some popular Anti-virus software installed, including add-ons (Norton, McAfee, AVG, ZoneAlarm, etc) Report AV Software and version used during testing and any incompatibilities	Simona, Paul	PASS - 2011-12-02
Reports
Litmus, Exploratory work spreadsheet

Update Tests

The following are details of update verifications.

Tested By: ashughes

Platform	Releasetest: report	Beta: report
Windows 2000	PASS - 2011-12-01	PASS - 2011-12-01
Windows XP	PASS - 2011-12-01	PASS - 2011-12-01
Windows Vista	PASS - 2011-12-01	PASS - 2011-12-01
Windows 7 32-bit	PASS - 2011-12-01	PASS - 2011-12-01
Windows 7 64-bit	PASS - 2011-12-01	PASS - 2011-12-01
Mac OS X 10.6	PASS - 2011-12-01	PASS - 2011-12-01
Mac OS X 10.7	PASS - 2011-12-01	PASS - 2011-12-01
Ubuntu 11.10 32-bit	PASS - 2011-12-01	PASS - 2011-12-01
Ubuntu 11.10 64-bit	PASS - 2011-12-01	PASS - 2011-12-01

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

QA+: 3 / 57 VERIFIED (report)

Issues Found

Bug ID	Summary	Reporter	Status
bug 705174	Scrolling is extremely slow on ftp.mozilla.org/pub/addons onLoad	ashughes	NEW

Known Issues

Bug ID	Summary	Status	Blocker?
bug 701404	Silverlight plugin keeps on crashing on MAC OS X 10.7	NEW
bug 697696	Highlight issue in amazon's password box	NEW
bug 697725	Amazon tab favicon flickers when page is loading	UNCONFIRMED
bug 681922	In google docs Upload -> Files has no highlighting	UNCONFIRMED
bug 692446	Firefox prompts to install windows media player plugin even though it is already installed	NEW
bug 679701	[Mac] Embedded window media player plugin is not displayed properly (Flip4Mac)	INVALID?
bug 638741	right click on flash content displays two context menus sometimes	DUPE?
bug 705174	Scrolling is sluggish on ftp.mozilla.org	NEW (not a regression)

Community Support

This section details how community can help test Beta 4. If you want to help test Firefox, please check below for areas we need help.

Download the Firefox 9.0b4 candidate builds and use it for your day to day tasks
Run some smoketests (you will need to sign up for a Litmus account if you don't have one)
Verify some bug fixes
Run some focused and exploratory tests

Please record all your test results using this etherpad document.

You can file any bugs found using this template (you will need to sign up for a Bugzilla account if you don't have one)

Any questions can be directed to Anthony Hughes via email or ashughes via IRC

Contributors

The following people signed up and help us triage bugs:

jsj0809

Beta 5

The following tasks need to be completed to sign-off Beta 5

Automated Tests

The following are the detailed results of the automated BFT test runs.

BFTs (tested by ashughes)

Platform	Result: report
Windows 2000	PASS - 2011-12-07
Windows XP	PASS - 2011-12-07
Windows Vista	PASS - 2011-12-07
Windows 7 32-bit	PASS - 2011-12-07
Windows 7 64-bit	PASS - 2011-12-07
Mac OS X 10.6	PASS - 2011-12-07
Mac OS X 10.7	PASS - 2011-12-07
Ubuntu 11.10 32-bit	PASS - 2011-12-07
Ubuntu 11.10 64-bit	PASS - 2011-12-07

l10n Testruns (tested by ashughes) [optional]

Platform	BFTs: [ report]	Updates: [ report]
Windows XP
Mac OS X 10.7
Ubuntu 9.10 32-bit

NOTE: These testruns are optional due to stability issues in the tests / Mozmill

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

Platform	Assignee	Result
Windows 2000	VladG	PASS - 2011-12-08
Windows XP	Virgil	PASS - 2011-12-08
Windows Vista	Ioana	PASS - 2011-12-08
Windows 7 32-bit	Mihaela	PASS - 2011-12-08
Windows 7 64-bit	Mihaela	PASS - 2011-12-08
Mac OS X 10.6	VladG	PASS - 2011-12-08
Mac OS X 10.7.2	VladG	PASS - 2011-12-08
Ubuntu 11.10 32-bit	Ioana	PASS - 2011-12-08
Ubuntu 11.10 64-bit	Virgil	PASS - 2011-12-08
Reports
Litmus

Focused Tests

The following are the detailed results of the focused feature testing. While testing, keep an eye out for inconsistency, incompatibility, crashes, and hangs. Any issues should be reported in the Issues Found section below.

Feature	Assignee	Status
Plugins Verify stability and compatibility of Flash, Java, Silverlight, QuickTime, etc Report plug-in versions and sites tested in this spreadsheet	Softvision	PASS - 2011-12-08
Performance Do some stress testing of Firefox 8, monitoring CPU and memory usage, then compare the same testing in Firefox 9 (keep an eye out for crashes/hangs) Report testing performed and comparative CPU/memory usage	Softvision	PASS - 2011-12-08
Rich Profile Updates Populate a profile with bookmarks, history, session data, form data, saved passwords, app-tabs, panorama groups, themes, add-ons, etc on a previous version of Firefox and update to Firefox 9.0b2 on betatest channel Report any lost data, crashes, and inconsistencies	Softvision	PASS - 2011-12-08
Anti-virus Integration Check that updating from a previous version of Firefox with some popular Anti-virus software installed, including add-ons (Norton, McAfee, AVG, ZoneAlarm, etc) Report AV Software and version used during testing and any incompatibilities	Softvision	PASS - 2011-12-08
Reports
Litmus, Exploratory work spreadsheet

Update Tests

The following are details of update verifications.

Tested By: ashughes

Platform	Releasetest: report	Beta: report
Windows 2000	PASS - 2011-12-08	PASS - 2011-12-09
Windows XP	PASS - 2011-12-08	PASS - 2011-12-09
Windows Vista	PASS - 2011-12-08	PASS - 2011-12-09
Windows 7 32-bit	PASS - 2011-12-08	PASS - 2011-12-09
Windows 7 64-bit	PASS - 2011-12-08	PASS - 2011-12-09
Mac OS X 10.6	PASS - 2011-12-08	PASS - 2011-12-09
Mac OS X 10.7	PASS - 2011-12-08	PASS - 2011-12-09
Ubuntu 11.10 32-bit	PASS - 2011-12-08	PASS - 2011-12-09
Ubuntu 11.10 64-bit	PASS - 2011-12-08	PASS - 2011-12-09

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

QA+: 21 / 61 VERIFIED (report)

Issues Found

Bug ID	Summary	Reporter	Status
bug 705174	Scrolling is extremely slow on ftp.mozilla.org/pub/addons onLoad	ashughes	NEW
bug 708557	Html5 youtube context menu not persistent when right clicking	Virgil	NEW

Known Issues

Bug ID	Summary	Status	Blocker?
bug 701404	Silverlight plugin keeps on crashing on MAC OS X 10.7	NEW
bug 697696	Highlight issue in amazon's password box	NEW
bug 697725	Amazon tab favicon flickers when page is loading	UNCONFIRMED
bug 681922	In google docs Upload -> Files has no highlighting	UNCONFIRMED
bug 692446	Firefox prompts to install windows media player plugin even though it is already installed	NEW
bug 679701	[Mac] Embedded window media player plugin is not displayed properly (Flip4Mac)	INVALID?
bug 638741	right click on flash content displays two context menus sometimes	DUPE?
bug 705174	Scrolling is sluggish on ftp.mozilla.org	NEW (not a regression)

Community Support

This section details how community can help test Beta 5. If you want to help test Firefox, please check below for areas we need help.

Download the Firefox 9.0b5 candidate builds and use it for your day to day tasks
Run some smoketests (you will need to sign up for a Litmus account if you don't have one)
Verify some bug fixes
Run some focused and exploratory tests

Please record all your test results using this etherpad document.

You can file any bugs found using this template (you will need to sign up for a Bugzilla account if you don't have one)

Any questions can be directed to Anthony Hughes via email or ashughes via IRC

Contributors

The following people signed up and help us triage bugs:

Beta 6

The following tasks need to be completed to sign-off Beta 6

Automated Tests

The following are the detailed results of the automated BFT test runs.

BFTs - tested by ashughes

Platform	Result: report
Windows 2000	PASS - 2011-12-13
Windows XP	PASS - 2011-12-13
Windows Vista	PASS - 2011-12-13
Windows 7 32-bit	PASS - 2011-12-13
Windows 7 64-bit	PASS - 2011-12-13
Mac OS X 10.6	PASS - 2011-12-13
Mac OS X 10.7	PASS - 2011-12-13
Ubuntu 11.10 32-bit	PASS - 2011-12-13
Ubuntu 11.10 64-bit	PASS - 2011-12-13

l10n Testruns (tested by ashughes) [optional]

Platform	BFTs: [ report]	Updates: [ report]
Windows XP
Mac OS X 10.7
Ubuntu 9.10 32-bit

NOTE: These testruns are optional due to stability issues in the tests / Mozmill

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

Platform	Assignee	Result
Windows 2000	Virgil	PASS - 2011-12-14
Windows XP	Ioana	PASS - 2011-12-14
Windows Vista	Virgil	PASS - 2011-12-14
Windows 7 32-bit	Mihaela	PASS - 2011-12-14
Windows 7 64-bit	Mihaela	PASS - 2011-12-14
Mac OS X 10.6.8	VladG	PASS - 2011-12-14
Mac OS X 10.7.2	VladG	PASS - 2011-12-14
Ubuntu 11.10 32-bit	Ioana	PASS - 2011-12-14
Ubuntu 11.10 64-bit	Virgil	PASS - 2011-12-14
Reports
Litmus

Update Tests

The following are details of update verifications.

Tested By: ashughes

Platform	Releasetest: report	Beta: report
Windows 2000	PASS - 2011-12-14	PASS - 2011-12-15
Windows XP	PASS - 2011-12-14	PASS - 2011-12-15
Windows Vista	PASS - 2011-12-14	PASS - 2011-12-15
Windows 7 32-bit	PASS - 2011-12-14	PASS - 2011-12-15
Windows 7 64-bit	PASS - 2011-12-14	PASS - 2011-12-15
Mac OS X 10.6	PASS - 2011-12-14	PASS - 2011-12-15
Mac OS X 10.7	PASS - 2011-12-14	PASS - 2011-12-15
Ubuntu 11.10 32-bit	PASS - 2011-12-14	PASS - 2011-12-15
Ubuntu 11.10 64-bit	PASS - 2011-12-14	PASS - 2011-12-15

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

QA+: 16 / 27 VERIFIED (report)

QA?: 5 / 28 bugs escalated by Alex Keybl

Softvision Sprint: 6 / 6 VERIFIED

Staff Sprint: 10 / 15 VERIFIED

Issues Found

Bug ID	Summary	Reporter	Status
bug 710701	New tab button underlays the tab scrolling arrow when at least six app tabs exist	Mihaela	WONTFIX

Known Issues

Bug ID	Summary	Status	Blocker?
bug 705174	Scrolling is extremely slow on ftp.mozilla.org/pub/addons onLoad	NEW
bug 708557	Html5 youtube context menu not persistent when right clicking	NEW
bug 701404	Silverlight plugin keeps on crashing on MAC OS X 10.7	NEW
bug 697696	Highlight issue in amazon's password box	NEW
bug 697725	Amazon tab favicon flickers when page is loading	UNCONFIRMED
bug 681922	In google docs Upload -> Files has no highlighting	UNCONFIRMED
bug 692446	Firefox prompts to install windows media player plugin even though it is already installed	NEW
bug 679701	[Mac] Embedded window media player plugin is not displayed properly (Flip4Mac)	INVALID?
bug 638741	right click on flash content displays two context menus sometimes	DUPE?

Community Support

This section details how community can help test Beta 5. If you want to help test Firefox, please check below for areas we need help.

Download the Firefox 9.0b6 candidate builds and use it for your day to day tasks
Run some smoketests (you will need to sign up for a Litmus account if you don't have one)
Verify some bug fixes
Run some focused and exploratory tests

Please record all your test results using this etherpad document.

You can file any bugs found using this template (you will need to sign up for a Bugzilla account if you don't have one)

Any questions can be directed to Anthony Hughes via email or ashughes via IRC

Contributors

The following people signed up and help us triage bugs:

Final

The following tasks need to be completed to sign-off the final release

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows 2000	PASS - 2011-12-17
Windows XP	PASS - 2011-12-17
Windows Vista	PASS - 2011-12-17
Windows 7 32-bit	PASS - 2011-12-17
Windows 7 64-bit	PASS - 2011-12-17
Mac OS X 10.6	PASS - 2011-12-17
Mac OS X 10.7	PASS - 2011-12-17
Ubuntu 11.10 32-bit	PASS - 2011-12-17
Ubuntu 11.10 64-bit	PASS - 2011-12-17

Smoketests

The following are the detailed results of the manual smoketests.

Platform	Assignee	Result
Windows 2000	Simona	PASS - 2011-12-19
Windows XP	Ioana	PASS - 2011-12-19
Windows Vista	Simona	PASS - 2011-12-19
Windows 7 32-bit	Mihaela	PASS - 2011-12-19
Windows 7 64-bit	Mihaela	PASS - 2011-12-19
Mac OS X 10.6.8	VladG	PASS - 2011-12-19
Mac OS X 10.7.2	VladG	PASS - 2011-12-19
Ubuntu 11.10 32-bit	Ioana	PASS - 2011-12-19
Ubuntu 11.10 64-bit	Virgil	PASS - 2011-12-19
Reports
Litmus

Update Tests

The following are details of update verifications.

Due to a top-crasher, Firefox 9.0 will be pushed to Windows-only, Firefox 9.0.1 will be pushed to Mac/Linux-only (see Firefox 9.0.1 test plan for details)

Tested By: ashughes

Platform	Betatest: report	Releasetest: report	Release: report
Windows 2000	PASS - 2011-12-18	PASS - 2011-12-19	PASS - 2011-12-20
Windows XP	PASS - 2011-12-18	PASS - 2011-12-19	PASS - 2011-12-20
Windows Vista	PASS - 2011-12-18	PASS - 2011-12-19	PASS - 2011-12-20
Windows 7 32-bit	PASS - 2011-12-18	PASS - 2011-12-19	PASS - 2011-12-20
Windows 7 64-bit	PASS - 2011-12-18	PASS - 2011-12-19	PASS - 2011-12-20
Mac OS X 10.6	PASS - 2011-12-18	PASS - 2011-12-19	PASS - 2011-12-20
Mac OS X 10.7	PASS - 2011-12-18	PASS - 2011-12-19	PASS - 2011-12-20
Ubuntu 11.10 32-bit	PASS - 2011-12-18	PASS - 2011-12-19	PASS - 2011-12-20
Ubuntu 11.10 64-bit	PASS - 2011-12-18	PASS - 2011-12-19	PASS - 2011-12-20

Note: a PASS on Mac and Ubuntu with the release channel means an update to 8.0.1 was found and installed

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

QA+: 2 / 11 VERIFIED (report)

Issues Found

Bug ID	Summary	Reporter	Status
bug 711929	Full screen and loading animation issues when playing OGG videos	Simona	RESOLVED FIXED

Regression Mitigation

Track testing progress here to mitigate some late-game regressions.

P1: Identify a reproducible test case for Firefox 10 GC/CC degradation
P2: Compare the effect of that test case on Firefox 9

Bugs

bug 711794 - Firefox 9.0 Crash [@ js::Shape::finalize ]
bug 711900 - Make Firefox 10 not have more GC/CC related hangs than Firefox 9
bug 707162 - Regression in GC scheduling between Firefox 9 and 10?
bug 712197 - Memory leak and high cpu load on crash-stats.mozilla.com when trying to drag a tab while the content

Test Scenarios

Focus on navigating to many pages loaded with stuff that could cause the system to GC/CC stuff, like ads, video, live content, flash, etc ... Also I would use multiple tabs to try and push the memory limits of the system under test. Firefox 8 is our baseline.

10 Gmail tabs, 10 Facebook tabs, and 10 Slashdot tabs (all 30 logged in), then putting up the http://mrdoob.com/131/Clouds demo
- Picked this because I frequently leave multiples of these particular sites open, and have had slowdown problems in past.
- Compared behavior b/w 8.0.1 release, 9.0b6, 10.0a2
- Timings varied on whether tabs were created in-session (longer GCs) or loaded from previous session (shorter GCs)
- Numbers below represent -after- all pages loaded (no more progress icons in tabs) and with Mr. Doob as the top tab.
- 8.0 and 9.0 were pretty similar, around 400-500ms GCs depending on above. GCs tended to happen less often over time. Pages deactivating in BG?
- Cycling through tabs seemed to restart the GCs if they'd slowed down, supporting above.
- 10.0 had overall slightly longer GCs, more like 450-550ms. GCs did -not- seem to fall off as quickly, and often saw 2 GCs per CC.
- CCs were usually 40ms-60ms on all builds. Could not find a clear correlation.
- 10.0 had a bunch of swing on CCs and GCs shortly after all pages loaded. Saw GCs up to 2500ms, CCs up to 1000ms. Eventually settled to steady state as above.
- 8.0 and 9.0 had much smaller swing. Rarely saw GCs above steady state, rare CCs up to maybe 300ms.
- Tried doing idle vs. non-idle testing by steering around Mr. Doob Clouds. Didn't notice a difference in behavior.
- GC pauses all -definitely- noticeable.
- Had a couple of very slow closes (30s+), one in 9.0 and one in 10.0. I do not know what led to them, as retrying same scenario closed relatively quickly (5-8s). Have anecdotally seen this in 8.0 as well, so don't suspect new behavior.
- Conclusions: vague. It's possibly a good tab set to expose pauses and amplify slowdown behavior. But I saw so much variation b/w scenarios that I couldn't make any correlations I'd stand behind. I do not, however, currently think 9 is significantly worse than 8 based on what I saw.

open several etherpads, gmail tabs, facebook tabs, twitter tabs, in Fx10 see if I notice the scrolling problem, check the numbers next to GC/CC durations
compare GC/CC numbers between Firefox 8, 9 and 10
Download the test sessionstore file, restore and, open/close lots of bugzilla tabs, letting sit there for a bit, and closing various tbpl pages (both the full page for inbound and pages for particular try pushes), various blog posts, etc, etc.

(Henrik) Tests with sessionstore.js file above
- I let Firefox 10 run for about 1 day but was not able to get CC/GC times larger than 400ms - so no need for me to run Firefox 9 as long as I can't reproduce the core issue
- Also tested by putting my MBP into hibernation mode and woke it up again a couple of times - no increase of the times were noticeable
- I was able to notice lags during continuously scrolling pages with two fingers up and down
  - Whenever a CC/GC happens which is around 100ms or larger you can clearly see that Firefox hangs and scrolling is not possible during that time
  - A fresh profile and lesser memory usage of Firefox doesn't show this issue because CC/GC times are around 15-20ms
  - The same short hangs I experience in Firefox 8, but Firefox 9 is not worse nor better. Both seem to have the same timing with Boris profile 300-400ms for GC, and 150-200ms for CC.

(Tony)
- Environments
  - Mac OSX 10.7.2, i7, 4GB Ram
  - firefox 9 rc
  - firefox 8.0.1

- Observations
  - both profiles have about 30 tabs open
  - tabs were sync'd over
  - sites were a mix of facebook, gmail, news sites, yahoo pages, flash video, mozilla.com pages, etc.
  - CC ranges around 28-60 ms
  - GC ranges between 130-160ms
  - Firefox 9 memory fluctuates around ~600Mb, while Firefox 8.0.1 is around ~515Mb
  - kinetic scrolling through long pages pauses for a second or two, but picks up quickly on both apps.

- Conclusion
  - No major noticable differences between the two versions. Fx9 seems a bit higher in memory and GC/CC, but not noticably in a regression way.

Note: to test you will need to set javascript.options.mem.log = true in about:config and check error console for GC and CC timings

Knowns

GC/CC Hangs

Affects Firefox 10 worse than Firefox 9
Firefox 9 is no worse than Firefox 8
Averages (GC vs CC):
- 8.0: 601 | 142
- 9.0b6: 603 | 145
- 9.0RC: 604 | 159
- 10.0a2: 665 | 149

Bug 711794

First correlation found, please help via this etherpad
low volume in Firefox 8 and spiked in 9.0b6
Mac OS X 10.5, 10.6 and 10.7, and Linux
no obvious addon correlations
89% of the users have uptime < 1 min
Pending feedback from Cheng (SUMO) about users specific configurations

Unknowns

How wide-spread is the issue on Firefox 9?
What specifically causes the issue?
What is the regression range? is it a regression?

Investigation - Softvision

Firefox 9.0.1

The following tasks need to be completed to sign-off the 9.0.1 final release. The focus of this release will be on Mac and Linux only.

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows 2000	PASS - 2011-12-20
Windows XP	PASS - 2011-12-20
Windows Vista	PASS - 2011-12-20
Windows 7 32-bit	PASS - 2011-12-20
Windows 7 64-bit	PASS - 2011-12-20
Mac OS X 10.6	PASS - 2011-12-20
Mac OS X 10.7	PASS - 2011-12-20
Ubuntu 11.10 32-bit	PASS - 2011-12-20
Ubuntu 11.10 64-bit	PASS - 2011-12-20

Smoketests

The following are the detailed results of the manual smoketests. Tests will be run for ~~Mac and Linux only~~ all platforms.

Platform	Assignee	Result
Windows 2000	marcia	PASS - 2011-12-21
Windows XP	Tony	PASS - 2011-12-21
Windows Vista	Al	PASS - 2011-12-21
Windows 7 32-bit	Geo	PASS - 2011-12-21
Windows 7 64-bit
Mac OS X 10.6.8	VladG	PASS - 2011-12-21
Mac OS X 10.7.2	VladG	PASS - 2011-12-21
Ubuntu 11.10 32-bit	Virgil	PASS - 2011-12-21
Ubuntu 11.10 64-bit	Virgil	PASS - 2011-12-21
Reports
Litmus

Update Tests

The following are details of update verifications.

Tested By: ashughes

Platform	Releasetest: report	Release: report
Windows 2000	PASS - 2011-12-21	PASS - 2011-12-21
Windows XP	PASS - 2011-12-21	PASS - 2011-12-21
Windows Vista	PASS - 2011-12-21	PASS - 2011-12-21
Windows 7 32-bit	PASS - 2011-12-21	PASS - 2011-12-21
Windows 7 64-bit	PASS - 2011-12-21	PASS - 2011-12-21
Mac OS X 10.6	PASS - 2011-12-21	PASS - 2011-12-21
Mac OS X 10.7	PASS - 2011-12-21	PASS - 2011-12-21
Ubuntu 11.10 32-bit	PASS - 2011-12-21	PASS - 2011-12-21
Ubuntu 11.10 64-bit	PASS - 2011-12-21	PASS - 2011-12-21

Spotchecks

Automatic updates for 9.0 -> 9.0.1: PASS
Manual updates for 8.0 -> 9.0.1: PASS

Issues Found

Bug ID	Summary	Reporter	Status
bug 712790	GetPersonas.com CDN permissions issues	marcia reported to fligtar	NEW

Regression Mitigation

Built from changeset: https://hg.mozilla.org/releases/mozilla-release/rev/b78fe362789b

The following bug was backed out bug 708572 in 9.0.1.

If Firefox 9.0.1 builds are not available pre-qualify using smaug's tryserver build

Context (bug 711794)

First correlation found, please help via this etherpad
low volume in Firefox 8 and spiked in 9.0b6
Mac OS X 10.5, 10.6 and 10.7, and Linux. On Linux it has not been reproduced.
Crash is reproducible on Mac with the Nectar toolbar xpi which is Bug 711794 as well as other toolbars related to http://www.freecause.com/clients_clientlist/
89% of the users have uptime < 1 min

Test Scenarios specific to the crash

Run the following test scenarios on Firefox 9.0b6 before confirming on 9.0.1 (this confirms you can reproduce the bug).

Platforms

P1: Mac OS X 10.6, 10.7
P2: Mac OS X 10.5, Linux
Also ensure some test coverage of 32-bit mode OS

Crash Signatures

Mac OS X 10.6, 10.7 - js::Shape::finalize
Mac OS X 10.5 - js::Shape::removeChild
Linux - js::Shape::finalize

Scenario 1

Set pref extensions.blocklist.enabled = false and restart Firefox
Install the Nectar toolbar from Bugzilla or Nectar
Wait for doorhanger for Toolbar options.
Cancel at the option dialog.
Cancel at the Firefox dialog.
Confirm no Crash

Scenario 2

Set pref extensions.blocklist.enabled = false and restart Firefox
Install the Nectar toolbar from Bugzilla or Nectar
After doorhanger comes down press Install button.
Restart Firefox.
Confirm no crash.

Scenario 3

Set pref extensions.blocklist.enabled = false and restart Firefox
Install the Nectar toolbar from Bugzilla or Nectar
After doorhanger comes down press Install button.
Restart Firefox.
Press the button next to "Points Unavailable" so that it opens a doorhanger.
Confirm no crash.

Scenario 4

Start Firefox in 32 bit mode on Mac. You can do this by right clicking on the Firefox Icon in "Get Info" and checking the box that says "Open in 32 bit mode."
Set pref extensions.blocklist.enabled = false and restart Firefox
Install the Nectar toolbar from Bugzilla or Nectar
After doorhanger comes down press Install button.
Restart Firefox.
Press the button next to "Points Unavailable" so that it opens a doorhanger.
Confirm no crash.

Scenario 5

Start with a Firefox 9 Beta 5 build or earlier. (assumes updates are available on betatest -- you may need to test with a previous release build, like Firefox 8.0.1)
Set pref extensions.blocklist.enabled = false and restart Firefox
Install the Nectar toolbar from Bugzilla or Nectar
After doorhanger comes down press Install button.
Manually check for updates to 9.0.1. You have to change the update channel to betatest or releasetest to get that update offer.https://quality.mozilla.org/docs/firefox/updates-testing/ explains how to do that.
Confirm no crash following software update.

Results

jrgm
macosx10.7
9.0.1-candidates/build1/mac 64bit
scenarios: 1 to 4 - confirm no crash
scenario 5: no update available via 'About Firefox' with either 8.0 or 9.0b5 (perhaps builds not publically there yet?)
scenario 1-2 below: running a hacked up script to revisit some parts of my history in the browser, no crashes after a few thousand page visits.

marcia
Mac 10.5.8
Ran Scenarios 1-3 with no crashes

Cameron
- Mac 10.7.2
- Ran Scenarios 1-5 with no crashes
- Dogfooding today

General Test Scenarios (dogfooding)

Scenario 1

Use a dirty profile and run the hell out of the browser.
Install top addons such as Adblock, Firebug, etc and restart the browser.
Visit your favorite sites. Open lots of windows. Use Panorama.

Scenario 2

Use a clean profile and run the hell out of the browser.
Install addons and restart the browser.
Visit your favorite sites and load up some tabs
Add some Personas.

Results

name, platforms tested, firefox versions, scenario, detailed results, etc
davehunt, Mac OS X 10.7.2, Firefox 9.0.1, Scenarios 1-4, PASSED - Unable to replicate crash.
Softvision detailed results: https://mozqa.etherpad.mozilla.org/30
- Performed on Ubuntu 11.10 (i686, x86_64), Mac OS 10.5.8, 10.6.8 , 10.7.2
- No crashes occurred after following scenarios 1-5
- Two other crashes occurred in Firefox 9.0.1 on Ubuntu 11.10 i686 and Mac OS 10.5.8 (details in Etherpad)

Summary of Results

Raw data and results are in this etherpad

FIXED

Nectar-toolbar regression appears to be fixed in 9.0.1 -- does not crash

NEW

Loading gsp.ro and ziuadecj.ro in two tabs at the same time results in a hang
- Firefox 7 hangs for 10 seconds on load then eventually clears
- Firefox 9.0.1 does not hang on load, 30s hang on scrolling then Flash crashes (Shockwave Flash 11.1 r102)
Favicon regression on Mac - filed bug 712496 - duped to bug 705516 (FIXED in 10, not shipping in 9.0.1)
- Workaround is to set image.mem.decodeondraw to false -- SUMO article written up
We're reintroducing 702813 by backing out the back out in 708572, which is tracked in 711794 (Mac & Linux only)
- Implication is potential CC loss on sites like IRCCloud and AMCTV - was probably not 100% fixed with original fix

CRASHES

NEW libpthread-2.13.so@0x8d87 - filed in bug 712680 - occurs when loading getpersonas.com
NEW xpc::XrayWrapper<JSCrossCompartmentWrapper>::createHolder - loaded 26 tabs in 4 panorama groups then open Tools > Addons > Get Addons
NEW nsXULTemplateBuilder::Uninit - correlated to bug 700493 (Yandex Toolbar), reported in Firefox 8
bug 701425 - top-crash fixed with Norton Internet Security 2011/2012
- SUMO article will be created

FINAL SIGN-OFF CRITERIA

[team] getpersonas.com hang/crashiness confirmations
[marcia] Add test to Litmus to confirm submitting crash reports via Crash Reporter UI and run across all platforms
[ashughes] Updates automation on all channels

Releases/Firefox 9/Test Plan

Contents

Firefox 9 Test Plan

Summary

Details

Automation

BFTs

Endurance

Bug Triage

Features

Full Functional Tests

Website Compatibility

Nightly Merge

Aurora Merge

Updates

Beta 1

Automated Tests

Manual Tests

Smoketests

Focused Tests

Update Tests

Bug Fix Verifications

Issues Found

Known Issues

Beta 2

Automated Tests

Manual Tests

Smoketests

Focused Tests

Update Tests

Bug Fix Verifications

Issues Found

Known Issues

Beta 3

Automated Tests

Manual Tests

Smoketests

Focused Tests

Update Tests

Bug Fix Verifications

Issues Found

Known Issues

Beta 4

Automated Tests

Manual Tests

Smoketests

Focused Tests

Update Tests

Bug Fix Verifications

Issues Found

Known Issues

Community Support

Beta 5

Automated Tests

Manual Tests

Smoketests

Focused Tests

Update Tests

Bug Fix Verifications

Issues Found

Known Issues

Community Support

Beta 6

Automated Tests

Manual Tests

Update Tests

Bug Fix Verifications

Issues Found

Known Issues

Community Support

Final

Automated Tests

Smoketests

Update Tests

Bug Fix Verifications

Issues Found

Regression Mitigation

Bugs

Test Scenarios

Knowns