QA/Firefox3/TestResults/M9

« QA/Firefox3/TestResults

Firefox 3 M9 Test Results

Related Pages

Most of these pages are linked below in this document.

• Smoketests from 2007-11-06
• Wishlist of Release Notes for M9
• Extensions Test Results
• L10n Spotchecks
• Leopard & Mac Testing
• RC3 Smoketest Results
• Results from test days

Contact Info

Build: John O'Duinn (joduinn)
QA: Tony (tchung)

Schedule

Code Freeze

11/2

Code Build

11/7

QA test signoff

ETA 11/13

Release Notes

• Wishlist of Release Notes for M9

Build Location and IDs

RC1 builds

• Mac Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b1) Gecko/2007110618 Firefox/3.0b1
• Vista Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
• XP Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
• Linux Build ID: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b1) Gecko/2007110618 Firefox/3.0b1

---

RC3 builds

• Mac Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b1) Gecko/2007110903 Firefox/3.0b1
• Vista Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1
• Linux Build Id: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b1) Gecko/2007110903 Firefox/3.0b1
• XP Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1

Features landed

Firefox

• List coming soon

Gecko & Platform

• List coming soon

Strategy

• Areas Covered:
  • Smoketests
  • FFTs
  • M9 and past focused Tests
  • Top Extensions
  • Top Plugins
  • Top Sites (manual by QAE, automation by bc)
  • Memory Leak Testing (handled by Tomcat on debug builds)
  • Lite L10N spotchecks only. (need list of supported locales)
  • M9 bug verifications

• Areas Not Covered:
  • L10N
  • Distribution
  • Updates

Tests are ran against en-US locale only.

Test Results

Types of Tests	Win XP	Win Vista (include Vista Litmus Test Suite)	Mac Tiger	Mac Leopard	Linux
Smoketest	Tomcat - PASS	Tomcat - PASS	Tracy - PASS	N/A	Stephend - PASS
FFT	Tomcat - PASS	Tomcat (partial) - PASS	Tchung - PASS	Marcia - PASS	Tomcat - PASS
Top Sites Testing (see below)	Juanb - PASS bc (automation) - results	N/A	Juanb - PASS	N/A	bc (automation) - results
Top Extensions Testing (see below)	Abillings - PASS	N/A	Abillings - PASS	N/A	Tracy - PASS (only Adblock Plus and Gmail nofication compatible)
Top Plugins Testing (see below)	Stephend - PASS	N/A	Stephend (Marcia, really, in her FFT testing; thanks Marcia!) - PASS	N/A	Tracy - PASS (only able to install Flash and PDF)
l10N Pri 1 locales	Tracy (de, en-GB, pl, fr) - PASS, en-GB is n/a - Tomcat (DE) PASS (one regression found, see results)	N/A	Tchung (ja, es-ES) - PASS	N/A	N/A
Bug Verifications	Everyone - PASS	Everyone - PASS	Everyone - PASS	Everyone - PASS	Everyone - PASS

• Windows XP Smoketest, en-US, - Tomcat
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
  • PASS - addons testing currently not possible, because of no addons ready for maxversion b1

Top Sites Testing

Select the top 50 sites, and run through a combination of various web applications.

• Alexa Top 50
  • Note: Used Alexa's Top Global list
  • Yahoo Mail not working on Fx3: Bug 398381
  • aol.com has a layout problem bug 396821
• Banking and financial institutions
  • need a way to coordinate this with the community

Testcases

• Visit various sites on the list, verifying web compatibility
• Detect Layout issues, Font issues, formatting problems, broken functionality
  • Some baseline comparisons is to use Fx 2.0.0.9, Safari, and Trunk nightly builds
• Change browser font sizes, languages and verify web compatibility
• Operate end to end functionality of a website. Examples:
  • track an auction item on ebay
  • save a book on amazon checkout
  • register for a new email account on Yahoo Mail
  • Create a document on Google docs
  • Visit a secure site and perform online banking

Top Extensions Testing

Select a few top extensions from this list and run tests to install/update/uninstall/disable extensions. Available for download at AMO.

Or, refer to this list: Most-Used Addons

Testcases:

• Enabling extensions and confirming web compatibility
• Disabling or Uninstalling extensions and confirming proper messaging and behavior
• Load about 10+ extensions, and fire them up all at once, verifying functionality and performance
• Updating old extension versions to the latest if applicable
• Switch profiles with different extensions, and confirm they do not get carried over.

Extensions Test Results

Top Plugins Testing

Select a few top plugins on the web. Use various websites supporting these plugins. Available for download at AMO.

Consider:

• Adobe PDF
• Flash
• Java
• Shockwave
• Embedded Win Player
• Embedded Divx player
• Quicktime player
• Viewpoint on both Vista and XP
• Ipix (360 degree view)

Testcases:

• Enabling Plugins and confirming web compatibility
• Disabling Plugins and confirming proper messaging and behavior
• Check about:plugins and Addon Managers to verify plugins still exist
• Updating old plugin versions to the latest

Issue found:

Pre-existing bug 328675

L10N P1 Spotchecks

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Automation: We'll use 2.0.0.9 to create the Minotaur reference files.

P1 Locales to be tested:

• English (en-GB) not available, localizer not ready
• French (fr) PASS
• German (de) PASS
• Japanese (ja) PASS
• Polish (pl) PASS
• Spanish (es-ES) PASS

I put all the beta1 available P1's through a Minotaur check against 2.0.0.9. That looked good; nothing glaringly bad jumped out at me. However, I need to get those files loaded up somewhere so Axel and Mic can have a look.

Testcases and Results are tracked QA/Firefox3/TestResults/M9/L10N.

Bug Verifications

Bugs that are marked RESOLVED FIXED that are checked in for beta. Please verify them.

• Frontend bugs: http://tinyurl.com/22whpr - PASS! Zarro boogs
• Platform bugs: http://tinyurl.com/yvqd4u - PASS! Zarro boogs

Detailed Results

Tchung Results

• Opened Bug 402964 – Import Safari bookmarks shows up in Bookmarks Toolbar. Found on Mac OSX FFT.
• UPDATE: dupe of bug 396259 Opened Bug 403083 - new Hotmail interface does not load in Fx3 beta 1. Not a blocker, but assigned to techevangelism to consult with Microsoft.
• Opened bug 403083 - Search plugins get multi-selected when moving up and down list. minor bug found when running litmus search FFT.
• Opened bug 403942 - Cancelling "add new toolbar" will freeze tab switching functionality. Nominating for b2 since restoring means force > quitting

marcia's leopard bugs and problem areas

• Leopard Bugs and Mac issues uncovered

Tomcat Results

• RC 3 Results all PASS

• FFT`s PASS
  • Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1

Focus testing Windows / Windows Vista : PASS

• • Bugs filed:
    • bug 403497 Places Toolbar Folder not renamed when language version is changed
    • bug 403401 Wrong Link to Store "Firefox on CD" on www.mozilla.org
    • bug 403257 Localized Extension description not maintained when the Add-on is disabled
    • bug 403246 Last line of Anti Phishing Warning Page contains de and en-US text
    • bug 403185 "Open All in Tabs" appends, rather than replaces, current tabset
    • bug 403628 Failed to load module "gnomebreakpad - Error output on Trunk Start with Fedora 8 -
    • bug 392919Breakpad System itself is broken on Fedora 8 - this should block a beta 2 i guess
    • bug 403842 Default RSS Feed (Aktuelle Nachrichten) fails to load on DE Firefox 3 Beta 1 RC - Dupe of Bug 381357
    • bug 404081 "Warn me when i close multiple tabs" Pref can be overridden
    • bug 404074 Clear Private Data Menu is cut off in french firefox 3 beta 1
    • bug 403937 Missing title on Mac for "about Mozilla Firefox" Window

• DE Locale Spot Check - PASS was running into bug 403842, see above

Stephend Results

• Prexisting bug 328675 - I nominated it for "blocking-1.9?", but jst claims plugins aren't really being actively worked on :-(

Juanb results

(topsite)

• Yahoo Mail not working on Fx3: Bug 398381
• aol.com has a layout problem bug Bug 396821

11/9/07 Testday Results

Full Summary here. Highlights:

• 1300 litmus tests by Community
• 500 litmus tests by l10n Community
• 56 failed testcases
• 60 unclear/broken testcases
• 50 new Community Members total, and 35 using Litmus

Live Spot Checks for M9

When we go live at 3 PM it would be great team members could download an M9 build quickly and make sure everything looks ok.

• Windows XP -> abillings - PASS
• Windows Vista -> stephend/abillings (en-US, de) - PASS
• Mac PPC/Intel-> tchung - PASS
• Mac Leopard -> abillings - PASS
• Linux -> stephend - (en-US/nl) PASS

DevTracking Bugs for M9

Refer to dev-tracking bugs at: [M9 firefox]|[M9 platform]