MozillaWiki

QA/Firefox3/TestResults/M9

< QA‎ | Firefox3‎ | TestResults
Revision as of 06:15, 17 November 2007 by Samuelsidler (talk | contribs) (→‎Top Sites Testing)

Back to QA Fx3 Home

Firefox 3 M9 Test Results

Nightly Smoke Test Results on 11/6/2007

Test Results from the nightly smoke test before the official M9 builds.

Contact Info

Build: John O'Duinn (joduinn)
QA: Tony (tchung)

Schedule

Code Freeze
11/2
Code Build
11/7
QA test signoff
ETA 11/13

Release Notes

Build Location and IDs

RC1 builds

  • Mac Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b1) Gecko/2007110618 Firefox/3.0b1
  • Vista Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
  • XP Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
  • Linux Build ID: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b1) Gecko/2007110618 Firefox/3.0b1

RC3 builds

  • Mac Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b1) Gecko/2007110903 Firefox/3.0b1
  • Vista Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1
  • Linux Build Id: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b1) Gecko/2007110903 Firefox/3.0b1
  • XP Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1

Features landed

Firefox

  • List coming soon

Gecko & Platform

  • List coming soon

Strategy

  • Areas Covered:
    • Smoketests
    • FFTs
    • M9 and past focused Tests
    • Top Extensions
    • Top Plugins
    • Top Sites (manual by QAE, automation by bc)
    • Memory Leak Testing (handled by Tomcat on debug builds)
    • Lite L10N spotchecks only. (need list of supported locales)
    • M9 bug verifications
  • Areas Not Covered:
    • L10N
    • Distribution
    • Updates

Tests are ran against en-US locale only.

Test Results

Types of Tests Win XP Win Vista (include Vista Litmus Test Suite) Mac Tiger Mac Leopard Linux
Smoketest Tomcat - PASS Tomcat - PASS Tracy - PASS N/A Stephend - PASS
FFT Tomcat - PASS Tomcat (partial) - PASS Tchung - PASS Marcia - PASS Tomcat - PASS
Top Sites Testing (see below) Juanb - PASS
bc (automation) - results 		N/A Juanb - PASS N/A bc (automation) - results
Top Extensions Testing (see below) Abillings - PASS N/A Abillings - PASS N/A Tracy - PASS (only Adblock Plus and Gmail nofication compatible)
Top Plugins Testing (see below) Stephend - PASS N/A Stephend (Marcia, really, in her FFT testing; thanks Marcia!) - PASS N/A Tracy - PASS (only able to install Flash and PDF)
l10N Pri 1 locales Tracy (de, en-GB, pl, fr) - PASS, en-GB is n/a - Tomcat (DE) PASS (one regression found, see results) N/A Tchung (ja, es-ES) - PASS N/A N/A
Bug Verifications Everyone - PASS Everyone - PASS Everyone - PASS Everyone - PASS Everyone - PASS
  • Windows XP Smoketest, en-US, - Tomcat
    • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
    • PASS - addons testing currently not possible, because of no addons ready for maxversion b1

Top Sites Testing

Select the top 50 sites, and run through a combination of various web applications.

Testcases

  • Visit various sites on the list, verifying web compatibility
  • Detect Layout issues, Font issues, formatting problems, broken functionality
    • Some baseline comparisons is to use Fx 2.0.0.9, Safari, and Trunk nightly builds
  • Change browser font sizes, languages and verify web compatibility
  • Operate end to end functionality of a website. Examples:
    • track an auction item on ebay
    • save a book on amazon checkout
    • register for a new email account on Yahoo Mail
    • Create a document on Google docs
    • Visit a secure site and perform online banking

Top Extensions Testing

Select a few top extensions from this list and run tests to install/update/uninstall/disable extensions. Available for download at AMO.

Or, refer to this list: Most-Used Addons

Testcases:

  • Enabling extensions and confirming web compatibility
  • Disabling or Uninstalling extensions and confirming proper messaging and behavior
  • Load about 10+ extensions, and fire them up all at once, verifying functionality and performance
  • Updating old extension versions to the latest if applicable
  • Switch profiles with different extensions, and confirm they do not get carried over.

Extensions Test Results

Top Plugins Testing

Select a few top plugins on the web. Use various websites supporting these plugins. Available for download at AMO.

Consider:

  • Adobe PDF
  • Flash
  • Java
  • Shockwave
  • Embedded Win Player
  • Embedded Divx player
  • Quicktime player
  • Viewpoint on both Vista and XP
  • Ipix (360 degree view)

Testcases:

  • Enabling Plugins and confirming web compatibility
  • Disabling Plugins and confirming proper messaging and behavior
  • Check about:plugins and Addon Managers to verify plugins still exist
  • Updating old plugin versions to the latest

Issue found:

Pre-existing bug 328675

L10N P1 Spotchecks

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Automation: We'll use 2.0.0.9 to create the Minotaur reference files.

P1 Locales to be tested:

  • English (en-GB) not available, localizer not ready
  • French (fr) PASS
  • German (de) PASS
  • Japanese (ja) PASS
  • Polish (pl) PASS
  • Spanish (es-ES) PASS

I put all the beta1 available P1's through a Minotaur check against 2.0.0.9. That looked good; nothing glaringly bad jumped out at me. However, I need to get those files loaded up somewhere so Axel and Mic can have a look.

Testcases and Results are tracked here.

Bug Verifications

Bugs that are marked RESOLVED FIXED that are checked in for beta. Please verify them.

Detailed Results

Tchung Results
marcia's leopard bugs and problem areas
Tomcat Results
  • FFT`s PASS
    • Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1
    • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1

Focus testing Windows / Windows Vista : PASS

    • Bugs filed:
      • bug 403497 Places Toolbar Folder not renamed when language version is changed
      • bug 403401 Wrong Link to Store "Firefox on CD" on www.mozilla.org
      • bug 403257 Localized Extension description not maintained when the Add-on is disabled
      • bug 403246 Last line of Anti Phishing Warning Page contains de and en-US text
      • bug 403185 "Open All in Tabs" appends, rather than replaces, current tabset
      • bug 403628 Failed to load module "gnomebreakpad - Error output on Trunk Start with Fedora 8 -
      • bug 392919Breakpad System itself is broken on Fedora 8 - this should block a beta 2 i guess
      • bug 403842 Default RSS Feed (Aktuelle Nachrichten) fails to load on DE Firefox 3 Beta 1 RC - Dupe of Bug 381357
      • bug 404081 "Warn me when i close multiple tabs" Pref can be overridden
      • bug 404074 Clear Private Data Menu is cut off in french firefox 3 beta 1
      • bug 403937 Missing title on Mac for "about Mozilla Firefox" Window


  • DE Locale Spot Check - PASS was running into bug 403842, see above


Stephend Results
  • Prexisting bug 328675 - I nominated it for "blocking-1.9?", but jst claims plugins aren't really being actively worked on :-(
Juanb results
(topsite)
11/9/07 Testday Results

Full Summary here. Highlights:

  • 1300 litmus tests by Community
  • 500 litmus tests by l10n Community
  • 56 failed testcases
  • 60 unclear/broken testcases
  • 50 new Community Members total, and 35 using Litmus

Live Spot Checks for M9

When we go live at 3 PM it would be great team members could download an M9 build quickly and make sure everything looks ok.

  • Windows XP -> Tomcat - TBD
  • Windows Vista -> abillings TBD
  • Mac PPC/Intel-> tchung TBD
  • Mac Leopard -> Marcia
  • Linux -> stephend TBD

DevTracking Bugs for M9

Refer to dev-tracking bugs at: [M9 firefox]|[M9 platform]

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox3/TestResults/M9&oldid=75055"