QA/Firefox3/TestResults/M11

« QA/Firefox3/TestResults

Firefox 3 M11 Test Results

Related Pages

Most of these pages are linked below in this document.

• Extensions Test Results
• Update snippets Test Results
• L10n Spotchecks Test Results
• Accessibility Test Results
• Beta 3 Testday Results
• QA Beta 3 Relnotes Tracking

Contact Info

Build: John O'Duinn (joduinn)
QA: Tony (tchung)

Schedule

Code Freeze

1/22

Code Build

ETA 2/4

QA test signoff

ETA 2/10

• Smoketests (2/5-2/6)
• BFTs (2/5-2/7)
• M11 and past focused Tests (2/5-2/10)
• Top Websites (2/7-2/8)
• Major Updates (Fx2 -> Fx3 trial run) (Completed)
• Update snippets from Alpha X, Beta 1,2 -> Beta 3 (2/7 - 2/8)
• Memory Leak Testing (handled by Tomcat on debug builds)
• P1 L10N spotchecks (2/8)
• M11 bug verifications (ongoing)
• Beta 3 Testday (2/8)
• Release notes, Websites (2/9, 2/10)

Build Location and IDs

RC 1: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b3-candidates/rc1/

Build ID's RC1:

• Linux: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b3) Gecko/2008020421 Firefox/3.0b3

• Mac 10.4: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b3) Gecko/2008020418 Firefox/3.0b3

• Mac 10.5: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020418 Firefox/3.0b3

RC2

• Mac 10.4: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3

• Mac 10.5: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3

• Linux: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b3)Gecko/2008020513 Firefox/3.0b3

• WinXP: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b3)Gecko/2008020514 Firefox/3.0b3

• WinVista: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b3)Gecko/2008020514 Firefox/3.0b3

Strategy

• Areas Covered:
  • Smoketests
  • BFTs
  • M11 and past focused Tests
  • Top Websites
  • Major Updates (Fx2 -> Fx3 trial run)
  • Update snippets from Alpha X, Beta 1,2 -> Beta 3
  • Memory Leak Testing (handled by Tomcat on debug builds)
  • P1 L10N spotchecks
  • M11 bug verifications
  • Beta 3 Testday

• Areas Not Covered:
  • localization (String changes) - 3rd party will cover
  • Distribution

Test Results

Types of Tests	Win XP	Win Vista	Mac Tiger	Mac Leopard	Linux
Smoketest	marcia - PASS	marcia - PASS	Tomcat - PASS	Tomcat - PASS	Tomcat - PASS
BFT	Tomcat (running leak tests on FFT) - result	Tomcat (running leak tests) - result	Tomcat (running leak tests on FFT) - result	Marcia - result	Tracy - result - beta2 in progress
Top Websites Testing (see below)	Juanb - result	N/A	Juanb - result	N/A	Tracy - result
Updates Testing (see below)	Tomcat - result	Marcia - result	Tomcat - result	Marcia - result	Tchung - result
l10N Pri 1 locales	Tomcat - result	N/A	Tchung - result	N/A	Tracy - result
Bug Verifications	Everyone - result	Everyone - result	Everyone - result	Everyone - result	Everyone - result

Top Sites Testing

Select the top 50 sites, and run through a combination of various web applications.

• Alexa Top 50
• Banking and financial institutions

Testcases

• Visit various sites on the list, verifying web compatibility
• Detect Layout issues, Font issues, formatting problems, broken functionality
  • Some baseline comparisons is to use Fx 2.0.0.11, Safari, and Trunk nightly builds
• Change browser font sizes, languages and verify web compatibility
• Operate end to end functionality of a website. Examples:
  • track an auction item on ebay
  • save a book on amazon checkout
  • register for a new email account on Yahoo Mail
  • Create a document on Google docs
  • Visit a secure site and perform online banking

Updates Testing

For b3, we will be doing updates of all Alpha and Beta 1 users to Beta 3. Tests will include different locales for beta 1, but only en-US for all past versions. We will be checking partial updates, complete updates, and failed status.

Will be available on betatest, releasetest, and beta channels

• Betatest channel - result
• Releasetest channel - result
• Beta channel (live) - result

Testcases and Results are tracked here.

We will also do a special round of major update testing from Fx2-> Fx3 trial run. Testplan is held here.

• Major Update - result

L10N P1 Spotchecks

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Automation inspection: supported locales will be ran through Minotaur, and results will be reviewed by axel and mic.

Axel's Localization Status Page

P1 Locales to be tested:

• French (fr) result
• Chinese (zh-CN) result
• German (de) result
• Japanese (ja) result
• Spanish (es-ES) result
• Netherlands (nl) result
• Italian (it) result
• Finnish (fi) result

Testcases and Results are tracked here.

Bug Verifications

Bugs that are marked RESOLVED FIXED that are checked in for beta. Please verify them.

Firefox bugs
Platform P1 bugs
Platform P2 bugs

Testday

A beta 3 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found at the QMO post.

Testday results are posted here.

Detailed Results

Tomcats Results

* Smoketests (RC1)

• • Linux
    • Date: 5.February 2008
    • Build id: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b3) Gecko/2008020421 Firefox/3.0b3
    • Status: PASS Results in Litmus

• • Mac 10.4
    • Date: 5.February 2008
    • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b3) Gecko/2008020418 Firefox/3.0b3
    • Status: PASS Results in Litmus (only known bugs found - no regressions)

• • Mac 10.5
    • Date: 5.February 2008
    • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020418 Firefox/3.0b3
    • Status: PASS Results in Litmus

* Smoketest RC 2

• • Mac 10.4
    • Date: 5.February 2008
    • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b3)Gecko/2008020511 Firefox/3.0b3
    • Status: PASS Results in Litmus (only known bugs found - no regressions)

• • Mac 10.5
    • Date: 5.February 2008
    • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
    • Status: PASS Results in Litmus

• • Linux
    • Date: 6.February 2008
    • Build ID: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b3)

Gecko/2008020513 Firefox/3.0b3

• • • Status: PASS - Results in Litmus

• Bugs filed
  • bug 415826 Application Handler -> Web Feed -> Application details is empty

marcia's Results

* Smoketest RC 2

• • Win Vista Business
  • Tested on a QA Lab Machine
    • Date: 6.February 2008
    • Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3
    • Status: PASS Results in Litmus
    • Issues noted: Bug 415968

• • Win XP
  • Tested on a clean Win XP VM
    • Date: 6.February 2008
    • Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3
    • Status: PASS Results in Litmus
    • Issues noted: Mismatch between Text on Crash Reporter and Updater icons. They both say "Mozilla Foundation" while Firefox.exe says corporation. They are digitally signed by the Corporation. Filed Bug 415993

*BFT

• • Mac 10.5
  • Intel build tested on a MacBook Pro; PPC build tested on a G5
    • Date: 5.February 2008
    • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3 and Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
    • Status: In progress Results in Litmus

Live Spot Checks for M10

Live site downloads for M11, after bits are pushed out. This includes verifying websites and release notes.

• Windows XP -> abillings - Result
• Windows Vista -> Tomcat - Result
• Mac PPC/Intel-> Juan - Result
• Mac Leopard -> Marcia - Result
• Linux -> stephend - Result