QA/Firefox3/TestResults/Beta5: Difference between revisions

From MozillaWiki
< QA‎ | Firefox3‎ | TestResults
Jump to navigation Jump to search
Line 67: Line 67:
|-
|-
| Smoketest
| Smoketest
| Tomcat - '''PASS'''
| Tomcat - '''result'''
| juanb - '''PASS'''
| juanb - '''result'''
| marcia - '''PASS'''
| marcia - '''result'''
| marcia - '''PASS'''
| marcia - '''result'''
| Tomcat - '''PASS'''  
| Tomcat - '''result'''  
|-
|-
| BFT
| BFT
| Tomcat - '''PASS'''  
| Tomcat - '''result'''  
| Tomcat (Vista test suite) - '''PASS'''  
| Tomcat (Vista test suite) - '''result'''  
| juanb - '''PASS'''  
| juanb - '''result'''  
| Marcia (Leopard test suite/PPC and Intel) - '''PASS on Intel, PASS on PPC'''   
| Marcia (Leopard test suite/PPC and Intel) - '''result'''   
| Tracy - '''PASS''' ''results in Litmus''
| Tracy - '''result'''
|-
| Top Plugins Testing (see below)
| stephend - '''PASS''' - results in Litmus
| N/A
| stephend - '''PASS'''- results in Litmus
| N/A
| Abillings - '''PASS'''  
|-
|-
| Updates Testing (see below)
| Updates Testing (see below)
| Tchung - Betatest '''PASS''', Releasetest '''PASS''', Beta '''PASS'''
| Tchung - Betatest '''result''', Releasetest '''result''', Beta '''result'''
| Juanb - Betatest '''PASS''', Releasetest '''PASS''', Beta '''PASS'''
| Juanb - Betatest '''result''', Releasetest '''result''', Beta '''result'''
| Stephend - Betatest '''PASS''', Releasetest PASS, Beta '''PASS'''
| Stephend - Betatest '''result''', Releasetest result, Beta '''result'''
| Marcia - Betatest '''PASS''', Releasetest '''PASS''', Beta '''PASS'''
| Marcia - Betatest '''result''', Releasetest '''result''', Beta '''result'''
| Abillings - Betatest '''PASS''', Releasetest '''PASS''', Beta '''PASS'''
| Abillings - Betatest '''result''', Releasetest '''result''', Beta '''result'''
|-
|-
| l10N Pri 1 locales
| l10N Pri 1 locales
| Tchung - '''PASS'''   
| Tchung - '''result'''   
| N/A  
| N/A  
| Tchung - '''PASS'''
| Tchung - '''result'''
| N/A
| N/A
| Tchung - '''PASS'''
| Tchung - '''result'''
|-
|-
| Bug Verifications
| Bug Verifications
Line 108: Line 101:
| Everyone - '''result'''
| Everyone - '''result'''
|}
|}
=== Top Plugins Testing ===
Query the top plugins supported for the web, and verify they are compatible with Firefox 3.  See [https://addons.mozilla.org/en-US/firefox/browse/type:7 AMO list] for recommended plugins. 
Testcases
* On a clean environment, install plugins
* Test updating of old plugins
* Disable plugins in Addons Manager, verify plugins stop working
* Uninstall plugins, verify websites request a plugin to download
* Verify websites with popular plugins:
** www.nintendo8.com (java)
*** Linux - '''PASS'''
** www.youtube.com (flash)
*** Linux - '''PASS'''
** www.shockwave.com (shockwave)
*** Linux - '''PASS'''
** www.apple.com/trailers (quicktime)
*** Linux - '''PASS'''
**** On Linux, use [http://linuxappfinder.com/package/mozilla-plugin-vlc vlc plugin]. A test clip is [http://www.zeroimpactproductions.com/newfiles/stocksamplebroadbandQT.html available].
** www.msnbc.com, www.cnn.com (WMV, RealVideo)
*** Linux - '''FAIL'''
**** RealPlayer 10 crashes under Firefox 3 - [https://bugzilla.mozilla.org/show_bug.cgi?id=421381 Bug 421381]
**** On Linux, use [http://linuxappfinder.com/package/mozilla-plugin-vlc vlc plugin] for WMV. Browser detection will often block it though.
** www.earthwatch.org (Adobe PDF)
*** Linux - '''PASS'''
** [http://www.viewpoint.com/technologies/viewpoint-media-player.shtml Viewpoint Media Player] plugin
*** Linux - '''N/A''' (not supported)
***Mac OS X:
**** Filed [https://bugzilla.mozilla.org/show_bug.cgi?id=421026 bug 421026] on the Viewpoint plugin for Mac, and requested blocking for Firefox 3
*** Windows:
**** Filed [https://bugzilla.mozilla.org/show_bug.cgi?id=421030 bug 421030] for an nsIView::GetView crasher when reloading a demo page using Viewpoint
** Others like divx, virtual tour, online ads, etc..
*** '''PASS'''
Additional Plugin litmus cases can be found [https://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=15&testgroup_id=56&subgroup_id=801 here].


=== Updates Testing ===
=== Updates Testing ===
For b5, we will be doing updates of all Alpha and Beta users to Beta 5. Tests will include different locales for beta 1, but only en-US for all past versions.  We will be checking partial updates, complete updates, and failed status.   
We will be doing updates of all Alpha and Beta users to Beta 5. Tests will include different locales for beta 1, but only en-US for all past versions.  We will be checking partial updates, complete updates, and failed status.   


Will be available on ''betatest'', ''releasetest'', and ''beta'' channels
Will be available on ''betatest'', ''releasetest'', and ''beta'' channels


* Betatest channel - '''PASS'''
* Betatest channel - '''result'''
* Releasetest channel - '''PASS'''
* Releasetest channel - '''result'''
* Beta channel (live) - '''PASS'''
* Beta channel (live) - '''result'''


Testcases and Results are tracked [http://wiki.mozilla.org/QA/Firefox3/TestResults/Beta4/Updates here].
Testcases and Results are tracked [http://wiki.mozilla.org/QA/Firefox3/TestResults/Beta4/Updates here].
Line 164: Line 121:


P1 Locales to be tested:
P1 Locales to be tested:
* German (de) '''PASS'''
* German (de) '''result'''
* Japanese (ja) '''PASS'''
* Japanese (ja) '''result'''
* Spanish (es-ES) '''PASS'''
* Spanish (es-ES) '''result'''
* Netherlands (nl) '''PASS'''  
* Netherlands (nl) '''result'''  
* Italian (it) '''PASS - Opened bug 420988'''  
* Italian (it) '''result'''  
* Finnish (fi) '''PASS'''  
* Finnish (fi) '''result'''  
* Hindi (pa-IN) '''PASS'''
* Hindi (pa-IN) '''result'''
* Chinese (zh-CN) '''PASS - Opened bug 421213, 421219'''  
* Chinese (zh-CN) '''result'''  
* French (fr) '''PASS'''
* French (fr) '''result'''  
 
Testcases and Results are tracked [http://wiki.mozilla.org/QA/Firefox3/TestResults/Beta4/L10N here] - '''PASS'''.
 
=== Special Respin Tests ===
A build issue was found late in the game, and required a respin to all the Windows builds.  This was due to a late discovery that [http://gemal.dk/blog/2008/03/04/firefox_3_with_profileguided_optimization_speeding_ticket/ PGO] or Profile Guided Optimization, was late to be included into the build bits. 


A special testplan was created to verify these bits.  See Testplan [http://wiki.mozilla.org/QA/Firefox3/TestResults/Beta4/RespinTest here].
Testcases and Results are tracked [http://wiki.mozilla.org/QA/Firefox3/TestResults/Beta4/L10N here] - '''result'''.


=== Bug Verifications ===
=== Bug Verifications ===
Bugs that are marked RESOLVED FIXED that are checked in for beta.  Please verify them.
Bugs that are marked RESOLVED FIXED that are checked in for beta.  Please verify them.


B5 Firefox bugs - http://tinyurl.com/3y3u4u <BR>
B5 Firefox bugs - <BR>
B5 Platform bugs - http://tinyurl.com/2mqmvx
B5 Platform bugs -  


=== Testday ===
=== Testday ===
A beta 5 testday will be held and call for the community support to assist with testing.  Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented.  More can be found on [http://quality.mozilla.org/events/test-days QMO].
A beta 5 testday will be held and call for the community support to assist with testing.  Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented.  More can be found on [http://quality.mozilla.org/events/test-days QMO].


Testday results are posted [http://wiki.mozilla.org/QA/Firefox3/TestResults/Beta4/Testday here].
Testday results are posted [http://wiki.mozilla.org/QA/Firefox3/TestResults/Beta5/Testday here].


== <FONT COLOR = "RED">Detailed Results </FONT> ==
== <FONT COLOR = "RED">Detailed Results </FONT> ==

Revision as of 21:40, 13 March 2008

« QA/Firefox3/TestResults

Firefox 3 Beta 5 Test Results

Related Pages

Links to separate test results. Some are referenced further below.

Release Tracking Page

Contact Info

PM: Mike Beltzner (beltzner)
Build: John O'Duinn (joduinn)
QA: Tony (tchung)

Schedule

Code Freeze
3/18
Code Build
ETA 3/25
QA test signoff
ETA 3/24-3/27
  • Smoketests (3/24) result (3 testers)
  • BFTs (3/24-3/25) result (4 testers)
  • Beta 5 and past focused Tests (ongoing)
  • Update snippets from Alpha X, Beta 1,2 -> Beta 3 (3/25-3/26) result betatest, result releasetest, result beta (4 testers)
  • P1 L10N spotchecks (3/26) result (1 tester)
  • B5 bug verifications (ongoing)
  • Beta 5 Testday (3/27) result (3 testers)
  • Release notes, Websites, live tests (3/27) result (3 testers)

Release Notes

See list of release notes here

Build Location and IDs

Strategy

  • Areas Covered:
    • Smoketests
    • BFTs
    • Beta4 and past focused Tests
    • Top Plugins
    • Update snippets from Alpha X, Beta X -> Beta 5
    • Memory Leak Testing (handled by Tomcat on debug builds)
    • P1 L10N spotchecks
    • Bug verifications
    • Beta 5 Testday
  • Areas Not Covered:
    • localization (String changes) - 3rd party will cover
    • Distribution
    • Major Updates

Test Results

Types of Tests Win XP Win Vista Mac Tiger Mac Leopard Linux
Smoketest Tomcat - result juanb - result marcia - result marcia - result Tomcat - result
BFT Tomcat - result Tomcat (Vista test suite) - result juanb - result Marcia (Leopard test suite/PPC and Intel) - result Tracy - result
Updates Testing (see below) Tchung - Betatest result, Releasetest result, Beta result Juanb - Betatest result, Releasetest result, Beta result Stephend - Betatest result, Releasetest result, Beta result Marcia - Betatest result, Releasetest result, Beta result Abillings - Betatest result, Releasetest result, Beta result
l10N Pri 1 locales Tchung - result N/A Tchung - result N/A Tchung - result
Bug Verifications Everyone - result Everyone - result Everyone - result Everyone - result Everyone - result

Updates Testing

We will be doing updates of all Alpha and Beta users to Beta 5. Tests will include different locales for beta 1, but only en-US for all past versions. We will be checking partial updates, complete updates, and failed status.

Will be available on betatest, releasetest, and beta channels

  • Betatest channel - result
  • Releasetest channel - result
  • Beta channel (live) - result

Testcases and Results are tracked here.

L10N P1 Spotchecks

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Automation inspection: supported locales will be ran through Minotaur, and results will be reviewed by axel and mic.

Axel's Localization Status Page

P1 Locales to be tested:

  • German (de) result
  • Japanese (ja) result
  • Spanish (es-ES) result
  • Netherlands (nl) result
  • Italian (it) result
  • Finnish (fi) result
  • Hindi (pa-IN) result
  • Chinese (zh-CN) result
  • French (fr) result

Testcases and Results are tracked here - result.

Bug Verifications

Bugs that are marked RESOLVED FIXED that are checked in for beta. Please verify them.

B5 Firefox bugs -
B5 Platform bugs -

Testday

A beta 5 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found on QMO.

Testday results are posted here.

Detailed Results

Enter results from testing here.

Live Spot Checks

Live site downloads for Beta4, after bits are pushed out. Please cover en-US, and 2 or 3 additional Tier 1 locales as well. Pay attention to the following areas:

  • Check for signed installer and applications (windows only)
  • Firstrun page, whatsnew page, Home page, Search plugin
  • Installer, uninstall
  • General spotchecks and browser functionality

Windows XP -> abillings - PASS
Windows Vista -> stephend - PASS
Mac Tiger -> Juan - PASS

Mac Leopard -> Marcia - see results below

  • he, Hebrew - fails on Mac with UI problems. Sent screenshots to tchung
  • fr, Frisian - PASS
  • Identifikues montimi: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; sq; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4 - PASS

Linux -> stephend - PASS

Top Issues Reported in Hendrix Against Firefox 3 Beta 5

  • Enter here

Top Issues in b4 in Reporter

  • Enter here
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox3/TestResults/Beta5&oldid=86632"