QA/Firefox3.5/TestPlan/RC1

« QA/Firefox3.5/TestPlan

Firefox 3.5 RC1 Test Results

The covered testing for this release will focus on smoketests, bfts, l10n, top plugins, top financial sites, updates, and spotchecks.

Related Pages

Firefox 3.5 Tracking Site

• Release Schedule
• RC1 features

Project contacts

PM: Mike Beltzner (beltzner)
Build: Nick Thomas (nthomas)
QA: Juanb

Schedule

Code Freeze

5/21

Builds hand to QA

5/25

QA test signoff

6/5?

• See Test Coverage below

Build Location and IDs

Download:

Builds:

Test Coverage

• Areas Covered:
  • Smoketests
  • BFTs
  • FFTs (will carry over into nightly testing, but not dependent on rc1 ship date)
  • L10n
  • Topsites (covered by Tomcat's automation)
  • Top Financial Sites (covered by website project)
  • Top Extensions & Themes
  • Top Plugins
  • Accessibility
  • Updates
  • Spotchecks & Webpages
  • Bug Fixes

• Areas Not Covered:

Test Results

Types of Tests	Windows XP	Mac OSX	Linux
Smoketests	tracy - result	henrik - result	ashughes - result
BFTs	tracy/aakashd - result	henrik/tchung - result	ashughes/juanb - result
FFTs (run them after RC1 ships)	juanb/tracy - result	henrik/tester - result	tester/tester - result
L10n	tracy - result	tchung - result	ashughes - result
Top Extensions & Themes	tracy - result	Murali/Aaron - result	ashughes - result
Top Plugins	tracy - result	tchung/Aaron - result	ashughes - result
Accessibility	MarcoZ - result	N/A	MarcoZ - result
Updates	tracy Betatest - result releasetest - result Beta - result	Murali Betatest - result releasetest - result Beta - result	ashughes Betatest - result releasetest - result Beta - result
Additional Focused Testing	QA team - result	QA team - result	QA team - result
Topsite Testing (automation)	tomcat - result	tomcat - result	tomcat - result
Staging/Live Webpages & Spotchecks & Webpages	tester - result	henrik - result	ashughes - result

Smoketest Testing

Lite verification tests (or smoketests) will be first ran when the builds are ready. Tests are available on litmus under the Firefox 3.5 test suite. In any case the Smoketests result, we will raise blockers asap before proceeding with the bfts.

Please record your detailed results in the section below.

BFT / FFT Testing

BFTs are a more comprehensive set of litmus tests that cover the build testing after smoketests result. Tests are available on litmus under the Firefox 3.5 test suite.

FFTs are a full comprehensive set of litmus tests that extend coverage in areas that BFTs dont often get to. These are run at least once during the major milestone timeline.

If you are running a BFT or FFT, Please record your detailed results in the section below.

Updates Testing

We will be doing updates of all Alpha/Beta users to RC1. We will be checking complete updates, and failover scenarios.

Will be available on betatest, releasetest, and beta channels.

Testcases and Results are tracked here.

L10N

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to Fx3.1 L10N Requirements for more specifics.

Testcases and Results are tracked here.

Top Extensions & Themes Testing

Select a few top extensions from AMO and perform the following testcases. See Fx3 Compatibility Addons List Here.

Also, this is the top 50 most popular addons voted by mozdev.

Testcases:

• Enabling extensions/themes and confirming web compatibility
• Disabling or Uninstalling extensions/themes and confirming proper messaging and behavior
• Load about 10+ extensions and themes, and fire them up all at once, verifying functionality and performance
• Updating old extension and themes versions to the latest if applicable
• Switch profiles with different extensions and themes, and confirm they do not get carried over.

Testcases and Results are tracked here.

Top Plugins Testing

Select a few top plugins from AMO and perform the following testcases. (examples: Flash, Acrobat, Java, Real Player)

Testcases:

• Enabling extensions and confirming web compatibility
• Disabling or Uninstalling plugins and confirming proper messaging and behavior
• Switch profiles and verify plugins are still working properly

Testcases and Results are tracked here.

Top Site Testing

• Tests on all platforms result without a crash on the top-500 Sites
• Started a more complex run with more pages

Staging/Live Spotchecks & Webpages

1) Verify the following webpages that gets pushed to staging and then to the live sites. Look for content messaging, spelling mistakes, broken links & imnages, etc..

• Firstrun page (Staging link)
• Whatsnew Page (Staging link)
• Releasenotes page (Staging link)

2) Check live site download links on all platforms. Please cover en-US, and 2 or 3 additional Tier 1 locales as well. Pay attention to the following areas:

• Check for signed installer and applications (windows only)
• Firstrun page, whatsnew page, Home page, Search plugin
• Install, uninstall
• General spotchecks and browser functionality
• Confirm Build IDs

Additional Resource Testing

Given the minimized focus testing around RC1 due to time constraints, there are a few regression areas that will need some focus testing. This area will call for additional resources to help QA in this effort to ship RC1 in a timely manner.

• bug 488015, bug 452498 - Test around sites and content that this javascript fix may touch
• bug 475864 - Test around video controls and file input handling
• bug 488497 - Test around Geolocation work
• bug 488015, bug 470487 - Test around crash fixes that need better repro steps
• bug 475066 - Test around tearing away tabs
• bug 486236 - Test around Ts performance
• bug 394759 - Test around Session Restore / undo close window feature
• bug 480169 - Test around new Clear Recent History (sprint) feature
• bug 422526 - Test around localstorage (web developers)
  • Others: bug 485396, bug 484180
• bug 484180 - Test around Session Storage Update: May be backed out for 3.5
• Fixed P1,P2 1.9.1 bugs from Beta4 (April 27th) to present, that need to be verified

Fixed Bugs Verifications

The following bugs are fixed for RC1 and need verifications:

• fixed1.9.1 keyword bugs: http://tinyurl.com/8guodt
• fixed1.9.1 keyword bugs from Beta4 to now: http://tinyurl.com/qq3uql

Testday

A RC1 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found on QMO.

Willing to help? Sign Up Here

• nickname <email (optional)> <details>
• ctalbert <ctalbert at mozilla dot com> Can help with any back end coding stuff
• kbrosnan <kbrosnan at gmail dot com> Help with anything except OS X
• juanb <juanb at mozilla dot com> Smoketest, BFT, FFT, and misc., on any platform.

Detailed Results

Testday Results

• RC1 Testday scheduled may 29th