Firefox:1.5.0.2:Test Plan

Firefox 1.5.0.2 Test Plan

Bug Fix Verification

• Identify list of bugs
  • Triage fixed bugs and categorize
    • ready for testing
    • test case needed
      • request more info from interested parties
    • no verification necessary
      • usually internal architecture changes with no externally visible effects
• Set threshold for which bug fixes must be verified
  • 60% of the Firefox specific bugs (Example: not seamonkey, non-core platform, or Thunderbird specific fixes)
  • All critical security relevant bugs
• Verify security fixes and some non-security fixes in house
• Get community to help verify fixes for non-security bugs
• Get members of target market to help verify strategic marketing changes
  • IBM for enterprise deployment functionality
  • Mozilla Japan for jp-critical issues

Regression Testing

• Smoke Tests
• BFT on all three platforms
• Java Script
• Top Sites (See FFT)
• Key extensions
• Performance
  • Gross memory usage
• Security Test Library tests

Install Testing

• Test en-US after initial packaging.
• Test locales after locale packaing.

L10n Testing

• More in depth testing of 5 locals using Litmus L10n tests (AKA Sara's checklist)
• basic install/version id checks
• L10n test matrix (add link here!)
• Axel's external L10n status and owner info

Extension Testing

• Top 10 AMO extensions according to download numbers.
• Recommended sites
• http://developer.mozilla.org/contests/extendfirefox/

Update Testing

This is the testing of the partial and full update .mar files for the various locales once the main downloads are released.

• Firefox 1.5.0.1 will use the following urls to determine if there is an update available:
  • Mac
  • Win
  • Linux

• Replace "release" in the url with the channel name to test other channels

Community Testing

• Internal Community Test Day page
• External Community Test Day page

See internal and external results in each of these pages

Location of Downloads

• Nightlies

Windows: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-aviary1.0.1/firefox-1.0.8.en-US.win32.installer.exe MacOS: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-aviary1.0.1/firefox-1.0.8.en-US.mac.dmg Linux: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-aviary1.0.1/firefox-1.0.8.en-US.linux-i686.installer.tar.gz

• Location of RC builds:

TBD

Configuration Testing

We need to describe our configuration testing approach better. For now, we are generally testing these OS's:

• Windows XP
• MacOS 10.3, 10.4
• Linux Ferdora Core 4

But we need to consider our system requirements statements. I have seen at least three versions. Here is one:

mozilla.com->Products Tab -> Firefox -> System Requirements link

Mac OS X Universal Builds

• Done:
  • Verification of bugs on ppc-only build [done]
  • Smoke test on ppc-only build [done]
  • Smoke test of universal build on ppc 10.2, 10.3, and 10.4
  • Smoke test of universal build running native x86 [done]
  • Smoke test of universal build running ppc-emulated on x86 mac [done]
  • focused testing of universal build issues
  • Flash on 10.4.6 works native (as version 8,0,24,0)
  • Flash on 10.4.6 works under rosetta (as version 8,0,27,0)
  • Flash on 10.4.4 (clean install) works native (as version 8,0,22,0)
  • Flash on 10.4.4 (clean install) works under rosetta (as version 8,0,22,0)

• TODO:
  • bft on universal build on x86 (native and emulated)
    • Can do now: 2 x 4hrs
    • anyone with x86 mac
      • Marcia: native: PASS
      • Timr/marcia: under Rosetta: in progress - marcia wrapping up (7/11/06)
  • NEW: check talkback symbols when UB crashes on ppc mac
    • Can do now: 10 minutes - PASS - Davel 7/11/06
  • NEW: BFT of UB on ppc mac running 10.2 and 10.4.x
    • Can do now: have PPC Mac with these OS level in the QA lab. ~ 2 x 4 hrs
    • Who: marcia, tim, jay, davel?
      • Jay: 10.2 PASS
      • tracy: 10.4.x PASS
  • NEW: validate l10n of UB (new platform issues)
    • Need build. Builds can be fragile. 1-3 days depending on if issues come up
    • Testing: davel can use a diff tool. Then 4-8 hrs depending on what diffs say. Can be split up by 1-4 people
  • NEW: validate software update of UB (new platorm issues)
    • partial mar applies correctly
      • Need to builds mars. This process can be fragile. 1-3 days depending on issues
      • testing: davel- can tool help here?
    • aus serves correct pointers to updates
      • Easy to do once partials area available
      • Who: mark, josh?
    • spot check of builds and update paths
      • When updates are available.
      • Who: 1-4 people running through the rvarious update paths depending on what automated testing can check
      • 4-8 hrs depending on if tool available

• • Do we need an RC? If this is the primary download then we better be happy with it. IWC no RC needed.

Overall I think this adds up to 2-6 days depending on if anything goes wrong and how focused Build and QA can be. Two days could happen but that is probably not realistic.

Note: NEW = new as of the UB meeting on 2006.04.06

Test Results

• Gross Memory Usage
• Basic Functional Tests
• Extension Testing
• Update Testing
• Bug Fix Verification
• Security
• Regression Testing
• Community test day results
• l10N testing
• l10N update testing