Firefox:2.0 QA Activities

Test Plan

Strategy

• Focus on new feeatures and develop automated tests or manual tests in Litmus
• Ramp up from smoke tests (old regression tests and new functional tests) to FFTs (old regression and new functional) to peak with the FFTs during the Beta 2 and RC1 test cycles. Then switch back to BFTs on Windows and smoke tests for RC2, RC3 and any respins.

Milestone by Milestone plans:

Milestone	Smoke test	BFT	FFT	Focused Testing
Alpha3	XXX			XXX
Beta 1		XXX		XXX
Beta 2			XXX	XXX
RC 1		XXX		XXX
RC 2		XXX		XXX
RC 3	XXX			XXX
Respins	XXX			XXX

Old Feature Regression testing

• This testing is accomplished with the old (FFx 1.5) Smoge tests, BFTs and FFTs.
  • See Beta2 FFT plans and results
• We count on the unit tests run from tinderbox after the builds for a great deal of the regresion testing along
• We also count on Community testing. Bake time on the trunk is important along with our nightly testers and the bi-weekly commnity test days.
• We run Old BFT weekly
  • BFT Schedule and Results
  • We plan to do this until FF2 stabalizes
    • Primary platform; weekly: Win
    • Secondary platforms; alternate Intel Mac with Linux, PPC Mac every other week.
• We run JS Test Library
• We run Security Test Library

New Feature Functional Testing

• New Feature Test Plans
• Run new feature tests as code in integrated and at milestones. Generally BFT level functional tests will be ready for Beta 1. FFT level tests will be ready for Beta 2. But this is dependent on when code is ready and info is available to QA. User based tests go into Litmus. Programmatic test are checked into CVS.

L10n Testing

General strategy:

• Verify trademark issues with info from product management
• Thorough translation, context and layout testing with a 3rd party service for top locales and supplement with simplistic spot checks in house as needed for last minute checks, etc.
• Test special regional distributions.

Overall Status of Locales

Detailed L10n Test Plan

Software Update Testing

General strategy:

• Run update script to identify issues
• Test the following matrix to identify any potential issues
• Run the suite of Software Update tests in Litmus

	Windows	Mac	Linux
Alpha1	en-US	en-US Fallback
Alpha2		en-US fallback
Alpha3			en-US
Beta1 (digg)	One Tier 1 locale
Beta1 (official)		One Tier 1 locale	One Tier 1 locale fallback

Results

Extended Extension testing

There are some extension tests in the BFTs and FFTs. The extended extension testing goes beyond that to test 5 major classes of extensions:

• Recently Updated (10)
• 10 Top extensions (based on downloads)(10)
• Recommended Extensions (12)
• Newest (lots)
• Top Rated (based on community ratings) (Lots)

The Criteria here is to test all the Recently added extensions, Top Extensions, Recommended Extensions. Then also test roughly the top 10 each of the Newest and Top Rated Extensions. This is a lot! This is a total of 52 extensions. We need community help with this!

The testing of this is not very detailed. The plan is to try out ~3 main features to see if the extension can be installed, used in it basic modes of operation, and uninstalls cleanly.

Anyone helping should either add their results info about what you tested (your name, feature tested, build id, Version string, pass/fail status, and notes) to the

• Result Page

or email results to qa@mozilla.org.

Profile Testing

• Migration from Firefox 1.0, Firefox 1.5.0.x Profiles
  • Need to create a set of profiles and have them available for download
• Profiles with History
  • Engage community help with this?
• Profiles with the Top 10 AMO Extensions installed
• Profiles with recommended extensions installed

Detailed Profile Test Plan

Organized Community Testing

• Biweekly community test days
• Weekly community bug days

Special Distributions

• Y! CJK
• Google CJK
• PCHome
• Seznam

Meeting Notes

• Firefox:2.0 QA Activities:PlanningMtg2006.02.06

Beta1 Feature Testing Update

Please list if feature is complete for Bon Echo beta. If not, please reference significant bugs that will not be fixed for beta1 or functionality that will not be present

• Session Restore
  • This feature is in. Most testing is done. BC will complete his test plan by mid day Tuesday (7/11).

• Tabbed Browsing
  • Adam needs a few more hours to finish BFTs. This should be done by COB today (7/10)

• RSS Feed Handling

• Anti-Phishing

• Visual Refresh
  • Make sure that Mac popup dialog styling bug is addressed when visual refresh is done - bug 340710

• Search Engine UI

• In-line Spell Check
  • Not ready yet. brettw@google will let us know.

• Microsummaries

• Preferences Panel
  • Areas of Focus
    • Make sure new profiles and all 1.5.0.x profiles work, as well as backwards compatible profiles.
    • See bug 340677 comment 37 for specifics.
  • This is *not* in for B1. Should be there for B2.

Milestone Tasks

Beta 1 Tasks

• Beta 1 Tasks Archive

Beta 2 Tasks

• Beta 2 Tasks Archive

• Tentative code freeze on 2006-08-01
• FFT Schedule and Results

Verifications

mconnor has asked that we verify fixed crash bugs and regressions in Firefox 2.

When you verify the bug is fixed, resolve it as VERIFIED FIXED and replace the fixed1.8.1 keyword with verified1.8.1.

• Unverified fixed1.8.1 crasher bugs that are blocking-firefox2/blocking1.8.1
• Unverified fixed1.8.1 regressions that are blocking-firefox2/blocking1.8.1

Tips:

• Some bugs have been verified on 1.8.0 bug still need to be verified on 1.8.
• If you don't see an easy way to verify that a bug is fixed comment in the bug or add "[needs testcase]" (without quotes) to the bug's Status Whiteboard.

Release Activities

L10n Handoff to Build

• Axel attaches his recommended shipped-locales file to bug 344295 and requests a review of the attachment
• QA spot checks those locales
• Tim r+'s the attachment or adds an updated version based on the testing and r+'s it.
• Build (Preed) sees the approved review, checks in the shipped-locales file and builds/packaged the final list of locales based on that.

Results

Beta2 RC1 Test Results

• Archive of Beta 2 Results

RC1 Test Results

Regression Testing

• Smoke tests
  • Win - Tomcat - PASS
    • Specific OS Version: Win2K
    • Version ID: Mozilla/5.0 (Windows; U; Windows NT 5.0;

en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0

• • • Talkback Build ID 2006091818
    • Notes:
  • Mac - Marcia - PASS
    • Specific OS Version: Intel Mac
    • Version ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0
    • Talkback Build ID: 2006091817
    • Notes:
  • Linux - Tracy - PASS
    • Specific OS Version: FC5
    • Version ID: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0
    • Talkback Build ID: 2006091817
    • Litmus results
    • Notes:

• BFT - Win is minimum
  • Tomcat PASS
    • Specific OS Version: Windows 2000/XP
    • Version ID: Mozilla/5.0 (Windows; U; Windows NT 5.0;

en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0

• • • Talkback Build ID 20060918
    • Notes: Unclear about bring me back "anti phishing" feature

• • Marcia
    • Specific OS Version: PPC Mac
    • Version ID: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0
    • Talkback Build ID: 2006091817
    • Litmus Link to results: http://tinyurl.com/mkmz5
    • Notes:
      • Did not complete a full BFT, but tested the following areas:
        • Installation, Options, prefs, General, Location Bar, RSS, Toolbar, History, PW and Form Manager, Anti-Phishing and Technical Tools.
      • No crashes observed during testing.
    • Issues:
      • Anti Phishing testcase "Get me out of here" failed, but it turned out that the behavior had changed. I fixed the test case to reflect the current behavior.
    • Bugs filed:
      • https://bugzilla.mozilla.org/show_bug.cgi?id=353419 - Can't close 132 tabs
      • https://bugzilla.mozilla.org/show_bug.cgi?id=353368 - Tab Reflow could be smoother

• Security Test Library - Bob

• JS Test Suite - Bob

Focused testing

Suggestions from mconnor:

• Themes
  • This is overall area is well tested by community
  • Hover state
  • Click state on home button in help viewer
• RSS
  • Feed preview
• Preferences Dialog
  • Lot of late changes in this area. Test all the buttons, tabs, controls thoroughly
• Accessibility
  • General concern

• Tabbed Browsing
  • Throw tons of tests at the tab strip
  • There should be a warning if > 15 tabs are about to be opened at once

• JS
  • Page compatibility

• Overall
  • General testing
  • web compatibility testing testing

dbaron and brendan were also consulted for focus areas to test.

Test Results

Template:

 * Tester name:
 * Platform:
 * version string:
 * Link to litmus FFT results:
 * General description (if testing was without litmus):

• marcia - focused testing on RSS and Tabbed Browsing

• RSS Testing: Ran set of test in Full Functionals on 9-20-06
  • Linux - Marcia -
    • Specific OS Version: Fedora Core 5
    • Version ID: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0
    • Talkback Build ID: 2006091817
    • Issues Noted:
      • https://bugzilla.mozilla.org/show_bug.cgi?id=353580 - Feed Reader Preview does not render correctly
      • Had issues with finding a client side feed reader with Linux. More investigation needed.

• RSS Testing: Ran set of test in Full Functionals on 9-21-06
  • Mac - Marcia -
    • Specific OS Version: Intel Mac running 10.4.7
    • Version ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0
    • Talkback Build ID: 2006091817
    • Issues Noted:
      • Added a new test case for Feed Reader First Run UI.

• Preferences Testing - Ran Full Functionals on 9-21-06
  • Mac - Marcia -
    • Specific OS Version: Intel Mac running 10.4.7
    • Version ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0
    • Talkback Build ID: 2006091817
    • Issues Noted:

• Marcia's Bugs Filed During Testing
  • https://bugzilla.mozilla.org/show_bug.cgi?id=353419
  • https://bugzilla.mozilla.org/show_bug.cgi?id=353580
  • https://bugzilla.mozilla.org/show_bug.cgi?id=353694
  • https://bugzilla.mozilla.org/show_bug.cgi?id=353726

• Tomcat - Running FFT Parts for Tabbed Browsing and RSS
  • Windows 2000 SP4 /XP Sp2
    • version string: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0
    • Talkback Build ID: 2006091818
    • Notes
      • Failed to open Bookmark and History Element on Background, will re-test later with my Testarray

• Search Plugins
  • Tester name: marcia
  • Platform: Intel Mac
  • version string: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0
  • Link to litmus FFT results:
  • General description (if testing was without litmus): Tested installing and uninstalling various plugins. I tested plugins from addons page as well as mycroft.
    • https://bugzilla.mozilla.org/show_bug.cgi?id=353694
    • Backward compatibility issues noted and deserve further investigation at some point

• Themes
  • Tester name: timr
  • Platform: Windows/Mac OS

• Accessibility
  • Tester name: jay
  • Platform: MacOS

• Add ons - Run FFT of Add-ons tests in Litmus
  • Tester name: marcia
  • Platform: WinXP
  • version string: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0
  • Link to litmus FFT results: in Litmus
  • General description (if testing was without litmus):

• Install Extensions in 1.5 and then update to 2.0 (testing compatibility check)
  • Tester name: marcia
  • Platform: WinXP
  • version string:Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0
  • Link to litmus FFT results:
  • General description (if testing was without litmus): I tested installing a mix of extensions in a 1.5 Profile from both add-ons and extension mirror that were both compatible and incompatible with FFRC1. I then moved to FF 2 to make sure the compatibility check was done properly. PASS.

Extended Extension Testing

Tested by Tomcat on 23.September 2006 for RC 1

• Results for RC1

Extended Top Site Testing

Tested by Tomcat on 23.September 2006 for RC1

• Results for RC1

L10n Testing

See test matrix

RC1 Software Update Testing

General strategy:

• Build Runs update script to identify issues before QA begins testing
• Test the following matrix to identify any potential issues
• Strong emphasis on purposely failing updates to make sure we are handling failures correctly. The script will not pick up errors in this area.
• Run the suite of Software Update tests in Litmus

	Windows	Mac	Linux
Alpha1	en-US	en-US Fallback
Alpha2		en-US fallback
Alpha3			en-US
Beta1		One Tier 1 locale	One Tier 1 locale fallback
Beta2		One Tier 1 locale	One Tier 1 locale fallback

• RC1 Software Update Results

Live Testing

• Spot checks of Software Update and all.html

RC1 Live Software Update Results

RC2 Testing

Path to builds:

  http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2.0rc2-candidates/rc1/

Smoke tests

Template

• <platform> - <tester name> - <Status PASS/FAIL>
  • Specific OS Version:
  • Version string:
  • Talkback Build ID:
  • Notes:

Results

• • Win - Tomcat - PASS
    • Specific OS Version: ???
    • Version ID: ???
    • Talkback Build ID ???
    • Notes:

• • Mac PPC - Marcia - TBD
    • Specific OS Version: 10.3
    • Version string:
    • Talkback Build ID:
    • Notes:

• • Mac Intel - timr - TBD
    • Specific OS Version: 10.3
    • Version string:
    • Talkback Build ID:
    • Notes:

• • Linux - juan - TBD
    • Specific OS Version:
    • Version string:
    • Talkback Build ID:
    • Notes:

• • Linux - Aleksej - PASS (but see notes)
    • Specific OS Version: Debian GNU/Linux testing
    • Version string: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061002 Firefox/2.0
    • Talkback Build ID: 2006100202
    • Notes: 1221: Performance check -- a spike today or yesterday. Didn't test: standard installation, [mac only] confirm xforms not part of build, basic printing.