Firefox:1.5.0.8:Test Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
= General Firefox 1.5.0.x Test Plan =
== [[Firefox:1.5.0.x:Test_Plan | General Firefox 1.5.0.x Test Plan]] ==
'''NOTES: This will be updated shortly, for 1.5.0.8'''
This is the basic set of tests we run for every security & maintenance release.
 
 
== Assumptions ==
 
* Goal of this release is security fixes and important regressions, other fixes will not be included
* Builds should have gone through the l10n meta-diff checker and the update checker before they are released to QA
* Thunderbird will be released a few days after Firefox
* Monday, September 4th is a holiday
 
== Current Tasks ==
 
* Bug triage
* Test plan completion
* Bug verification
 
== Schedule ==
 
=== Monday, August 28th ===
* RC1 delayed till Monday night/Tuesday morning.  Shifting Monday's planned activities to Tuesday
 
=== Tuesday, August 29th ===
* Firefox builds should be available for Win/Linux/PPC
* Smoketests (rc2) on Windows, Linux, Mac en-us
** [[Firefox:1.5.0.7:Smoketests|1.5.0.7 Firefox and Thunderbird Smoketest results]]
 
=== Wednesday, August 30th ===
* Firefox BFTs - PASS
** [[Firefox:1.5.0.7:BFTs|1.5.0.7 Firefox and Thunderbird BFT results]]
* Build should begin l10n packaging after BFTs signoff (Rhelmer/TR)
* Get post up on the QA blog with link to candidate builds
* Testing of QA identified hot spots
* Targeted exploratory testing over Developer identified hot spots
 
=== Thursday, August 31st ===
* Continued testing of developer identified hot spots and bug verifications
 
=== Friday, September 1st ===
* Community Testday for 1.5.0.7 RC builds
* Smoketests (rc2) on Windows, Linux, Mac en-us
** [[Firefox:1.5.0.7:Smoketests|1.5.0.7 Firefox and Thunderbird Smoketest results]]
* Thunderbird BFTs - PASS
** [[Firefox:1.5.0.7:BFTs|1.5.0.7 Firefox and Thunderbird BFT results]]
* Bug verifications and focus testing
 
=== Tuesday, September 5th ===
* Further bug verifications and testing of hot spots
* Thunderbird Smoketests
* Start Thunderbird BFTs
 
=== Wednesday, September 6th ===
* Firefox RC4 will be released this afternoon
* Smoketest Firefox RC4
** [[Firefox:1.5.0.7:Smoketests|1.5.0.7 Firefox Smoketest results]]
* Thunderbird bug verifications
 
=== Thursday, September 7th ===
* Respins for rc5 out late Sept. 6
* Smoketest rc5 builds
** [[Firefox:1.5.0.7:Smoketests|1.5.0.7 Firefox Smoketest results]]
* Firefox security testing for RSA/Certificate changes
** https://bugzilla.mozilla.org/show_bug.cgi?id=340198
** https://bugzilla.mozilla.org/show_bug.cgi?id=351079
** https://bugzilla.mozilla.org/show_bug.cgi?id=350640
* Thunderbird bug verifications
 
=== Friday, September 8th ===
* Spot check P1 locales
** [[Firefox:1.5.0.7:LocalizationTesting|1.5.0.7 Firefox and Thunderbird Localization testing results]]
** Check other tier 2 locales as time permits
**[[Firefox:1.5.0.7:Extension Testing|1.5.0.7 Firefox and Thunderbird Extension Testing results]]
 
=== Monday, September 11th ===
* Smoketest rc6 builds
** [[Firefox:1.5.0.7:Smoketests#RC6_builds_-_tested_09.2F10|1.5.0.7 Firefox Smoketest results]]
* Spot check Thunderbird P1 locales
** [[Firefox:1.5.0.7:LocalizationTesting|1.5.0.7 Firefox and Thunderbird Localization testing results]]
** Check other tier 2 locales as time permits
* Spot check a few CJK builds (WinXP)
** zhCN, ko Yahoo!: PASS (new build ID: 2006090918)
** ja, zhTW Google: PASS (new build ID: 2006090918)
** Other locales verified by diff scripts (rc5 vs rc6 PASS, no changes)
** Original test results:
*** https://intranet.mozilla.org/QA:FF1.5.0.7:GoogleCJK:Test_Results
*** https://intranet.mozilla.org/QA:FF1.5.0.7:YahooCJK:Test_Results
 
=== Tuesday, September 12th ===
* Focus/Exploratory testing for SSL/SMIME bug fixes
* Update testing for P1 locales
** [[Firefox:1.5.0.7:UpdateTesting|1.5.0.7 Firefox and Thunderbird Update testing results]]
 
=== Wednesday, September 13th ===
* Update testing for P1 locales
** [[Firefox:1.5.0.7:UpdateTesting|1.5.0.7 Firefox and Thunderbird Update testing results]]
* Update testing for Google/Yahoo! CJK
** [[Firefox:1.5.0.7:CJKUpdateTesting|1.5.0.7 Firefox Google/Yahoo! CJK Update testing results]]
 
=== Thursday, September 14th ===
* Firefox 1.5.0.7 is released, probably 4pm
* Spot check live updates


== Bug Verifications ==
== Bug Verifications ==


'''Identify bugs to be verified'''
When you verify the bug is fixed, resolve it as VERIFIED FIXED and replace the fixed1.8.0.7 keyword with verified1.8.0.7
* Triage fixed bugs and categorize
** Focus on security and critical bugs first
** Add "testcase" keyword for those bugs that have reproducible steps and/or attached testcase(s)
** Identify difficult to reproduce bugs that do not have a testcase
*** add [need testcase] in status whiteboard.
*** request more info from reporter, developer, or anyone interested in the bug
** Spot check "no verification necessary" bugs
*** Usually internal architecture changes with no externally visible effects
*** Quick code inspection and checkin verification with bonsai


'''Confidence metrics and sign-off requirements'''
Tips:
* Security bug verifications: 100%
* Bugs verified on 1.8.1 still need to be verified in 1.8.0.7
* Critical bug verifications: 100%
* If you don't see an easy way to verify or need further clarification add a comment to the bug or add "[needs testcase]" (without quotes) to the bug's Status Whiteboard.  
* Non-critical bug verifications will be done on a case by case basis, depending on developer feedback and community testing.


'''QA resources'''
'''Fixed'''
* MoCo QA is responsible for:
*[https://bugzilla.mozilla.org/buglist.cgi?keywords_type=allwords&keywords=fixed1.8.0.7&order=Bug+Number fixed 1.5.0.7 bugs]
** Security and critical bug verifications
** Some non-critical bugs and special test cases
** Providing bug lists to the community and pointing people to the wiki page
* Mozilla Community tasks:
** Test and help verify fixes for non-security bugs
* Target markets and Mozilla partners:
** IBM for enterprise deployment functionality
** Mozilla Japan for jp-critical issues


== Regression Testing ==
'''Verified'''
* Smoke Tests
*[https://bugzilla.mozilla.org/buglist.cgi?keywords_type=allwords&keywords=verified1.8.0.7&order=Bug+Number verified 1.5.0.7 bugs]
* BFT on all three platforms
* Java Script
* Top Sites (See FFT)
* Key extensions
* Performance
** Gross memory usage
* Security Test Library tests


== Install Testing ==
=== Priority Verifications ===
* Test en-US after initial packaging.
* 7 mail bugs ("fixed1.8.0.7" bugs, component is "mailnews" or product is "thunderbird")
* Test locales after locale packaing.
**[https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&keywords_type=allwords&keywords=fixed1.8.0.7&chfieldto=Now&field0-0-0=component&type0-0-0=substring&value0-0-0=mailnews&field0-0-1=product&type0-0-1=substring&value0-0-1=thunderbird mail bugs]


== L10n Testing ==
* 20 critical bugs ("fixed1.8.0.7" bugs, no "needs testcase" in status whiteboard, and severity is "critical")
* More in depth testing of 5 locals using Litmus L10n tests (AKA Sara's checklist)
**[https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&keywords_type=allwords&keywords=fixed1.8.0.7&chfieldto=Now&field0-0-0=status_whiteboard&type0-0-0=notsubstring&value0-0-0=need+testcase&field0-1-0=bug_severity&type0-1-0=equals&value0-1-0=critical critical bugs]
* basic install/version id checks
* L10n test matrix (add link here!)
* [http://wiki.mozilla.org/L10n:Home_Page Axel's external L10n status and owner info]


== Extension Testing ==
* 11 security bugs ("fixed1.8.0.7" bugs, no "needs testcase" in status whiteboard, and is marked "security-sensitive"):
* [https://addons.mozilla.org/extensions/showlist.php?application=firefox&category=Popular Top 10 AMO extensions] according to download numbers.
**[https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&keywords_type=allwords&keywords=fixed1.8.0.7&chfieldto=Now&field0-0-0=bug_group&type0-0-0=substring&value0-0-0=security&field0-1-0=status_whiteboard&type0-1-0=notsubstring&value0-1-0=need+testcase security bugs]
* [https://addons.mozilla.org/recommended.php Recommended sites]
* http://developer.mozilla.org/contests/extendfirefox/


== Update Testing ==
Note: If you run into a difficult bug, just add the "[need testcase]" in the status whiteboard and move on.
This is the testing of the partial and full update .mar files for the various locales once the main downloads are released.


== Community Testing ==
* [http://wiki.mozilla.org/Mozilla_QA_Community Mozilla QA Community Wiki] - Please go here for all community related updates and activities.


== Configuration Testing ==
== Need Verification ==


We need to describe our configuration testing approach better.  For now, we are generally testing these OS's:
bc's JS bugs:
* Windows XP
* https://bugzilla.mozilla.org/show_bug.cgi?id=348532
* MacOS 10.3, 10.4
* Linux Ferdora Core 4


But we need to consider our system requirements statements.  I have seen at least three versions.  Here is one:
== Cannot Verify ==
* [http://www.mozilla.com/firefox/system-requirements.html mozilla.com->Products Tab -> Firefox -> System Requirements link]
* https://bugzilla.mozilla.org/show_bug.cgi?id=347725 (no crash with 1506, 1507, or 2)
* https://bugzilla.mozilla.org/show_bug.cgi?id=350081 (not sure what to look for with testcase)

Revision as of 20:17, 20 October 2006

General Firefox 1.5.0.x Test Plan

This is the basic set of tests we run for every security & maintenance release.


Assumptions

  • Goal of this release is security fixes and important regressions, other fixes will not be included
  • Builds should have gone through the l10n meta-diff checker and the update checker before they are released to QA
  • Thunderbird will be released a few days after Firefox
  • Monday, September 4th is a holiday

Current Tasks

  • Bug triage
  • Test plan completion
  • Bug verification

Schedule

Monday, August 28th

  • RC1 delayed till Monday night/Tuesday morning. Shifting Monday's planned activities to Tuesday

Tuesday, August 29th

Wednesday, August 30th

  • Firefox BFTs - PASS
  • Build should begin l10n packaging after BFTs signoff (Rhelmer/TR)
  • Get post up on the QA blog with link to candidate builds
  • Testing of QA identified hot spots
  • Targeted exploratory testing over Developer identified hot spots

Thursday, August 31st

  • Continued testing of developer identified hot spots and bug verifications

Friday, September 1st

Tuesday, September 5th

  • Further bug verifications and testing of hot spots
  • Thunderbird Smoketests
  • Start Thunderbird BFTs

Wednesday, September 6th

Thursday, September 7th

Friday, September 8th

Monday, September 11th

Tuesday, September 12th

Wednesday, September 13th

Thursday, September 14th

  • Firefox 1.5.0.7 is released, probably 4pm
  • Spot check live updates

Bug Verifications

When you verify the bug is fixed, resolve it as VERIFIED FIXED and replace the fixed1.8.0.7 keyword with verified1.8.0.7

Tips:

  • Bugs verified on 1.8.1 still need to be verified in 1.8.0.7
  • If you don't see an easy way to verify or need further clarification add a comment to the bug or add "[needs testcase]" (without quotes) to the bug's Status Whiteboard.

Fixed

Verified

Priority Verifications

  • 7 mail bugs ("fixed1.8.0.7" bugs, component is "mailnews" or product is "thunderbird")
  • 20 critical bugs ("fixed1.8.0.7" bugs, no "needs testcase" in status whiteboard, and severity is "critical")
  • 11 security bugs ("fixed1.8.0.7" bugs, no "needs testcase" in status whiteboard, and is marked "security-sensitive"):

Note: If you run into a difficult bug, just add the "[need testcase]" in the status whiteboard and move on.


Need Verification

bc's JS bugs:

Cannot Verify

Retrieved from "https://wiki.mozilla.org/index.php?title=Firefox:1.5.0.8:Test_Plan&oldid=38632"