Firefox:1.5.0.8:Test Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
 
(35 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= General Firefox 1.5.0.x Test Plan =
== [[Firefox:1.5.0.x:Test_Plan | General Firefox 1.5.0.x Test Plan]] ==
'''NOTES: This will be updated shortly, for 1.5.0.8'''
This is the basic set of tests we run for every security & maintenance release.


== Bug Verifications ==


'''Identify bugs to be verified'''
== Assumptions ==
* Triage fixed bugs and categorize
 
** Focus on security and critical bugs first
* This release will be piggy-backing on 1.8.1
** Add "testcase" keyword for those bugs that have reproducible steps and/or attached testcase(s)
* Builds should have gone through the l10n meta-diff checker and the update checker before they are released to QA
** Identify difficult to reproduce bugs that do not have a testcase
 
*** add [need testcase] in status whiteboard.
== Current Tasks ==
*** request more info from reporter, developer, or anyone interested in the bug
 
** Spot check "no verification necessary" bugs
* Bug triage
*** Usually internal architecture changes with no externally visible effects
* Test plan completion
*** Quick code inspection and checkin verification with bonsai
* Minor/Major Software Updates discussions and testing
* Bug verification
* Tuesday, October 24 - Smoketests - tracy - '''PASS'''
 
== Schedule ==
 
=== Tuesday, October 17 ===
* Start bug verifications for bugs marked fixed1.8.0.8
* Post to QA blog for Community Testing ?
 
=== Friday, October 20 ===
* Continue bug verifications
* Start testing minor/major softare updates with mock-ups provided by Seth
 
=== Tuesday, October 24 ===
* FF 1.5.0.8rc1 builds ready
* Firefox Smoketets on all platforms - tracy - '''PASS'''
* L10n builds ready to be tested
* BFT's started [[Firefox:1.5.0.8:Test_Plan:BFT | BFT Results]]
 
=== Wednesday, October 25 ===
*Smoketest and BFTs for Thunderbird
*Start Firefox L10 Testing
*Continue Verifications
*Minor/Major Software Updates Testing
 
=== Thursday, October 26 ===
'''If not ahead of schedule:'''
*Smoketest and BFTs for Thunderbird
*Start Firefox L10 Testing
*Continue Verifications
*Minor/Major Software Updates Testing
 
=== Friday, October 27 - Friday, November 3 ===
* Start [http://wiki.mozilla.org/Firefox:1.5.0.8:Test_Plan:l10n l10n testing] for both Firefox and Thunderbird
* Testing of QA identified hot spots: [http://wiki.mozilla.org/Firefox:1.5.0.8:Test_Plan:Software_Update Software Updates]
* [[Firefox:1.5.0.8:Test_Plan:Minor Major Software Update | Minor-Major Software Update Testing]]
* L10n + Updates testing
* Community Test Day
 
=== Monday, November 6th ===
* End-to-end Major update testing for 1.5.0.8 -> 2.0
* Wrap up bug verifications
* Final l10n + update testing on staging server.
 
=== Tuesday, November 7th ===
* Firefox 1.5.0.8 is released, probably 4pm
* Push website changes and security notes
* Spot check live updates and download links
 
=== Wednesday, November 8 - Monday, November 13 ===
* Start special distro spotchecks, and continue until Monday, Nov 13
 
=== Tuesday, November 14 ===
* Test updates for special distributions
 
=== Thursday, November 16 ===
* Special distributions are released.
* Live updates for special distros.
 
== Test Results ==
Any 1.5.0.8 testing results that will help us track progress, identify regressions, and verify fixes should go here.
 
* [[Firefox:1.5.0.8:Test_Plan:BFT | Basic Functional Tests (BFTs)]]
* [[Firefox:1.5.0.8:Test_Plan:l10n | l10n related testing]]
* [[Firefox:1.5.0.8:Test_Plan:Software Update | Updates Testing]]
* [[Firefox:1.5.0.8:Test_Plan:Minor Major Software Update | Minor-Major Software Update Testing]]
*[https://intranet.mozilla.org/Firefox:Distribution:1.5:Google:1.5.0.8_Test_Results  Google Results]
 
'''The above list is just a sample of what we will be testing and links to blank pages we should us to track our test results.  Please add any other test results here.  Thanks!'''


'''Confidence metrics and sign-off requirements'''
== Bug Verifications ==
* Security bug verifications: 100%
* Critical bug verifications: 100%
* Non-critical bug verifications will be done on a case by case basis, depending on developer feedback and community testing.


'''QA resources'''
When you verify the bug is fixed, resolve it as VERIFIED FIXED and replace the fixed1.8.0.8 keyword with verified1.8.0.8
* MoCo QA is responsible for:
** Security and critical bug verifications
** Some non-critical bugs and special test cases
** Providing bug lists to the community and pointing people to the wiki page
* Mozilla Community tasks:
** Test and help verify fixes for non-security bugs
* Target markets and Mozilla partners:
** IBM for enterprise deployment functionality
** Mozilla Japan for jp-critical issues


== Regression Testing ==
Tips:
* Smoke Tests
* Bugs verified on 1.8.1 still need to be verified in 1.8.0.8
* BFT on all three platforms
* If you don't see an easy way to verify or need further clarification add a comment to the bug or add "[needs testcase]" (without quotes) to the bug's Status Whiteboard.
* Java Script
* Top Sites (See FFT)
* Key extensions
* Performance
** Gross memory usage
* Security Test Library tests


== Install Testing ==
'''Fixed'''
* Test en-US after initial packaging.
*[https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=fixed1.8.0.8&emailtype1=substring&email1=&emailtype2=subst fixed 1.5.0.8 bugs]
* Test locales after locale packaing.


== L10n Testing ==
'''Verified'''
* More in depth testing of 5 locals using Litmus L10n tests (AKA Sara's checklist)
*[https://bugzilla.mozilla.org/buglist.cgi?keywords_type=allwords&keywords=verified1.8.0.8&order=Bug+Number verified 1.5.0.8 bugs]
* basic install/version id checks
* L10n test matrix (add link here!)
* [http://wiki.mozilla.org/L10n:Home_Page Axel's external L10n status and owner info]


== Extension Testing ==
=== Priority Verifications ===
* [https://addons.mozilla.org/extensions/showlist.php?application=firefox&category=Popular Top 10 AMO extensions] according to download numbers.
* 6 mail bugs ("fixed1.8.0.8" bugs, component is "mailnews" or product is "thunderbird")
* [https://addons.mozilla.org/recommended.php Recommended sites]
**[https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&keywords_type=allwords&keywords=fixed1.8.0.8&chfieldto=Now&field0-0-0=component&type0-0-0=substring&value0-0-0=mailnews&field0-0-1=product&type0-0-1=substring&value0-0-1=thunderbird mail bugs]
* http://developer.mozilla.org/contests/extendfirefox/


== Update Testing ==
* 16 critical bugs ("fixed1.8.0.8" bugs, no "needs testcase" in status whiteboard, and severity is "critical")
This is the testing of the partial and full update .mar files for the various locales once the main downloads are released.
**[https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&keywords_type=allwords&keywords=fixed1.8.0.8&chfieldto=Now&field0-0-0=status_whiteboard&type0-0-0=notsubstring&value0-0-0=need+testcase&field0-1-0=bug_severity&type0-1-0=equals&value0-1-0=critical critical bugs]


== Community Testing ==
* 5 security bugs ("fixed1.8.0.8" bugs, no "needs testcase" in status whiteboard, and is marked "security-sensitive"):
* [http://wiki.mozilla.org/Mozilla_QA_Community Mozilla QA Community Wiki] - Please go here for all community related updates and activities.
**[https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&keywords_type=allwords&keywords=fixed1.8.0.8&chfieldto=Now&field0-0-0=bug_group&type0-0-0=substring&value0-0-0=security&field0-1-0=status_whiteboard&type0-1-0=notsubstring&value0-1-0=need+testcase security bugs]


== Configuration Testing ==
Note: If you run into a difficult bug, just add the "[need testcase]" in the status whiteboard and move on.


We need to describe our configuration testing approach better.  For now, we are generally testing these OS's:
== Need Verification ==
* Windows XP
* MacOS 10.3, 10.4
* Linux Ferdora Core 4


But we need to consider our system requirements statements.  I have seen at least three versions.  Here is one:
== Cannot Verify ==
* [http://www.mozilla.com/firefox/system-requirements.html mozilla.com->Products Tab -> Firefox -> System Requirements link]

Latest revision as of 19:51, 10 November 2006

General Firefox 1.5.0.x Test Plan

This is the basic set of tests we run for every security & maintenance release.


Assumptions

  • This release will be piggy-backing on 1.8.1
  • Builds should have gone through the l10n meta-diff checker and the update checker before they are released to QA

Current Tasks

  • Bug triage
  • Test plan completion
  • Minor/Major Software Updates discussions and testing
  • Bug verification
  • Tuesday, October 24 - Smoketests - tracy - PASS

Schedule

Tuesday, October 17

  • Start bug verifications for bugs marked fixed1.8.0.8
  • Post to QA blog for Community Testing ?

Friday, October 20

  • Continue bug verifications
  • Start testing minor/major softare updates with mock-ups provided by Seth

Tuesday, October 24

  • FF 1.5.0.8rc1 builds ready
  • Firefox Smoketets on all platforms - tracy - PASS
  • L10n builds ready to be tested
  • BFT's started BFT Results

Wednesday, October 25

  • Smoketest and BFTs for Thunderbird
  • Start Firefox L10 Testing
  • Continue Verifications
  • Minor/Major Software Updates Testing

Thursday, October 26

If not ahead of schedule:

  • Smoketest and BFTs for Thunderbird
  • Start Firefox L10 Testing
  • Continue Verifications
  • Minor/Major Software Updates Testing

Friday, October 27 - Friday, November 3

Monday, November 6th

  • End-to-end Major update testing for 1.5.0.8 -> 2.0
  • Wrap up bug verifications
  • Final l10n + update testing on staging server.

Tuesday, November 7th

  • Firefox 1.5.0.8 is released, probably 4pm
  • Push website changes and security notes
  • Spot check live updates and download links

Wednesday, November 8 - Monday, November 13

  • Start special distro spotchecks, and continue until Monday, Nov 13

Tuesday, November 14

  • Test updates for special distributions

Thursday, November 16

  • Special distributions are released.
  • Live updates for special distros.

Test Results

Any 1.5.0.8 testing results that will help us track progress, identify regressions, and verify fixes should go here.

The above list is just a sample of what we will be testing and links to blank pages we should us to track our test results. Please add any other test results here. Thanks!

Bug Verifications

When you verify the bug is fixed, resolve it as VERIFIED FIXED and replace the fixed1.8.0.8 keyword with verified1.8.0.8

Tips:

  • Bugs verified on 1.8.1 still need to be verified in 1.8.0.8
  • If you don't see an easy way to verify or need further clarification add a comment to the bug or add "[needs testcase]" (without quotes) to the bug's Status Whiteboard.

Fixed

Verified

Priority Verifications

  • 6 mail bugs ("fixed1.8.0.8" bugs, component is "mailnews" or product is "thunderbird")
  • 16 critical bugs ("fixed1.8.0.8" bugs, no "needs testcase" in status whiteboard, and severity is "critical")
  • 5 security bugs ("fixed1.8.0.8" bugs, no "needs testcase" in status whiteboard, and is marked "security-sensitive"):

Note: If you run into a difficult bug, just add the "[need testcase]" in the status whiteboard and move on.

Need Verification

Cannot Verify

Retrieved from "https://wiki.mozilla.org/index.php?title=Firefox:1.5.0.8:Test_Plan&oldid=41009"