« QA/Firefox3/Planning Meetings

Notes from Fx3 Planning meeting

Scheduling

• Current Fx3 plan is to have 6 alphas, 2 betas or more, unknown # of RCs.
• Plan is to place a test case milestone for QA to finish writing test cases by a certain date:
  • Try to finish writing test cases for feature by Alpha 6 time frame (end of june)
  • Try to finish executing test cases for feature 2-4 weeks after Feature complete

   Eg. if Places development is done by End of June, Places testing will be completed by end of July

• Create a schedule for QA to prioritize tests based on the release time frame.

   Eg. BFTs every release cycle, FFTs for Beta 1, L10N tests for RC1, etc...

• Having a timeline will help track % of test progress at each milestone. It will give us an idea how realistic our goals are, as well as look at where external issues are tracking and could be blocking QA. Items like:
  • Feature Development progress. If dev is late to deliver, this could affect QA scheduing
  • Build and respin roadblocks
  • Need for better documentation, dev design notes, PRD clarity
  • Any other firedrills or randomization of duties that could pull QA off track from the original schedule
• Need to incorporate a section for L10N test tracking into the schedule. l10N section includes test results for Trademark tests, Smartware tests, and QA l10N spotchecks (tier 3)

Documentation

• QA will assume ownership of Fx3 areas and become the resident expert of these areas. Areas of ownership can be found here: http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_3.0_TestPlan
• QA will create a test plan that encompasses test cases in Litmus based on the feature they own
• QA will have a test plan sign off stage, which will incorporate feedback from other QA peers, feature developers, project managers, and even a few VP's if necessary. There may be multiple revisions of the test plan document
• A test plan will include items like Fx3 feature test cases, identify regression areas from Fx2 that are relevant to test for Fx3, and divvy out areas that belong in BFTs, smoketests, and/or FFTs.

Reporting Results

• Currently, Litmus Test cases only record results for test case execution
• Compatibility tests that fall into a special category are recorded in the wiki. These include:
  • extensions
  • Plugins
  • top site
  • accessibility?
  • Final tests (bookmarks, customization, help, community, about, firstrun, what's new)
• Can we create a form or template where QA can report results (OS, build, pass/fail, notes, Release stage), and port the results to Litmus? (talk to coop, tracy)

Miscellaneous Notes

• Instead of having a special compatibility test suite for Profile Tests, we should create a test suite in Litmus and place it there.
• Review current ways of bug tracking and bug regression. is there a better way to find bugs out there that require regression test cases to be created in litmus?