1,285
edits
QA/Test Automation/2011-01-12: Difference between revisions
→Goals Overview
| (15 intermediate revisions by 4 users not shown) | |||
| Line 7: | Line 7: | ||
=Attendees= | =Attendees= | ||
* | * Geo, Matt, Al, Henrik, Aaron, Anthony, Adrian | ||
= Last weeks action Items = | = Last weeks action Items = | ||
* Geo: Update documentation for handling of failing Mozmill tests | * Geo: Update documentation for handling of failing Mozmill tests | ||
** Did not update this, but the handling is somewhat up in the air right now. After the roundtable from this meeting should know better how to proceed. | ** Did not update this, but the handling is somewhat up in the air right now. After the roundtable from this meeting should know better how to proceed. | ||
* Anthony: Update coding styles and keep discussion on-going to get remaining important rules defined | * {{done|}} Anthony: Update coding styles and keep discussion on-going to get remaining important rules defined | ||
= Goals Overview = | = Goals Overview = | ||
* [https://intranet.mozilla.org/QA/Q42010_QAgoals#Automation_QA_Goals Q4 goals] | * [https://intranet.mozilla.org/QA/Q42010_QAgoals#Automation_QA_Goals Q4 goals] | ||
** Letter Grade '''B''' for the quarter | |||
* [https://intranet.mozilla.org/QA/Q12011_QAgoals#Automation_QA_Goals Q1 goals] planning | * [https://intranet.mozilla.org/QA/Q12011_QAgoals#Automation_QA_Goals Q1 goals] planning | ||
| Line 21: | Line 22: | ||
** Triggered Update Tests | ** Triggered Update Tests | ||
*** Integrate triggered Mozmill update tests into nightly automation process | *** Integrate triggered Mozmill update tests into nightly automation process | ||
**** One goal, but lower priority because it's an incremental change | |||
*** what about release candidate builds? For Q2? | *** what about release candidate builds? For Q2? | ||
**** Release candidate tests are the higher priority for Q1. They will be on-demand, rather than triggered | |||
**** Separate goal | |||
**** Measurement will be "Matt can run them." | |||
** Endurance Tests | ** Endurance Tests | ||
*** Create suite of Mozmill tests to stress-test Firefox in different areas | *** Create suite of Mozmill tests to stress-test Firefox in different areas | ||
| Line 32: | Line 37: | ||
*** We never found a good way to specify a realistic goal, whether via number of tests nor per component. Do we have other options? | *** We never found a good way to specify a realistic goal, whether via number of tests nor per component. Do we have other options? | ||
*** Given the Firefox 4 priority and other goals, how much time we would have this quarter? | *** Given the Firefox 4 priority and other goals, how much time we would have this quarter? | ||
**** Note: let's carry over panorama test goal for this. | |||
*** Should we better move our efforts to keep the testsuite green? | *** Should we better move our efforts to keep the testsuite green? | ||
**** Not a Collaboration goal, can be a process goal | |||
** Testing Server | ** Testing Server | ||
*** Setting up a testing server which allows kill our dependencies on external websites. | *** Setting up a testing server which allows kill our dependencies on external websites. | ||
**** iterative effort | |||
**** need to create a feature milestone implementation plan to define what will be delivered in chronological order | |||
= Risky Goals = | = Risky Goals = | ||
| Line 46: | Line 55: | ||
* Proposed Team Goal: All non-blocking tests automated | * Proposed Team Goal: All non-blocking tests automated | ||
== | ==Software updates (Geo/Henrik)== | ||
* | * Finished PoC for triggered nightly updates. | ||
** Takes Pulse message | |||
** Downloads appropriate build | |||
** Runs update tests, including checking that end-build == expected build ID. | |||
* Emphasis on updates for next few weeks will be streamlining on-demand release updates, since that's expected to be the immediate process pain point | |||
==Mozmill Crowd Extension (Henrik/Aaron)== | ==Mozmill Crowd Extension (Henrik/Aaron)== | ||
| Line 61: | Line 74: | ||
==Redesign Shared Modules (Geo/Henrik) == | ==Redesign Shared Modules (Geo/Henrik) == | ||
* | * PoC stage reached in December, running "toy" tests. | ||
* Currently porting real tests over to shake out early-stage issues. | |||
* Determining a milestone-based plan including pilot periods where the team as a whole can help contribute to porting (and also test out how easy the framework is to develop against) | |||
==Mozmill Result Dashboard (Henrik/Anthony)== | ==Mozmill Result Dashboard (Henrik/Anthony)== | ||
| Line 74: | Line 89: | ||
;DOM Walker + L10n API (Adrian) | ;DOM Walker + L10n API (Adrian) | ||
* Slower progress but working on the last major items to get fixed by Adrian | * Slower progress but working on the last major items to get fixed by Adrian | ||
** For the most important bug left ({{bug|614579}}), we need input from people knowing about theming stuff, as it seems to be a Firefox bug... | |||
* Can be executed by the Crowd extension | * Can be executed by the Crowd extension | ||
| Line 81: | Line 97: | ||
* Test Refactoring (Anthony) | * Test Refactoring (Anthony) | ||
** '''Following is moot until Shared Modules are refactored''' | |||
** project in place to begin once Firefox 4b9 work dies down | ** project in place to begin once Firefox 4b9 work dies down | ||
** will need team assistance to complete everything this quarter (please let me know if you are willing to help) | ** will need team assistance to complete everything this quarter (please let me know if you are willing to help) | ||
| Line 92: | Line 109: | ||
*** We have now [https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Test_Writing/Owners owners] for the different sections which have to take care of broken tests | *** We have now [https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Test_Writing/Owners owners] for the different sections which have to take care of broken tests | ||
** Firefox 4.0 (Geo) | ** Firefox 4.0 (Geo) | ||
*** | *** No particular branch-based news, see General above and Round Table below. | ||
** Firefox 3.5/3.6 (Anthony) | ** Firefox 3.5/3.6 (Anthony) | ||
*** Focusing on 4.0 failures for b9 release | *** Focusing on 4.0 failures for b9 release | ||
| Line 107: | Line 124: | ||
=Roundtable= | =Roundtable= | ||
;General | ;General | ||
* Styleguide & Refactoring | * <strike>Styleguide & Refactoring</strike> - on hold until Shared Module refactoring complete | ||
** we should fork into two separate projects | ** we should fork into two separate projects | ||
** freeze the styleguide | ** freeze the styleguide | ||