QA/TDAI/WorkWeek Planning: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
| (11 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
= HTML 5 Test Suite = | = HTML 5 Test Suite = | ||
* we have tests | * '''done''' we have tests | ||
* we have mochitest framework (created by dev for parser error) | * '''done''' we have mochitest framework (created by dev for parser error) | ||
** can leverage that for more testing | ** can leverage that for more testing | ||
* c++ & java DOM comparison testing (needed) make use of dev tool for that | * '''done''' c++ & java DOM comparison testing (needed) make use of dev tool for that | ||
* tokenizer, serializer, test development | * '''done''' tokenizer, serializer, test development | ||
* real-world site compare (mikeal) | * '''done''' real-world site compare (mikeal) | ||
=== By August 14 === | === By August 14 === | ||
* c++ & java DOM comparison done | * '''done''' c++ & java DOM comparison done | ||
* tokenizer, serializer | * '''done''' tokenizer, serializer | ||
=== By Sept 11 === | === By Sept 11 === | ||
* real-world site compare (working with mikeal) | * '''done''' real-world site compare (working with mikeal) | ||
* '''DONE''', maybe document.write stuff | * '''DONE''', maybe document.write stuff | ||
= Gfx Hardware Acceleration = | = Gfx Hardware Acceleration = | ||
=== By August 14 === | === By August 14 === | ||
* harness around reftest ext | * '''done''' harness around reftest ext | ||
* get resutls, might need to get a program together that will grab all the information for inclusion when we send the data back up to the results server | * '''pushed to Q4''' get resutls, might need to get a program together that will grab all the information for inclusion when we send the data back up to the results server | ||
* first pass canvas 2d | * '''pushed to Q4''' first pass canvas 2d | ||
=== By Sept 11 === | === By Sept 11 === | ||
* finish harness - work with jonathan | * '''done''' finish harness - work with jonathan | ||
* more canvas 2d | * '''pushed to Q4''' more canvas 2d | ||
* canvas 3d (bundle canvas 3d ext with it) | * '''pushed to Q4''' canvas 3d (bundle canvas 3d ext with it) | ||
* punt on results reporting | * '''pushed to Q4''' punt on results reporting | ||
= Multi Process = | = Multi Process = | ||
=== By August 14 === | === By August 14 === | ||
* have a build, keep an eye on the project | * '''done''' have a build, keep an eye on the project | ||
=== By Sept 11 === | === By Sept 11 === | ||
* understand how to get crash reporting data | * '''dev team aware, issue pushed out''' understand how to get crash reporting data | ||
* begin understanding how to help test this, xpcshell multiprocess, etc | * '''done''' begin understanding how to help test this, xpcshell multiprocess, etc | ||
* how do we have to retrofit the crashtest so that it wills still detect crashes. | * '''pushed out to content process work''' how do we have to retrofit the crashtest so that it wills still detect crashes. | ||
* see about qae tools (if they will be needed, what will be needed etc) | * '''done, too early to decide, some thoughts talked about with dev team''' see about qae tools (if they will be needed, what will be needed etc) | ||
* ''' | * '''done''' attend platform meeting next week see what we can do. | ||
* '''addition''' started on plugin tests | |||
* '''addition''' investigated IPCShell test framework | |||
= JS Reftests = | = JS Reftests = | ||
=== By August 14 === | === By August 14 === | ||
* Browser test completed - patch under review | * '''done''' Browser test completed - patch under review | ||
** '''DONE''' Reftests Done | ** '''DONE''' Reftests Done | ||
=== By Sept 11 === | === By Sept 11 === | ||
* Shell test completed | * '''in progress''' (working with dmandelin on this) Shell test completed | ||
* Integrated with buildbot - all they need is the python line and they can integrate it into the buildbot slaves uch that they can integrate it into the parallel setup. (in other words, the build team will own this) | * '''pushed - had to push this out due to late landing of patches''' Integrated with buildbot - all they need is the python line and they can integrate it into the buildbot slaves uch that they can integrate it into the parallel setup. (in other words, the build team will own this) This will be run on tracemonkey and m-c, then 1.9.2 and then 1.9.1 (in terms of decreasing priority | ||
= Fennec Test Results = | = Fennec Test Results (Log Compare)= | ||
* Discuss data format for results | * '''done''' Discuss data format for results | ||
=== By August 14 === | === By August 14 === | ||
* Scripts running and uploading to server | * '''done''' Scripts running and uploading to server | ||
** corner cases/fx & build to get buy in on log parsing | ** '''done''' corner cases/fx & build to get buy in on log parsing | ||
* Version 1 of Web site to view the results will be ready | * '''DONE''' Version 1 of Web site to view the results will be ready | ||
* Aki ready to plug them into his buildbot slaves | * '''done''' Aki ready to plug them into his buildbot slaves | ||
=== By Sept 11 === | === By Sept 11 === | ||
* Addressing Feedback, better views possibly | * '''done''' Scripts parsing correctly | ||
* '''done''' Scripts integrated into buildbot slaves | |||
* '''done''' Addressing Feedback, better views possibly or data mining | |||
= Mobile TUnit Integration = | = Mobile TUnit Integration = | ||
=== By August 14 === | === By August 14 === | ||
* Patches for test harnesses on WinMo for Reftest and Mochitest | * ''' DONE''' Patches for test harnesses on WinMo for Reftest and XPCShell | ||
* Manual testing as required | * '''done''' Beginning of patch for 1 by 1 Mochitest | ||
* Settle on the requirements for beta and release builds | * '''done''' Manual testing as required | ||
* '''done, but still working to get dev buy-in''' Settle on the requirements for beta and release builds | |||
=== By Sept 11 === | === By Sept 11 === | ||
* Maemo Testing (with qae as well) for 1.0 release | * '''done''' Plan of action to get faster cycling test boxes (possibly using fennec desktop), do we do tracemonkey, unit tests for mobile 1.9.2, m-c, how often? | ||
* '''done - still working on best way to run mochitests, but all test frameworks have been attempted on each platform''' Results generated for end to end unit tests with patches for winmo and wince | |||
* '''done''' Maemo Testing (with qae as well) for 1.0 release | |||
= Automatic Crash Repro = | = Automatic Crash Repro = | ||
=== By August 14 === | === By August 14 === | ||
* Currently need help getting the crash URLs minimized/tested/bugged (weekly, ongoing) | * '''done''' Currently need help getting the crash URLs minimized/tested/bugged (weekly, ongoing) | ||
** folks will need builds, !exploitable, crashwrangler, lithium | ** '''done''' folks will need builds, !exploitable, crashwrangler, lithium | ||
** can we get security folks to help us with it? | ** '''done''' can we get security folks to help us with it? | ||
** most important thing is to get the bug filed with a version of the saved page and then invite folks to minimize | ** '''done''' most important thing is to get the bug filed with a version of the saved page and then invite folks to minimize | ||
** Bob will email the crash URLs that are repro'd | ** '''done''' Bob will email the crash URLs that are repro'd | ||
* work out the procedure for dealing the URL duplicates | * '''done''' work out the procedure for dealing the URL duplicates | ||
* web app interface for jobs | * '''done''' web app interface for jobs | ||
* pool mgmt on web app | * '''framework done, ui pushed''' pool mgmt on web app | ||
* adding the proper interfaces to sisyphus for communication with the web app | * '''in progress, should make end of quarter''' adding the proper interfaces to sisyphus for communication with the web app | ||
=== By Sept 11 === | === By Sept 11 === | ||
* prioritize and implement the items from "phase 2" (see wiki), probably not all of them, but the highest priority ones first. | * '''done''' getting good lists from socorro daily now 3.0, 3.5, 3.6 | ||
* '''done (decided to go back to centos)''' what to do with ubuntu | |||
* '''pushed to Q4''' prioritize and implement the items from "phase 2" (see wiki), probably not all of them, but the highest priority ones first. | |||
* clean up web app | * clean up web app | ||
= Test Case Mgr = | = Test Case Mgr = | ||
=== By August 14 === | === By August 14 === | ||
* Test case repository | * '''DONE''' Test case repository | ||
** First version of repo to store and retrieve testcases | ** '''done''' First version of repo to store and retrieve testcases | ||
** Data model for the test cases | ** '''done''' Data model for the test cases | ||
** web interface for running them (maybe) | ** '''pushed''' web interface for running them (maybe) | ||
** internationalization for test cases | ** '''done'''internationalization for test cases | ||
=== By Sept 11 === | === By Sept 11 === | ||
** simple test case writer UI | ** '''DONE''' simple test case writer UI | ||
* '''done (except for runner)''' tags, test runner, collections | |||
* '''done, need to have follow up discussion''' have discussion about porting litmus meta data over | |||
* '''done (addressing as they are brought to light)''' other workflow issues | |||
= XBL 2 = | = XBL 2 = | ||
=== By August 14 === | === By August 14 === | ||
* Plan in place for changes to frameworks | * '''done''' Plan in place for changes to frameworks | ||
* Plan for sample tests for development review | * '''done''' Plan for sample tests for development review | ||
* Plan to address emulation mode (if one is implemented) | * '''non-issue, dropped''' Plan to address emulation mode (if one is implemented) | ||
=== By Sept 11 === | === By Sept 11 === | ||
* Begin Implementing Plan (share into Q4) | * '''done'''Begin Implementing Plan (share into Q4) | ||
= Compositor = | = Compositor = | ||
* In patches, not reviewed, passing existing tests | * '''done''' In patches, not reviewed, passing existing tests | ||
* May not need any framework help | * '''done, actually it does, but those changes have not landed yet, waiting on roc''' May not need any framework help | ||
=== By August 14 === | === By August 14 === | ||
* plugin tests | * '''done, manual''' plugin tests | ||
* iframes | * '''done, manual''' iframes | ||
* scrolling tests | * '''done, manual''' scrolling tests | ||
=== By Sept 11 === | === By Sept 11 === | ||
'''done'''Follow up with QAE about the tests that martijn and I found | |||
= JS Animation = | = JS Animation = | ||
=== By August 14 === | === By August 14 === | ||
* use reftest to do the simple tests | * '''DONE (for SMIL)''' use reftest to do the simple tests | ||
* use mochitest for frame by frame comparison | * '''in progress, support landed''' use mochitest for frame by frame comparison | ||
* first pass of tests for css transitions and SMIL individually | * '''done (for smil), pushed (for transitions)''' first pass of tests for css transitions and SMIL individually | ||
=== By Sept 11 === | === By Sept 11 === | ||
* tests for CSS transitions and SMIL used together (for instance bug 504652) | * '''done''' SMIL tests, testing animating other properties than length | ||
* '''pushed''' Transition tests | |||
* '''pushed''' tests for CSS transitions and SMIL used together (for instance bug 504652) | |||
= QAC = | = QAC = | ||
* Bugzilla tab is done | * '''done''' Bugzilla tab is done | ||
* design lunch next week | * '''done''' design lunch next week | ||
* Litmus tab this week | * '''done''' Litmus tab this week | ||
* Aaron's patch for login stuff | * '''done''' Aaron's patch for login stuff | ||
* end of the July --> DONE | * '''done''' end of the July --> DONE | ||
=== By August 14 === | === By August 14 === | ||
n/a | n/a | ||
=== By Sept 11 === | === By Sept 11 === | ||
* Port to Fennec | * '''done''' Stats tab, bug fixes | ||
* Take/Upload screen shots | * '''done'''Port to Fennec | ||
* Collect meta data from the device (memory/uses etc) | ** '''done''' Bug reporting | ||
** '''pushed''' (Possibly) Take/Upload screen shots | |||
** '''pushed''' (possibly) Collect meta data from the device (memory/uses etc) | |||
= Mozmill Integration = | = Mozmill Integration = | ||
=== By August 14 === | === By August 14 === | ||
* Buildbot integration finished | * '''pushed to second sprint''' Buildbot integration finished | ||
=== By Sept 11 === | === By Sept 11 === | ||
* '''working with build team to address''' Issues with windows | |||
* '''in progress''' Do mac and linux first with buildbot, do it soon | |||
* '''framework done, ui pushed''' Web interface for mozmill results to see the results up there. | |||
* '''done''' Adding interface to brasstacks for reporting information to hook to mozmill | |||
= Intermittent Orange = | = Intermittent Orange = | ||
=== By August 14 === | === By August 14 === | ||
=== By Sept 11 === | === By Sept 11 === | ||
* jquery tests | * '''done, still needs more work''' jquery tests | ||
* geolocation | * '''done, still needs more work''' geolocation | ||
* follow up with assignments made at the all hands | * '''pushed''' follow up with assignments made at the all hands | ||
* results server stuff shoudl eb able to help track this data like ted's top fails report | * '''pushed (prioritized getting fennec log compare solid before moving to ffx)''' results server stuff shoudl eb able to help track this data like ted's top fails report | ||
= Fennec Test Cases = | = Fennec Test Cases = | ||
=== By August 14 === | === By August 14 === | ||
=== By Sept 11 === | === By Sept 11 === | ||
* (UNOWNED) Great to get a form of mozmill running on fennec | |||
= Extension Performance Testing = | = Extension Performance Testing = | ||
=== By August 14 === | === By August 14 === | ||
=== By Sept 11 === | === By Sept 11 === | ||
* Report for a set of extensions run through a fixed page set | * '''done''' Report for a set of extensions run through a fixed page set | ||
* Give them a list of load times | * '''done''' Give them a list of load times | ||
* Reuse talos infrastructure locally if possible | * '''done, decided to use sisyphus instead''' Reuse talos infrastructure locally if possible | ||
---> '''DONE''' | |||
Latest revision as of 16:08, 14 September 2009
HTML 5 Test Suite
- done we have tests
- done we have mochitest framework (created by dev for parser error)
- can leverage that for more testing
- done c++ & java DOM comparison testing (needed) make use of dev tool for that
- done tokenizer, serializer, test development
- done real-world site compare (mikeal)
By August 14
- done c++ & java DOM comparison done
- done tokenizer, serializer
By Sept 11
- done real-world site compare (working with mikeal)
- DONE, maybe document.write stuff
Gfx Hardware Acceleration
By August 14
- done harness around reftest ext
- pushed to Q4 get resutls, might need to get a program together that will grab all the information for inclusion when we send the data back up to the results server
- pushed to Q4 first pass canvas 2d
By Sept 11
- done finish harness - work with jonathan
- pushed to Q4 more canvas 2d
- pushed to Q4 canvas 3d (bundle canvas 3d ext with it)
- pushed to Q4 punt on results reporting
Multi Process
By August 14
- done have a build, keep an eye on the project
By Sept 11
- dev team aware, issue pushed out understand how to get crash reporting data
- done begin understanding how to help test this, xpcshell multiprocess, etc
- pushed out to content process work how do we have to retrofit the crashtest so that it wills still detect crashes.
- done, too early to decide, some thoughts talked about with dev team see about qae tools (if they will be needed, what will be needed etc)
- done attend platform meeting next week see what we can do.
- addition started on plugin tests
- addition investigated IPCShell test framework
JS Reftests
By August 14
- done Browser test completed - patch under review
- DONE Reftests Done
By Sept 11
- in progress (working with dmandelin on this) Shell test completed
- pushed - had to push this out due to late landing of patches Integrated with buildbot - all they need is the python line and they can integrate it into the buildbot slaves uch that they can integrate it into the parallel setup. (in other words, the build team will own this) This will be run on tracemonkey and m-c, then 1.9.2 and then 1.9.1 (in terms of decreasing priority
Fennec Test Results (Log Compare)
- done Discuss data format for results
By August 14
- done Scripts running and uploading to server
- done corner cases/fx & build to get buy in on log parsing
- DONE Version 1 of Web site to view the results will be ready
- done Aki ready to plug them into his buildbot slaves
By Sept 11
- done Scripts parsing correctly
- done Scripts integrated into buildbot slaves
- done Addressing Feedback, better views possibly or data mining
Mobile TUnit Integration
By August 14
- DONE Patches for test harnesses on WinMo for Reftest and XPCShell
- done Beginning of patch for 1 by 1 Mochitest
- done Manual testing as required
- done, but still working to get dev buy-in Settle on the requirements for beta and release builds
By Sept 11
- done Plan of action to get faster cycling test boxes (possibly using fennec desktop), do we do tracemonkey, unit tests for mobile 1.9.2, m-c, how often?
- done - still working on best way to run mochitests, but all test frameworks have been attempted on each platform Results generated for end to end unit tests with patches for winmo and wince
- done Maemo Testing (with qae as well) for 1.0 release
Automatic Crash Repro
By August 14
- done Currently need help getting the crash URLs minimized/tested/bugged (weekly, ongoing)
- done folks will need builds, !exploitable, crashwrangler, lithium
- done can we get security folks to help us with it?
- done most important thing is to get the bug filed with a version of the saved page and then invite folks to minimize
- done Bob will email the crash URLs that are repro'd
- done work out the procedure for dealing the URL duplicates
- done web app interface for jobs
- framework done, ui pushed pool mgmt on web app
- in progress, should make end of quarter adding the proper interfaces to sisyphus for communication with the web app
By Sept 11
- done getting good lists from socorro daily now 3.0, 3.5, 3.6
- done (decided to go back to centos) what to do with ubuntu
- pushed to Q4 prioritize and implement the items from "phase 2" (see wiki), probably not all of them, but the highest priority ones first.
- clean up web app
Test Case Mgr
By August 14
- DONE Test case repository
- done First version of repo to store and retrieve testcases
- done Data model for the test cases
- pushed web interface for running them (maybe)
- doneinternationalization for test cases
By Sept 11
- DONE simple test case writer UI
- done (except for runner) tags, test runner, collections
- done, need to have follow up discussion have discussion about porting litmus meta data over
- done (addressing as they are brought to light) other workflow issues
XBL 2
By August 14
- done Plan in place for changes to frameworks
- done Plan for sample tests for development review
- non-issue, dropped Plan to address emulation mode (if one is implemented)
By Sept 11
- doneBegin Implementing Plan (share into Q4)
Compositor
- done In patches, not reviewed, passing existing tests
- done, actually it does, but those changes have not landed yet, waiting on roc May not need any framework help
By August 14
- done, manual plugin tests
- done, manual iframes
- done, manual scrolling tests
By Sept 11
doneFollow up with QAE about the tests that martijn and I found
JS Animation
By August 14
- DONE (for SMIL) use reftest to do the simple tests
- in progress, support landed use mochitest for frame by frame comparison
- done (for smil), pushed (for transitions) first pass of tests for css transitions and SMIL individually
By Sept 11
- done SMIL tests, testing animating other properties than length
- pushed Transition tests
- pushed tests for CSS transitions and SMIL used together (for instance bug 504652)
QAC
- done Bugzilla tab is done
- done design lunch next week
- done Litmus tab this week
- done Aaron's patch for login stuff
- done end of the July --> DONE
By August 14
n/a
By Sept 11
- done Stats tab, bug fixes
- donePort to Fennec
- done Bug reporting
- pushed (Possibly) Take/Upload screen shots
- pushed (possibly) Collect meta data from the device (memory/uses etc)
Mozmill Integration
By August 14
- pushed to second sprint Buildbot integration finished
By Sept 11
- working with build team to address Issues with windows
- in progress Do mac and linux first with buildbot, do it soon
- framework done, ui pushed Web interface for mozmill results to see the results up there.
- done Adding interface to brasstacks for reporting information to hook to mozmill
Intermittent Orange
By August 14
By Sept 11
- done, still needs more work jquery tests
- done, still needs more work geolocation
- pushed follow up with assignments made at the all hands
- pushed (prioritized getting fennec log compare solid before moving to ffx) results server stuff shoudl eb able to help track this data like ted's top fails report
Fennec Test Cases
By August 14
By Sept 11
- (UNOWNED) Great to get a form of mozmill running on fennec
Extension Performance Testing
By August 14
By Sept 11
- done Report for a set of extensions run through a fixed page set
- done Give them a list of load times
- done, decided to use sisyphus instead Reuse talos infrastructure locally if possible
---> DONE