|
|
| (5 intermediate revisions by the same user not shown) |
| Line 5: |
Line 5: |
| |class="header"|WAG time | | |class="header"|WAG time |
| |- | | |- |
| | '''Test Harnesses into Own Repo - (Independence Day)''' | | | Maemo Qt Agent & test Harenesses |
| | | |
| | * Expand our sutagent system to maemo qt |
| | | | | |
| * Enables us to run tests on all versions without backporting harness infrastrucutre
| | ?? |
| * Makes developers able to write tests for any branch in any test type
| |
| * Easier to maintain
| |
| * Easier to catch breakage
| |
| | 1.5 - 2 Quarters
| |
| |- | | |- |
| | '''Universal Manifest - (Rosetta)''' | | | Android Agent & test harnesses |
| | | | | |
| * Makes all tests use the same manifest structure | | * Aid RelEng in rolling out the Android harness and test system |
| * Gives us ability to manifest for mochi* tests | | * Finish Android agent |
| * Gives us the ability to handle different behaviors per product, per platform
| |
| * Gives us the ability to create a better manifest than what we have now
| |
| * Gives us flexibility to selectively enable/disable tests (could create an "orange" only run for debugging)
| |
| | | | | |
| * 1 Quarter design and experiment with manifest, selling the idea to dev, refactoring mochitest/xpcshell to use a manifest at all. | | * 1 month dev time |
| * 1 month implementation reftest style harnesses
| | * 2 months support releng |
| * 1 month porting and testing
| |
| * TODO: Would we want to keep old behavior going forward or just new behavior going forward? How does that change if we do the "test harness in own repo"? | |
| |- | | |- |
| | '''Universal Test Runner - (Loderunner)''' | | | Profile Manager Replacement |
| | | |
| | * Replace the profile manager with a good xulrunner based tool |
| | | | | |
| * One test runner for all test types
| | ?? |
| * Simplifies development? Simplifies harness code?
| |
| * What problem does this solve?
| |
| * Lowers barrier to entry?
| |
| | 1 Quarter, design and implementation, refactoring wrapper for all test classes.
| |
| |- | | |- |
| | '''User Configuration/Dirty Profile Test Infrastructure - (Dirty Harry)''' | | | Mozmill + Buildbot |
| |
| |
| * Re-use existing infrastructure for crash & valgrind infrastructure so that we can specify any type of profile we want.
| |
| * Create large bookmark/tag/history tests
| |
| * Vary extensions
| |
| * Vary saved passwords for sites
| |
| * Do this to test: startup time performance, page load performance, how our unit tests run in this configuration etc
| |
| * <font color="red">DEPENDS ON RESULT DB</font>
| |
| | | | | |
| * 1 month - experiment with extending sisyphus/infra to specify profile details | | * Get Mozmill unitests running in a buildbot framework |
| * 1 month to implement
| | | ?? |
| * 1 month to craft specific perf tests
| |
| |- | | |- |
| | '''Universal Results Reporting Db - (Bookkeeper)''' | | | Crash Automation UI 2.0 |
| | | | | |
| * Allows us to store all results we need in one place | | * Make it easier to view the correlations between crashes, assertions, valgrind messages |
| * Easy, extensible infrastructure to extend views to new data over time
| | | |
| * Open up an API to any dev member who crafts some kind of test to store data into the database
| | * 1 month design |
| * Provide analytics using off-the-shelf tools to analyze results metrics (intermittent tests for example)
| | * 1 month implementation |
| | | |
| * 1 month design db, perform performance experiments | |
| * 1 month implementation and begin on UI for views | |
| * 1 month to complete UI for views, add in the APIs needed for ease of adding data into the db.
| |
| * 1-2 months reviews, unforseen complications
| |
| |- | | |- |
| | '''Mozmill 2.0''' | | | Mozmill.next |
| |
| | | |
| * What do we need to get to 2.0?
| |
| * Will we do a 1.5 before that? (I think yes)
| |
| | | |
| ?? | | ?? |
| | | ?? |
| |- | | |- |
| | '''Coherent effort to find/fix [orange] issues - (Orangejuice)''' | | | Launch HAL Reftest |
| | | |
| | * Get testing on driver/version/os configurations in the wild |
| | | ?? |
| | |- |
| | | Get Purify turned on for testing |
| | | | | |
| * Provide seamless integration for record/replay vms (already done?) | | * So that we get purify messages on windows where we have no valgrind coverage |
| * Start busting a bunch of oranges
| | | 3 weeks? |
| | | |
| To make a dent we need to do at least 50 oranges before we see a noticeable decline in the rate of intermittent problems
| |
| |- | | |- |
| | '''Android Test Infra - (R2D2)''' | | | The next big dream (see email) |
| |* Find and fix issues with infrastructure on android | | | ?? |
| * Help RelEng with issues as they move ahead with putting infrastructure in staging
| | | ?? |
| * ??
| |
| |?? | |
| |} | | |} |