Auto-tools/Goals/Proposed 2010 Q2

Project	Purpose	WAG time
Test Harnesses into Own Repo - (Independence Day)	Enables us to run tests on all versions without backporting harness infrastrucutre Makes developers able to write tests for any branch in any test type Easier to maintain Easier to catch breakage	1.5 - 2 Quarters
Universal Manifest - (Rosetta)	Makes all tests use the same manifest structure Gives us ability to manifest for mochi* tests Gives us the ability to handle different behaviors per product, per platform Gives us the ability to create a better manifest than what we have now Gives us flexibility to selectively enable/disable tests (could create an "orange" only run for debugging)	1 Quarter design and experiment with manifest, selling the idea to dev, refactoring mochitest/xpcshell to use a manifest at all. 1 month implementation reftest style harnesses 1 month porting and testing TODO: Would we want to keep old behavior going forward or just new behavior going forward? How does that change if we do the "test harness in own repo"?
Universal Test Runner - (Loderunner)	One test runner for all test types Simplifies development? Simplifies harness code? What problem does this solve? Lowers barrier to entry?	1 Quarter, design and implementation, refactoring wrapper for all test classes.
User Configuration/Dirty Profile Test Infrastructure - (Dirty Harry)	Re-use existing infrastructure for crash & valgrind infrastructure so that we can specify any type of profile we want. Create large bookmark/tag/history tests Vary extensions Vary saved passwords for sites Do this to test: startup time performance, page load performance, how our unit tests run in this configuration etc DEPENDS ON RESULT DB	1 month - experiment with extending sisyphus/infra to specify profile details 1 month to implement 1 month to craft specific perf tests
Universal Results Reporting Db - (Bookkeeper)	Allows us to store all results we need in one place Easy, extensible infrastructure to extend views to new data over time Open up an API to any dev member who crafts some kind of test to store data into the database Provide analytics using off-the-shelf tools to analyze results metrics (intermittent tests for example)	1 month design db, perform performance experiments 1 month implementation and begin on UI for views 1 month to complete UI for views, add in the APIs needed for ease of adding data into the db. 1-2 months reviews, unforseen complications
Mozmill 2.0	What do we need to get to 2.0? Will we do a 1.5 before that? (I think yes)	??
Coherent effort to find/fix [orange] issues - (Orangejuice)	Provide seamless integration for record/replay vms (already done?) Start busting a bunch of oranges	To make a dent we need to do at least 50 oranges before we see a noticeable decline in the rate of intermittent problems
Android Test Infra - (R2D2)	* Find and fix issues with infrastructure on android Help RelEng with issues as they move ahead with putting infrastructure in staging ??	??