Test Harnesses into Own Repo - (Independence Day)
|
- Enables us to run tests on all versions without backporting harness infrastrucutre
- Makes developers able to write tests for any branch in any test type
- Easier to maintain
- Easier to catch breakage
|
1.5 - 2 Quarters
|
Universal Manifest - (Rosetta)
|
- Makes all tests use the same manifest structure
- Gives us ability to manifest for mochi* tests
- Gives us the ability to handle different behaviors per product, per platform
- Gives us the ability to create a better manifest than what we have now
- Gives us flexibility to selectively enable/disable tests (could create an "orange" only run for debugging)
|
- 1 Quarter design and experiment with manifest, selling the idea to dev, refactoring mochitest/xpcshell to use a manifest at all.
- 1 month implementation reftest style harnesses
- 1 month porting and testing
- TODO: Would we want to keep old behavior going forward or just new behavior going forward? How does that change if we do the "test harness in own repo"?
|
Universal Test Runner - (Loderunner)
|
- One test runner for all test types
- Simplifies development? Simplifies harness code?
- What problem does this solve?
- Lowers barrier to entry?
|
1 Quarter, design and implementation, refactoring wrapper for all test classes.
|
User Configuration/Dirty Profile Test Infrastructure - (Dirty Harry)
|
- Re-use existing infrastructure for crash & valgrind infrastructure so that we can specify any type of profile we want.
- Create large bookmark/tag/history tests
- Vary extensions
- Vary saved passwords for sites
- Do this to test: startup time performance, page load performance, how our unit tests run in this configuration etc
- DEPENDS ON RESULT DB
|
- 1 month - experiment with extending sisyphus/infra to specify profile details
- 1 month to implement
- 1 month to craft specific perf tests
|
Universal Results Reporting Db - (Bookkeeper)
|
- Allows us to store all results we need in one place
- Easy, extensible infrastructure to extend views to new data over time
- Open up an API to any dev member who crafts some kind of test to store data into the database
- Provide analytics using off-the-shelf tools to analyze results metrics (intermittent tests for example)
|
- 1 month design db, perform performance experiments
- 1 month implementation and begin on UI for views
- 1 month to complete UI for views, add in the APIs needed for ease of adding data into the db.
- 1-2 months reviews, unforseen complications
|
Litmus v2.0 - (Litmus 2.0)
|
- Placeholder in case the Litmus project comes back to us
- ??
|
??
|
Mozmill 2.0
|
- What do we need to get to 2.0?
- Will we do a 1.5 before that? (I think yes)
|
??
|
Coherent effort to find/fix [orange] issues - (Orangejuice)
|
- Provide seamless integration for record/replay vms (already done?)
- Start busting a bunch of oranges
|
To make a dent we need to do at least 50 oranges before we see a noticeable decline in the rate of intermittent problems
|
Android Test Infra - (R2D2)
|
- Find and fix issues with infrastructure on android
- Help RelEng with issues as they move ahead with putting infrastructure in staging
- ??
|
??
|
Crash/Assertion/Valgrind UI 2.0 - (CrystalBall)
|
- The true potential of the crash/automation/valgrind information we have is to help us define and track down hard to reproduce crashes and to identify unstable pieces of code.
- The proper UI for this will demonstrate these correlations and allow dev's to view the relationships between this data.
- This is actually a formalization/aid of a process we do ad-hoc in bugzilla.
|
- 1 month, designs/prototypes
- 1-1.5 month implementation
|
Bugzilla Improvements - (Hydra)
|
- We have talked a lot about improvements to bugzilla but we have never concretely listed what they are.
- What are they?
- What is actually desired/needed?
|
??
|
Symmetric Automation - (Doppelganger)
|
- Creating a secure mirror repo and infrastructure for security fixes
- Not sure how much of this we need to do
|
Depends on how much of this is ours versus other groups
|
Crowd Sourced Unittests - (FeedGrass)
|
- Allow for unittests to be run on various hardware models of beta testers. Sort of like GrafxBot, but for all tests.
- Ideal for Fennec where automation requires more resources and time while the hardware list is much larger.
|
Could piggyback off mozmill or grafxbot, need a lot of dev support. Probably 1 quarter to develop and get off the ground and 1 more quarter to polish up and evangalize
|
Toolset to engage users - (Powerball)
|
- Addon in the browser to help detect user actions to broken functionality (web site wrong, plugin crash, bug in firefox, usability issue), then redirect to bugzilla
- Bugzilla interface to help file bug in appropriate place and search for dups
- Engage user after filing bug to reproduce or verify similar bugs on same hardware, version - better for triage
- Allow end user to earn points and share on social networks for filing, reproducing and verifying bugs
- Make this tangible to really build hype and competition between friends QMO 2011 Roadmap
- Provide ways to tell users when their bug is fixed that it is released in which version...put their names in the credits and they share with all their friends!
|
A lot of pieces here that are not even half baked. Just some ideas of a work flow that makes a lot of sense. This would be more of an ongoing project that could be done bits and pieces each quarter, or probably 1 quarter closer to full time.
|