Auto-tools/Goals/BigDreams

• Enables us to run tests on all versions without backporting harness infrastrucutre
• Makes developers able to write tests for any branch in any test type
• Easier to maintain
• Easier to catch breakage

• Makes all tests use the same manifest structure
• Gives us ability to manifest for mochi* tests
• Gives us the ability to handle different behaviors per product, per platform
• Gives us the ability to create a better manifest than what we have now
• Gives us flexibility to selectively enable/disable tests (could create an "orange" only run for debugging)

• 1 Quarter design and experiment with manifest, selling the idea to dev, refactoring mochitest/xpcshell to use a manifest at all.
• 1 month implementation reftest style harnesses
• 1 month porting and testing
• TODO: Would we want to keep old behavior going forward or just new behavior going forward? How does that change if we do the "test harness in own repo"?

• One test runner for all test types
• Simplifies development? Simplifies harness code?
• What problem does this solve?
• Lowers barrier to entry?

• Re-use existing infrastructure for crash & valgrind infrastructure so that we can specify any type of profile we want.
• Create large bookmark/tag/history tests
• Vary extensions
• Vary saved passwords for sites
• Do this to test: startup time performance, page load performance, how our unit tests run in this configuration etc
• DEPENDS ON RESULT DB

• 1 month - experiment with extending sisyphus/infra to specify profile details
• 1 month to implement
• 1 month to craft specific perf tests

• Allows us to store all results we need in one place
• Easy, extensible infrastructure to extend views to new data over time
• Open up an API to any dev member who crafts some kind of test to store data into the database
• Provide analytics using off-the-shelf tools to analyze results metrics (intermittent tests for example)

• 1 month design db, perform performance experiments
• 1 month implementation and begin on UI for views
• 1 month to complete UI for views, add in the APIs needed for ease of adding data into the db.
• 1-2 months reviews, unforseen complications

• Placeholder in case the Litmus project comes back to us
•  ??

• What do we need to get to 2.0?
• Will we do a 1.5 before that? (I think yes)

??

• Provide seamless integration for record/replay vms (already done?)
• Start busting a bunch of oranges

To make a dent we need to do at least 50 oranges before we see a noticeable decline in the rate of intermittent problems

• Find and fix issues with infrastructure on android
• Help RelEng with issues as they move ahead with putting infrastructure in staging
•  ??

• The true potential of the crash/automation/valgrind information we have is to help us define and track down hard to reproduce crashes and to identify unstable pieces of code.
• The proper UI for this will demonstrate these correlations and allow dev's to view the relationships between this data.
• This is actually a formalization/aid of a process we do ad-hoc in bugzilla.

• 1 month, designs/prototypes
• 1-1.5 month implementation

• We have talked a lot about improvements to bugzilla but we have never concretely listed what they are.
• What are they?
• What is actually desired/needed?

??

• Creating a secure mirror repo and infrastructure for security fixes
• Not sure how much of this we need to do

• Allow for unittests to be run on various hardware models of beta testers. Sort of like GrafxBot, but for all tests.
• Ideal for Fennec where automation requires more resources and time while the hardware list is much larger.

• Addon in the browser to help detect user actions to broken functionality (web site wrong, plugin crash, bug in firefox, usability issue), then redirect to bugzilla
• Bugzilla interface to help file bug in appropriate place and search for dups
• Engage user after filing bug to reproduce or verify similar bugs on same hardware, version - better for triage
• Allow end user to earn points and share on social networks for filing, reproducing and verifying bugs
• Make this tangible to really build hype and competition between friends QMO 2011 Roadmap
• Provide ways to tell users when their bug is fixed that it is released in which version...put their names in the credits and they share with all their friends!