Auto-tools/Projects/Automated Test Ergonomics

In other words, make it simple.

Our Test Harnesses grew up by themselves organically and they are all different. There are a whole lot of things we can do to make this better. We would like to get as far as we can on these items this summer as possible. Our non-goal is to try and boil the ocean. Let's pick specific targets to attack first for clear wins.

Priority 0

• Parallelize the test harnesses - xpcshell
  • Bugs: bug 660788,bug 819048,bug 845748
  • Win: Reduce test run time.
  • Impact: Massive
  • Starting point - xpcshell
  • Possible leverage - existing parallelization in jittest.py, ongoing work on reftest parallelization in bug 813742, creating parallelization framework for all test runners
  • Notes: We'd take xpcshell parallelization without a generalized framework and could consider the generalized framework as a next step.

• Structured Logging in MozLog
  • Bugs: ??
  • Win: One test log format that is easily parseable for all frameworks
  • Impact: Medium - the impact here will come from inclusion of mozlog in the harnesses.
  • Starting point - mochitest?
  • Possible leverage - existing code in mach for this already.
  • Notes: Putting a structured logging in place has the potential to bring all our test harnesses together and to put a chokepoint in place for annotating the logs with relevant system measurements (CPU/RAM/writes etc). We would need to put in a "TBPL log outputter" into the mach log output for its "human" readable logging or we would need to update TBPL's parsing as we change each harness to make use of mozlog. Chose mochitest as the starting point because we're already working toward moving mochitest atop mozbase.
    • We would also need a JS implementation to log to as well that should show up in the same stream.
    • This should also implement a base level of information for all test frameworks

• Mochitest atop mozbase
  • Bugs: bug 746243 and dependents
  • Win: We are never going to start seeing the benefits of the work we put into the mozbase modules unless we start using them.
  • Impact: High
  • Possible Leverage - existing work porting b2g tests to mozbase
  • Notes: Porting mochitest to use mozbase will allow us to see and improve our mozbase modules by using them for a real-world harness. This will enable us to begin to see what items can be consolidated and generalized. First though, we'd just get mochitest using all the mozbase modules, verify that it works.

P1

• B2G Debugger Support
  • Bugs: ??
  • Win: We need a simple way to run a b2g test and have the harness drop us into the debugger as the test runs. Desktop tests already have this integration.
  • Impact: High
  • Possible leverage - existing work on desktop and android
  • Notes: It might depend on removing the use of the remote debugger server in marionette bug 797529