Many Projects, all Complementary

• MozBase
• Special Powers
• Going Faster

MozBase

• Project to refactor base libraries of test harnesses into a coherent set of tools and scripts.
  • Should have very limited impact on performance since most of the "run-time" performance of tests is dependent on the JavaScript side of the test harnesses.
  • MozProcess, MozRunner, MozInfo done
  • MozLog - fragmented
  • MozProfile, MozEnv, MozOptions - not complete
  • Goal: Tie into a generic low-level harness library that can be easily wired into MozHarness.

Special Powers

• API Implemented in Mochitest
• Needed in Reftest, Mozmill
• Needed in XPCShell?
• Porting Mochitests to API
• Will have some performance impact
• Goal: Enable test harnesses to run in out of process architecture and still have access to chrome privileges/APIs as required

GoFaster

• Goal: Reducing the end to end time as measured from pushing a change to getting results.
• Early Profiling results
  • packed.js seems to average about 110ms per load.
  • There are around 3100 test files in mochitest. So, eliminating packed.js would save almost 341 seconds (5 minutes) per run.
  • In the makeIframe script, I'm seeing the window.scrollTo call take 8-10ms per call. Not sure how much we can streamline that, but best case that's 24 seconds on every run.

Related to GoFaster and MozBase

• httpd.js could be a bottleneck.
• How to profile?
• How to go about trying out replacements to it?
• Even if we stay with httpd.js, we need a better story for packaging it so that you download one item and it "just works" (tm)

Discussion Points

• How to best work together without stepping on toes?
• How can we monitor test harness performance so we know if we check in something that blows our gains?

Meeting Notes

Special powers

• Cleanup harness
  • Many tools touched (unit tests, talos etc.)
• Moving tests to the right harnesses
• Update all the tests
  • need to contact other devs to fix their tests
• Mochitest monitoring to see if Special Powers will slow us down, need to check end to end time

Going Faster

• Getting rid of packed.js would shave ~5 mins from test run of mochitests. We only use a small bit of it, we don't need to load all.
• Write a Mochitest profiling test suite so we can run to profile our progress.
• Is our JS webserver a bottleneck? Should we switch to another server? Suggestions: nginx, lighttpd, etc.
  • Profiling: Run a suite that doesn't use .sjs or special content headers and test against current httpd.js server and the server we're testing
• There should be profilers for each harness, but start with mochitests.
• Should we combine some Mochitest test files into one huge generated test instead of a directory of many test files so we can load the overhead once?