Engineering:Infrastructure

This list is a work-in-progress list of improvements to our engineering infrastructure, in no particular order.

• Tinderbox+pushlog
  • Bring it in house
  • Enhance as needed
  • Tree status
  • Patch results (easily identifying the run performed against a specific patch)
  • Generalize so that we can leverage this across development projects (i.e. tracemonkey, fennec, thunderbird, etc...)
• Bugzilla
  • Identify desired workflows and the real underlying problems to be solved here rather than getting too bogged down in the next feature desired
  • Data extraction
  • Customized usage based upon task
  • Integration with other systems
  • Component watching
  • QA Contact back to original intent
  • Bug caused by (like depends on & blocks)
  • Bug triage workflow
  • Tagging bugs
  • Accepts review requests/Accepts super-review requests and associated restrictions
  • Integration with bug check-ins (change states?)
  • Lists of favorite bugs
  • Making bz branch/platform aware (fixed here but not there)
• Sheriff tools
  • Tools to facilitate performing the Sheriff duties (tbd)
  • Need a way to see who the current Sheriff is.
• Test infrastructure
  • Is python a limiter for new platforms
  • Do we need so many test frameworks?
  • Server based test framework
    • Small client/driver on SUT
    • Tests can be executed from server
    • Results gathered and stored at server for further analysis and integration
    • Benefit small footprint on the SUT
    • Explore server side performance metrics (what can be done)
  • Mobile device testing
    • Run UI tests in simulator?
    • Run others in simulator?
    • Do all mobile devices support extensions?
    • Do all mobile devices support AJAX?
    • Make talos mobile friendly
  • Implement rate controlled network tests to help identify performance on less than Lan based connections
  • Implement torture test environment to improve stability testing
    • environment to test lower limits of machine specs (disk space, memory, etc...)
    • environment to test error recovery when system resources are exhausted
  • Improve OS coverage for testing
• 3rd party package dependency tracking
• Mercurial WebViewer
  • Follow history through moves/renames/re basing
  • Search changes by author/subtree
• Build system
  • Currently a placeholder until I can meet with John O'Duinn
• Crash reporting
  • Crash reporting on ARM
  • Crash reporting for hangs/OOM situations - bug 429592, bug 493779
  • Output last known memory/CPU usage for the process at the time of the report (if possible)
  • Improve the ability to get specific URLs from the crash reporter database, know that we have the URL that we actually crashed on.
  • Improve and fix situations where crash reporter doesn't activate
• Plug-in tests
  • Improve test plug-in
  • Create a plugin test harness for driving plug-ins
• Extension tests
  • Create/improve test extension
  • Create a plugin test harness for driving extensions
• Test coverage
  • Discover/Report/Improve Test coverage for features
  • Code coverage
    • Research and implement code coverage measurement
    • Implement tests to fill the gaps
• JS training for QA and newbies
• DXR
  • Get DXR up and running in house
  • retire MXR
• Dashboard
  • Relevant metrics
  • Streamline inspecting what's going on
• Orange trees due to flaky tests
  • Identify, analyze, and fix
• Automated Test execution
  • Speed improvements - How long do they take now?
  • Are we running the right ones at the right times?
  • Are we testing on the right platforms?
  • Try servers - speed
  • Local builds - speed
  • make test execution scalable
    • reduce/eliminate the interdependence of tests
    • create "buckets" of tests to be run at specific times in the development cycle (i.e. daily, weekly, once per phase, etc...)
• Simplify and speed-up running the test suites locally
• Wiki improvements
  • Integration with Bugzilla (ex. enter bug id turn into a link)
• Valgrind for linux build/test runs
• Facilitate code reviews through the use of tools
• Webdev needs a way to quickly debug production issues ("... wish we'd pull machines OOR {out of rotation} and temporarily give access to devs in order to diagnose")
• Monitoring test systems for hangs (mobile?)
• Improve speed of readying new platforms.
• l10n placeholder (I'm meeting with them on wednesday)
• Tying together bugzilla, hendrix, reporter, breakpad, fuzzers, SUMO and various web based data sources (twitter, blogs, etc...) into some sort of coherent information.