These are the raw notes from the developer discussion we had in toronto

Turn-around time and volume

• Increase capacity with VMs for some suites
• Be smarter about which tests to run, depending on circumstances
  • Bisect in the cloud
• More flexible job scheduling to better support testing of environment changes and new frameworks
• Deep investigation of current tests, looking for redundancy and ineffective tests

Reliability of results

• Orange quarantine
• Availability of full, end-to-end environments identical to production (VMs or reserved systems)
• Make all test runners use the same modules/methodology (e.g. mozbase, mach)
• Deep investigation of current tests to look for antipatterns

Workflow

• Bugzilla dashboards
• Better integration of Bugzilla with review/try/check-in/test results process
• TBPL to hold all results, not just those from buildbot

Ideas?

• We have ideas to work on all these problems, but we want YOUR feedback to help us prioritize and take suggestions for solutions

Problems from the Developers

• Number one is no neon support but that is coming (panda boards)
• Windows 8
• Having VMs would be nice - things that people could download so they could repro things
  • Download a VM with talos env on it so you can download it and reproduce the issue that talos hit
    • Have a VM with as close to a configuration to what's in production so we can run it similarly to the production environment. The goal isn't to reporoduce the same numbers, but to reporoduce the regression.
• Ability to run tests in parallel -- we want that for running locally.
  • reftests should be possible to parallelize they could be the next step after xpcshell (reftests that don't require focus)
  • jsreftests is another good candidate for this
  • perhaps a method of attack: xpcshell -> jsreftest -> crashtest -> reftest

Workflow issues

• Autoland is barely running - it's not something that is being worked on right now.
  • Orange factor needs an owner to drive the number down
• Code coverage -- taking that further.
  • jcranmer has a setup that works on try. It's still a bit hackish but it is working, so that's a beginning. With more polish might be useful. Should be something to investigate with it.
  • http://people.mozilla.org/~choller/firefox/coverage/
• ASAN builds - decoder has a way to run ASAN builds, it would be good to get nightly builds with it. So that people can download them and use them. Mostly to run locally.

Bugzilla

• showed bugzilla dashboards - you can see them on https://bugzilla.allizom.org
• The new skin is in the Preferences called "mozilla" skin drop down list at https://bugzilla-stage-tip.mozilla.org
• Making bugzilla faster
  • There are changes to the backend to help with this:
    • Better backend hardware
    • Fixing search table inefficiencies
    • try to reduce the unnecessary load on bugzilla from the REST API
• XUL unit test
  • Integration of Gtest, helping make it easier to write tests for pieces that live inside of libXUL
  • Needs more work and investigation
• Improving quality of results
  • i.e. adding profiling with talos
  • reduces requrements for quick turn around time because you have enough data from one push and you don't need to push again to get more data.
  • The more information we can provide the better - we show backtraces when we crash that's useful.
  • But if we had profiling stats when we crash
  • It would be nice to have tests for some of this low level stuff - someone broke compilation error reporting on windows recently and that was difficult to understand and fix. This could be added to the "fast test suite" idea

Workflow Suggestions

• Ability to work in git locally and have a gitserver system that takes your pushes and then pushes into hg.
  • Someone has code for this -- follow up with Ehsan.
• Need some method for checking in someting "whenever" - perhaps autoland can do that? For patches that are not time sensitive
• Once the patch is r+ the patch should just land and the test results should show up in bugzilla automatically (the chrome/webkit automation story)

Need better docs/blogs about where things are

• what is mozbase? is it in mozcentral? where is the code?
• how do I do it?
• we need more links to the code from the running applications- make it very obvious.
• (SEO!) ensure our stuff shows up on google
• We should get rid of out of date crap - i.e. standalone talos.
• We should keep the projects page up to date and make it extremely obvious.
  • Just delete old/out of date information. If people really want to see history the wiki has a feature to let you view it.
• tbpl results on bug pages