Auto-tools/Projects/Futurama/2012-11-01
From MozillaWiki
< Auto-tools | Projects | Futurama
t= Turn-around time and volume =
- Increase capacity with VMs for some suites
- Be smarter about which tests to run, depending on circumstances
- Bisect in the cloud
- More flexible job scheduling to better support testing of environment changes and new frameworks
- Deep investigation of current tests, looking for redundancy and ineffective tests
Contents
Reliability of results
- Orange quarantine
- Availability of full, end-to-end environments identical to production (VMs or reserved systems)
- Make all test runners use the same modules/methodology (e.g. mozbase, mach)
- Deep investigation of current tests to look for antipatterns
Workflow
- Bugzilla dashboards
- Better integration of Bugzilla with review/try/check-in/test results process
- TBPL to hold all results, not just those from buildbot
Ideas?
- We have ideas to work on all these problems, but we want YOUR feedback to help us prioritize and take suggestions for solutions
Problems from the Developers
- Number one is no neon support but that is coming (panda boards)
- Windows 8
- Having VMs would be nice - things that people could download so they could repro things
- Download a VM with talos env on it so you can download it and reproduce the issue that talos hit
- Have a VM with as close to a configuration to what's in production so we can run it similarly to the production environment. The goal isn't to reporoduce the same numbers, but to reporoduce the regression.
- Download a VM with talos env on it so you can download it and reproduce the issue that talos hit
- Ability to run tests in parallel -- we want that for running locally.
- reftests should be possible to parallelize they could be the next step after xpcshell (reftests that don't require focus)
- jsreftests is another good candidate for this
- perhaps a method of attack: xpcshell -> jsreftest -> crashtest -> reftest
Workflow issues
- Autoland is barely running - it's not something that is being worked on right now.
- Orange factor needs an owner to drive the number down
- Code coverage -- taking that further.
- jcranmer has a setup that works on try. It's still a bit hackish but it is working, so that's a beginning. With more polish might be useful. Should be something to investigate with it.
- http://people.mozilla.org/~choller/firefox/coverage/
- ASAN builds - decoder has a way to run ASAN builds, it would be good to get nightly builds with it. So that people can download them and use them. Mostly to run locally.
Bugzilla
- showed bugzilla dashboards - you can see them on https://bugzilla.allizom.org
- The new skin is in the Preferences called "mozilla" skin drop down list at https://bugzilla-stage-tip.mozilla.org
- Making bugzilla faster
- There are changes to the backend to help with this:
- Better backend hardware
- Fixing search table inefficiencies
- try to reduce the unnecessary load on bugzilla from the REST API
- There are changes to the backend to help with this:
- XUL unit test
- Integration of Gtest, helping make it easier to write tests for pieces that live inside of libXUL
- Needs more work and investigation
- Improving quality of results
- i.e. adding profiling with talos
- reduces requrements for quick turn around time because you have enough data from one push and you don't need to push again to get more data.
- The more information we can provide the better - we show backtraces when we crash that's useful.
- But if we had profiling stats when we crash
- It would be nice to have tests for some of this low level stuff - someone broke compilation error reporting on windows recently and that was difficult to understand and fix. This could be added to the "fast test suite" idea
Workflow Suggestions
- Ability to work in git locally and have a gitserver system that takes your pushes and then pushes into hg.
- Someone has code for this -- follow up with Ehsan.
- Need some method for checking in someting "whenever" - perhaps autoland can do that? For patches that are not time sensitive
- Once the patch is r+ the patch should just land and the test results should show up in bugzilla automatically (the chrome/webkit automation story)
Need better docs/blogs about where things are
- what is mozbase? is it in mozcentral? where is the code?
- how do I do it?
- we need more links to the code from the running applications- make it very obvious.
- (SEO!) ensure our stuff shows up on google
- We should get rid of out of date crap - i.e. standalone talos.
- We should keep the projects page up to date and make it extremely obvious.
- Just delete old/out of date information. If people really want to see history the wiki has a feature to let you view it.
- tbpl results on bug pages