The problem is that right now our end to end time from checkin to all tests done is around 6-8 hours. That's wasting a lot of developer time and productivity. The Goal of this project is to increase developer productivity by both minimizing time spent waiting for results and by making the "waiting for results" less of a time consuming task in itself.
Most of the bugs are concerned with performance optimization. Productivity bugs that do not affect performance optimization are noted with (loosely related) in their descriptions.
The priority of the tasks below will be determined by the data we have on the gains associated with completing them. We'll evaluate all proposals in terms of that data. If you have a proposal, add it to the proposals section below (hopefully with links to data)
- The overall target we're shooting for is 2 hours from checkin to all tests done.
- Get time from build-available to all tests done to <= 30 minutes
- Get per-checkin build time to under 1.5 hours for all platforms
- There are other small advances in related bugs that can help free-up developer time.
- Measure and track our progress
- Do not boil the ocean - we need to be selective
- Ensure amount of work is worth the performance payoff
- Buying more minis - we cannot buy more minis at this time (our rev is discontinued)
Ensure any bugs that are filed for this effort have [buildfaster:pN] in the whiteboard. Where N is some number 1-5. P1 is highest priority, P5 is the lowest.
If you'd like to file a bug for something you'd like to see included in this effort, you can use [buildfaster:?] in the whiteboard.
The table below is for current tasks that are not tracked in bugzilla.
|bug 772446||Move all Linux builds to AWS||jhopkins||All Firefox builds are there now, just waiting on Thunderbird to catch-up.|
|bug 758624||Purchase new machines for win/linux testers that are not tied to aggressive obsolescence schedule||coop||Underway. Various test OSes are being setup and validated.|
|bug 784913||migrate linux xpcshell testsuite to AWS||jmaher||Blocked on panda work|
|bug 784913||migrate linux xpcshell testsuite to AWS||unowned||Needs an owner|
Long Term Ideas
- bug 657738 - Automatically determine when oranges happen and auto-star them (no owner/decision on steps yet)
- Windows builds on AWS are still too slow, but we can revisit this regularly
- Cross-compile Mac builds on AWS
- after legal review, *yes* we think we can do this. Need to find a developer owner to give this a try.
- bug 772579 - Can we run unittests on AWS for Linux (and eventually Windows)?
- Analyze methods of understanding and optimizing what we run on a per-checkin basis
- Test code related to the current patch?
- Run full tests only every x pushes or y time (whichever occurs first). Between those only run a set of tests related to the patch at hand. (Would need developer override and on-demand testing ability)
- less frequent debug builds
- use an optimized xpcshell + httpd.js to run debug tests (depends on
Data we need
- Profiling on all major harnesses
- Currently profiling is being done on mochitest harness and tracked here. Be sure to select either "opt" or "debug" builds, since choosing "any" is probably not what you want (profiling times differ greatly between the two).
- How many 10.5-only test failures have there been? Do we need to run 10.5 tests on-checkin?
- Where are we spending our time during build slave setup/teardown (catlee owns)
- Gather data on specific long-running tests on mochitest & mochitest other
- Jmaher reported to newsgroup some data on mochitest-plain. Moth is still TBD
Data we have
- Test Setup versus Run Time (thanks to armenzg)
- Data from Orange factor detailing 31,000 Mochitest other test runs. (Thanks to jgriffin)
- Here is the easy-to-parse version, showing we have something seriously wrong with mochitest-browser-chrome debug. (opt is no light weight either). Also, the ipcplugin test varies wildly on mac and could possibly improved for some minor wins.
- Chris AtLee (:catlee)
- Armen Zambrano (:armenzg)
- Joey Armstrong (:joey)
- Clint Talbert (:ctalbert)
- Will Lachance (:wlach)
- Ted Mielczarek (:ted)
- Chris Cooper (:coop)
- end-to-end time
- The time between a change being pushed to HG and all builds and test results being available
- per-checkin build
- Opt and debug builds that are triggered by a new checkin
- on-change build
- same as per-checkin
- as-available run
- Run the given test/build task only as fast as you can complete the previous run. Run the previous run, complete, then start the next task with the latest checkin.
Proposals for consideration
Please add proposals here...