Goals

The problem is that right now our end to end time from checkin to all tests done is around 6-8 hours. That's wasting a lot of developer time and productivity. The Goal of this project is to increase developer productivity by both minimizing time spent waiting for results and by making the "waiting for results" less of a time consuming task in itself.

Most of the bugs are concerned with performance optimization. Productivity bugs that do not affect performance optimization are noted with (loosely related) in their descriptions.

The priority of the tasks below will be determined by the data we have on the gains associated with completing them. We'll evaluate all proposals in terms of that data. If you have a proposal, add it to the proposals section below (hopefully with links to data)

The Goals:

The overall target we're shooting for is 2 hours from checkin to all tests done.
Get time from build-available to all tests done to <= 30 minutes
Get per-checkin build time to under 1.5 hours for all platforms
There are other small advances in related bugs that can help free-up developer time.
Measure and track our progress

Non-goals:

Do not boil the ocean - we need to be selective
Ensure amount of work is worth the performance payoff
Buying more minis - we cannot buy more minis at this time (our rev is discontinued)

Current activities

Bug	Description	Owner	Status
bug 669930	Set up metrics to quantify our current end to end time for tracking	catlee	In Design
bug 658313	Make on-change builds into non PGO, only do PGO on an as available basis	jhford	In progress
bug 669988	Investigate if we can do 64 bit only 10.6 builds per-checkin (how many 10.5 failures are there?)	jgriffin	In progress
bug 659328	Merge talos suites to amortize setup costs	armenzg	Has Patches
n/a	Profile Test suites to find individual tests are higher than average time to complete (so devs can fix them)	ctalbert & jgriffin	First analysis of mochitest-plain complete
bug 417044	Perform Universal binary builds as single-pass	:joey	In Progress
bug 659222	Branch prioritization broken for test jobs	catlee	In Progress
bug 661656	Improving network between colos	zandr	In Progress
bug 586418	Merge unit test suites	UNOWNED	Cannot finish until bug 594415 completed
bug 661649	Deploy talos pageset to all slaves to avoid download/unpack	UNOWNED	Needs Owner
bug 561235	Use server to deal with getting stacks for crashes (no download of symbols onto slave)	ted	Needs next steps defined
bug 561754	Create server for processing minidumps	catlee	In progress
bug 660124	Turn off txul and ts in favor of their -paint varieties	armenzg	ctalbert helping drive
??	Use sdwilsh's notification bot to help devs more easily watch the tree	sdwilsh?	Needs next steps, ctalbert to follow up
bug 558180	Check mozconfigs into source to avoid buildconfig checkout	catlee	In Progress
??	Profile mochitest plain	jgriffin	In Progress
n/a	Get data on test run time for each of the moth individual tests	ctalbert	COMPLETE
??	Purchase new machines for win/linux builders that are not tied to aggressive obsolescence schedule	zandr	Underway, getting gear for lab testing
bug 623617	Reduce recursion in makefiles to improve build times	:joey	Proposed/In design

Long Term Ideas

bug 657738 - Automatically determine when oranges happen and auto-star them (no owner/decision on steps yet)
bug 630534 - (related) Host tree status outside of tinderbox
- This would help us move away from tinderbox and would free up many of the constraints we currently operate under
Analyze using VMs for "dial-up" capacity
- Would need engineering resources to debug oranges that occur only on vms
- Would need to run some low numbers of vms on an ongoing basis so that we continue to ensure they are providing results we can trust
- We may need to solve bug 617763 so that these machines can be kept up to date (whether they live in the build VPN or not, i.e. if we use EC2 or something like that...). Either way we must address the problem that they could be out of date between one dial-in and another dial-in.
Analyze methods of understanding and optimizing what we run on a per-checkin basis
- Test code related to the current patch?
- Run full tests only every x pushes or y time (whichever occurs first). Between those only run a set of tests related to the patch at hand. (Would need developer override and on-demand testing ability)
less frequent debug builds
use an optimized xpcshell + httpd.js to run debug tests (depends on bug 669953)

Data we need

Profiling on all major harnesses
How many 10.5-only test failures have there been? Do we need to run 10.5 tests on-checkin?
Where are we spending our time during build slave setup/teardown (catlee owns)
Gather data on specific long-running tests on mochitest & mochitest other
- Jmaher reported to newsgroup some data on mochitest-plain. Moth is still TBD

Data we have

Test Setup versus Run Time (thanks to armenzg)
Data from Orange factor detailing 31,000 Mochitest other test runs. (Thanks to jgriffin)
- Here is the easy-to-parse version, showing we have something seriously wrong with mochitest-browser-chrome debug. (opt is no light weight either). Also, the ipcplugin test varies wildly on mac and could possibly improved for some minor wins.

People

Chris AtLee (:catlee)
Armen Zambrano (:armenzg)
Joey Armstrong (:joey)
Clint Talbert (:ctalbert)
Some as-yet undefined set of the A-team (probably Ted, Jmaher and Harth, but I'll put out the call today)

Definitions

end-to-end time: The time between a change being pushed to HG and all builds and test results being available
per-checkin build: Opt and debug builds that are triggered by a new checkin
on-change build: same as per-checkin
as-available run: Run the given test/build task only as fast as you can complete the previous run. Run the previous run, complete, then start the next task with the latest checkin.

Proposals for consideration

Please add proposals here...

Meetings

2011-07-07

ReleaseEngineering/BuildFaster

Contents