Confirmed users
3,816
edits
ReleaseEngineering/BuildFaster: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
|||
| Line 1: | Line 1: | ||
= Goals = | = Goals = | ||
The problem is that right now our end to end time from checkin to all tests done is around 6-8 hours. That's wasting a lot of developer time and productivity. The Goal of this project is to increase developer productivity by both minimizing time spent waiting for results and by making the "waiting for results" less of a time consuming task in itself. | |||
Most of the bugs are concerned with performance optimization. Productivity bugs that do not affect performance optimization are noted with (loosely related) in their descriptions. | |||
The priority of the tasks below will be determined by the data we have on the gains associated with completing them. We'll evaluate all proposals in terms of that data. If you have a proposal, add it to the proposals section below (hopefully with links to data) | |||
Specifically The Goals: | |||
* The overall target we're shooting for is 2 hours from checkin to all tests done. | |||
* Get time from build-available to all tests done to <= 30 minutes | * Get time from build-available to all tests done to <= 30 minutes | ||
* Get per-checkin build time to under 2 hours for all platforms | * Get per-checkin build time to under 2 hours for all platforms | ||
* There are other small advances in related bugs that can help free-up developer time. | |||
* Measure and track our progress | |||
Non-goals: | |||
* Do not boil the ocean - we need to be selective | |||
* Ensure amount of work is worth the performance payoff | |||
* Buying more minis - we cannot buy more minis at this time (our rev is discontinued) | |||
= Current activities = | = Current activities = | ||
| Line 16: | Line 31: | ||
{| style="width: 100%" class="fullwidth-table sortable" | {| style="width: 100%" class="fullwidth-table sortable" | ||
|- | |- | ||
| style="background:#DDD;" width= | | style="background:#DDD;" width=15%" | '''Bug''' | ||
| style="background:#DDD" | '''Description''' | | style="background:#DDD" | '''Description''' | ||
| style="background:#DDD" | '''Owner''' | | style="background:#DDD" | '''Owner''' | ||
| Line 70: | Line 85: | ||
| :salbiz | | :salbiz | ||
| {{StatusBlocked|status=Selective unpacking landed, cannot finish until {{bug|594415}} completed}} | | {{StatusBlocked|status=Selective unpacking landed, cannot finish until {{bug|594415}} completed}} | ||
|- | |||
| {{bug|594415}} | |||
| Change TBPL to report on individual test suites not buildbot jobs | |||
| Swatinem | |||
| {{StatusAtRisk|status=Patches complete, needs to be unstuck, ctalbert driving}} | |||
|- | |||
| {{bug|661649}} | |||
| Deploy talos pageset to all slaves to avoid download/unpack | |||
| '''UNOWNED''' | |||
| {{StatusAtRisk|status=Needs Owner}} | |||
|- | |||
| {{bug|561235}} | |||
| Use server to deal with getting stacks for crashes (no download of symbols onto slave) | |||
| ted | |||
| {{StatusHealthy|status=Needs next steps defined}} | |||
|- | |||
| {{bug|561754}} | |||
| Create server for processing minidumps | |||
| ted | |||
| {{StatusHealthy|status=Complete?}} | |||
|- | |||
| {{bug|659118}} | |||
| Reuse win7 64bit machines for supporting other OS's | |||
| armenzg | |||
| {{StatusHealthy|status=In Progress}} | |||
|- | |||
| {{bug|660124}} | |||
| Turn off txul and ts in favor of their -paint varieties | |||
| jmaher | |||
| {{StatusBlocked|status=Blocked on reaching consensus, ctalbert helping drive}} | |||
|- | |||
| ?? | |||
| Use sdwilsh's notification bot to help devs more easily watch the tree | |||
| sdwilsh? | |||
| {{StatusHealthy|status=Needs next steps, ctalbert to follow up}} | |||
|- | |||
| {{bug|558180}} | |||
| Check mozconfigs into source to avoid buildconfig checkout | |||
| catlee | |||
| {{StatusHealthy|status=In Progress}} | |||
|- | |||
| ?? | |||
| Profile mochitest plain | |||
| '''UNOWNED''' | |||
| {{StatusAtRisk|status=Needs Owner}} | |||
|- | |||
| n/a | |||
| Get data on test run time for each of the moth individual tests | |||
| ctalbert | |||
| {{StatusHealthy|status=In Progress}} | |||
|- | |||
| {{bug|528231}} | |||
| (loosely related) Get debug symbols for unit test reference platform into breakpad | |||
| nthomas | |||
| {{StatusBlocked|status=Blocked on nthomas's availability}} | |||
|- | |||
| {{bug|571886}} | |||
| (loosely related) Create calendar tool to aid Firefox sheriffs | |||
| harth and/or webdev | |||
| {{StatusBlocked|status=Blocked on harth's availability}} | |||
|- | |||
| ?? | |||
| Run debug builds on an "as available" basis once we hit capacity limits | |||
| catlee | |||
| {{StatusHealthy|status=Proposed/In design}} | |||
|- | |||
| ?? | |||
| Use a packaged, release build for httpd.js when running tests (improve httpd startup cost) | |||
| ctalbert | |||
| {{StatusHealthy|status=Proposed/Investigation}} | |||
|- | |||
| ?? | |||
| Purchase new windows and linux machines for test slaves that are not tied to aggressive obsolescence schedule | |||
| zandr | |||
| {{StatusBlocked|status=Underway, but issues getting machines from 3rd party provider}} | |||
|} | |} | ||
== Long Term Ideas == | |||
* {{bug|657738}} - Automatically determine when oranges happen and auto-star them (no owner/decision on steps yet) | |||
* {{bug|630534}} - (related) Host tree status outside of tinderbox | |||
** This would help us move away from tinderbox and would free up many of the constraints we currently operate under | |||
* Analyze using VMs for "dial-up" capacity | |||
** Would need engineering resources to debug oranges that occur only on vms | |||
** Would need to run some low numbers of vms on an ongoing basis so that we continue to ensure they are providing results we can trust | |||
** We may need to solve {{bug|617763}} so that these machines can be kept up to date (whether they live in the build VPN or not, i.e. if we use EC2 or something like that...). Either way we must address the problem that they could be out of date between one dial-in and another dial-in. | |||
* Analyze methods of understanding and optimizing what we run on a per-checkin basis | |||
** Test code related to the current patch? | |||
** Run full tests only every x pushes or y time (whichever occurs first). Between those only run a set of tests related to the patch at hand. (Would need developer override and on-demand testing ability) | |||
= Data we need = | |||
* Profiling on all major harnesses | |||
* How many 10.5-only test failures have there been? Do we need to run 10.5 tests on-checkin? | |||
* What is the test timing for the suites contained in the moth harness? (ctalbert owns) | |||
* Where are we spending our time during build slave setup/teardown (catlee owns) | |||
* Gather data on specific long-running tests on mochitest & mochitest other | |||
** Jmaher reported to newsgroup some data on mochitest-plain. Moth is still TBD | |||
= Data we have = | |||
* [https://spreadsheets.google.com/spreadsheet/ccc?key=tNfPnuOZvLcbnItGP6gCYSQ&authkey=COzKr8wL#gid=0 Test Setup versus Run Time] | |||
= People = | = People = | ||
| Line 76: | Line 189: | ||
* Armen Zambrano (:armenzg) | * Armen Zambrano (:armenzg) | ||
* Joey Armstrong (:joey) | * Joey Armstrong (:joey) | ||
* Clint Talbert (:ctalbert) | |||
* Some as-yet undefined set of the A-team (probably Ted, Jmaher and Harth) | |||
= Definitions = | = Definitions = | ||
; end-to-end time: The time between a change being pushed to HG and all builds and test results being available | ; end-to-end time: The time between a change being pushed to HG and all builds and test results being available | ||
; per-checkin build: Opt and debug builds that are triggered by a new checkin | ; per-checkin build: Opt and debug builds that are triggered by a new checkin | ||
; on-change build: same as per-checkin | |||
; as-available run: Run the given test/build task only as fast as you can complete the previous run. Run the previous run, complete, then start the next task with the latest checkin. | |||
= Proposals for consideration = | |||
Please add proposals here... | |||