Details

July 7, 2011 @ 8am PDT / 11am EDT

dial-in x92 304#

Backchannel #buildfaster on irc.mozilla.org

etherpad notes

Agenda

Review current activities

• See list of current activities
• Is everything up to date?
• Tag bugs with [buildfaster:p1], [buildfaster:p2], etc.
• What do we do with jgriffin's test analysis?

Notes

Goal: Speed up turn around time from patch submission to getting test results back

• Right now: we're about 8 hours
• Ideal goal: 2 hours - end to end.

• Metrics
  • catlee investigating
  • ted and ctalbert mention changes to mochitest to help measure end-to-end (measure harness overhead without noise from new tests being added)
  • armenzg had a spreadsheet with metrics and that could be automated
    • setup time, run time
    • for reference https://spreadsheets.google.com/spreadsheet/ccc?key=tNfPnuOZvLcbnItGP6gCYSQ&authkey=COzKr8wL#gid=0
  • "are we building fast yet?"

• onchange non-PGO builds
  • jhford owns this

• Investigate if we can do 10.6 only 64-bit builds onchange
  • jgriffin to analyze orangefactor data for 10.5-only failures
  • mac is incredibly slow (universal builds)
  • surprisingly linux is quite slow too with PGO
  • mac: treat like PGO where 'normal case' is to do a single arch build and then nightly do universal

• Determine which test suites can be merged
  • armenzg owns this

• Profile test suites to find individual slow tests
  • jmaher in progress
  • Some tests are slow due to Mochitest UI (big HTML table)
    • Want to cleanup Mochitest UI for other reasons
  • jgriffin getting some data out of orangefactor as well
  • identify slow tests on a weekly basis?

• Single-pass universal mac builds
  • longer-term goal, but worthwhile

• Split config/rules.mk
  • joey?

• (not currently listed) derecursify the build system
  • khuey has a WIP patch in https://bugzilla.mozilla.org/show_bug.cgi?id=623617

• Branch prioritization broken for test jobs
  • jobs on m-c can get handled before jobs on try or project branches
  • Need to properly prioritize branches
  • catlee has a fix

• Improve network between colos
  • blocked on firewall work & downtime, in progress
  • ETA next couple of weeks

• Merge some test suites
  • have been waiting on a TBPL bug to display merged suites separately
    • maybe this isn't critical? armenzg to gauge developer feedback
  • sfink suggests: would be nice to have a way to get commands to reproduce test runs locally

• Deploy Talos pageset to machines to avoid downloading
  • catlee says there were problems with this when deploying Tp4 (HFS sucking on OS X)
  • can we rsync pagesets over?
    • only one disk in the minis
    • doesn't work currently, test infra puts profile inside pageset dir
  • pageset doesn't change often
  • media on test packages barely change, can we rsync the differences somehow?

• Server for crash processing from test slaves
  • implemented for unittests
  • needs implemented for Talos: https://bugzilla.mozilla.org/show_bug.cgi?id=561235
  • blocked on getting CGI set up somewhere: https://bugzilla.mozilla.org/show_bug.cgi?id=561754
  • should save overhead of downloading symbols for every single run

• Reallocate Win7x64 slaves to support other platforms
  • Slight bump in capacity
  • Will need them back when we want to support x64 (next two weeks)

• Turn off Txul and Ts
  • blocked on reaching consensus
  • should just go ahead and apologize later

• sdwilsh's notification bot
  • IRC bot to notify of build results
  • other discussion about notifying, regression finder emails etc

• check mozconfigs into source
  • avoid buildbot-configs checkout
  • catlee has patches up

• Profile mochitest-plain
  • ctalbert covered some of this
  • See Metrics above

• Run debug builds "as available" when we hit capacity
  • lose assertions, leak checking
  • current debug builds are not optimized (makes them much slower)
  • debug builds enable tracemalloc (does this make them slower?)

• Use optimized xpcshell to run httpd.js for debug builds
  • debug xpcshell is much slower
  • packaging is a pain
  • xpcshell doesn't change very often
  • should look again if we can enable optimize on debug builds, maybe not a problem

• Change from Mac Minis to non-Apple hardware for non-Mac test slaves
  • Identical hardware cross-platform no longer a requirement
  • easier to manage, easier to procure

Review [buildfaster:p1] bugs

• Covered by review above

Questions

• run 10.5 debug builds on 10.6 machines? It increases capacity of 10.6 pool. Low low priority. Loosely related.
• shall we break up the current activities section into bug being worked on and bugs in the back burner?

Action items for next time

• catlee to file bug on getting metrics on pieces of end-to-end time. DONE bug 669930
• ctalbert to follow up on slow tests
• ctalbert to file bug on getting commands to reproduce test runs on developer machine (give URLs to packages to download + steps to run)
• ted to file a bug on investigating rsyncing of test package contents DONE bug 669952
• catlee to get IT bug filed for minidump processing DONE bug 669947
• ctalbert to follow up on sdwilsh'es tree status
• ted to file a bug on enabling optimize on debug builds DONE bug 669953
• ted to ask dbaron about tracemalloc on debug builds, is it a perf hit by default?

Set date for next meeting

• ~ 2 weeks from now. Will find day-of-week and time that works for most people.