Test Development, Automation, Infrastructure Discussion - Dec 18th, 2006

Attendees

Agenda

Status/ideas for buildbot
Plans/ideas for test extension
Extending search engine/trademark verifiers (search enginers, rss feeds, search strings, )
Next steps for collecting /reporting results

Start integrating JS tests
[bc] simplifying tree, test invocation scripts. Should work for all platforms
SSH working
getting VNC/RDC running
reporting status on tinderboxes
- every build triggers test
Older branches might show progressively more failures
[robcee] Initially testing against trunk
[bc] JS on all three branches
- 1.8, 1.8.0, 1.9
- Why not buildbot for all branches?
[bc] Tests take different amounts of time to run
- All may not be appropriate to run per build interation
- JS tests take a couple hours to run - once per day
- [robcee] Main buildbot tests now - RefTests - take 10-15 seconds
- Fast cycles tests - run per build
  - one VM per platform
- Long running tests - once per night
Build time
- [robcee] take 8 minute - make-depend build
- [bc] should probably do make-clobber builds - make-depend is not reliable
Buildbot should support debug builds, waytoomuch GC builds, or other special builds
Bob's tools:
- Steps: checkout, clean (clobber), build
- Parameters: for optomized or debug build, etc
Build FFx, TBird, XULRunner for all branches
Buildbot master is on private network
- Reports to tinderbox - public visibility
- Impacts for sec tests on reporting?
Need a couple xservs - RefTest and JS tests <<== AI timr
- Mac Intel xservers - to replace the minis - for the CoLo
Run 1.8 on one machine and 1.8.0 and 1.9 on another machine
- So they all complete within a day

Litmus is not necessarily the best place for results
Look at what buildbot displays
- easy to extend
Long running tests can still be run and reported with one of the late night builds
Litmus advanced query would be useful for future data mining and non-tinderbox oriented reporting