QA/TDAI/Meetings/2006-12-18: Difference between revisions
Jump to navigation
Jump to search
| Line 51: | Line 51: | ||
* Look at what buildbot displays | * Look at what buildbot displays | ||
** easy to extend | ** easy to extend | ||
* Long running tests can still be run and reported with one of | * Long running tests can still be run and reported with one of the late night builds | ||
* Litmus advanced query would be useful for future data mining and non-tinderbox oriented reporting | * Litmus advanced query would be useful for future data mining and non-tinderbox oriented reporting | ||
== Plans/ideas for test extension == | == Plans/ideas for test extension == | ||
Revision as of 02:04, 19 December 2006
- Test Development, Automation, Infrastructure Discussion
- Dec 18th, 2006
Attendees
- alice, rob, bc, coop, tracy, tony, timr, juanb, carsten (tomcat)
Agenda
- Status/ideas for buildbot
- Plans/ideas for test extension
- Extending search engine/trademark verifiers (search enginers, rss feeds, search strings, )
- Next steps for collecting /reporting results
Discussion
Status/ideas for buildbot
- Start integrating JS tests
- [bc] simplifying tree, test invocation scripts. Should work for all platforms
- SSH working
- getting VNC/RDC running
- reporting status on tinderboxes
- every build triggers test
- Older branches might show progressively more failures
- [robcee] Initially testing against trunk
- [bc] JS on all three branches
- 1.8, 1.8.0, 1.9
- Why not buildbot for all branches?
- [bc] Tests take different amounts of time to run
- All may not be appropriate to run per build interation
- JS tests take a couple hours to run - once per day
- [robcee] Main buildbot tests now - RefTests - take 10-15 seconds
- Fast cycles tests - run per build
- one VM per platform
- Long running tests - once per night
- Build time
- [robcee] take 8 minute - make-depend build
- [bc] should probably do make-clobber builds - make-depend is not reliable
- Buildbot should support debug builds, waytoomuch GC builds, or other special builds
- Bob's tools:
- Steps: checkout, clean (clobber), build
- Parameters: for optomized or debug build, etc
- Build FFx, TBird, XULRunner for all branches
- Buildbot master is on private network
- Reports to tinderbox - public visibility
- Impacts for sec tests on reporting?
- Need a couple xservs - RefTest and JS tests <<== AI timr
- Mac Intel xservers - to replace the minis - for the CoLo
- Run 1.8 on one machine and 1.8.0 and 1.9 on another machine
- So they all complete within a day
Next steps for collecting /reporting results
- Litmus is not necessarily the best place for results
- Look at what buildbot displays
- easy to extend
- Long running tests can still be run and reported with one of the late night builds
- Litmus advanced query would be useful for future data mining and non-tinderbox oriented reporting
Plans/ideas for test extension
- Seneca - David Hamp-Gonsalves
- See Seneca College Project - Buggy Bar
- grabs next unconfirmed bug from bugzilla
- Good as long as bug is formatted by the bug submitter tool
- Updated bug with confirmed status and comments
- Add to comment: confirmed via buggy bar
- Tracy to check if he can add unique text in comment so we can see how this is working. <<== AI tracy
- He has offered to create a litmus extension
- tracy - to lead this <<== AI tracy
- Waiting for AMO group to sign off on Buggy Bar
- RefTests - not good candidates for test ext
- Other testing extensions [bob]
- Spider - general purpose
- Fire off JS events
- See blog - about attempting to driver browser from KB, Mouse events, fill in forms, etc.
- Spider is currently very user un-friendly but a lot of potential
- Next step: discuss on weekly JS test automation mtg
- Spider - general purpose
Extending search engine/trademark verifiers (search enginers, rss feeds, search strings, etc)
- Didn't get to this