index | next mtg »

Test Development, Automation, Infrastructure Discussion - Dec 18th, 2006

Attendees

• alice, rob, bc, coop, tracy, tony, timr, juanb, carsten (tomcat)

Agenda

• Status/ideas for buildbot
• Plans/ideas for test extension
• Extending search engine/trademark verifiers (search enginers, rss feeds, search strings, )
• Next steps for collecting /reporting results

Discussion

Status/ideas for buildbot

• Start integrating JS tests
• [bc] simplifying tree, test invocation scripts. Should work for all platforms
• SSH working
• getting VNC/RDC running
• reporting status on tinderboxes
  • every build triggers test
• Older branches might show progressively more failures
• [robcee] Initially testing against trunk
• [bc] JS on all three branches
  • 1.8, 1.8.0, 1.9
  • Why not buildbot for all branches?
• [bc] Tests take different amounts of time to run
  • All may not be appropriate to run per build interation
  • JS tests take a couple hours to run - once per day
  • [robcee] Main buildbot tests now - RefTests - take 10-15 seconds
  • Fast cycles tests - run per build
    • one VM per platform
  • Long running tests - once per night
• Build time
  • [robcee] take 8 minute - make-depend build
  • [bc] should probably do make-clobber builds - make-depend is not reliable
• Buildbot should support debug builds, waytoomuch GC builds, or other special builds
• Bob's tools:
  • Steps: checkout, clean (clobber), build
  • Parameters: for optomized or debug build, etc
• Build FFx, TBird, XULRunner for all branches
• Buildbot master is on private network
  • Reports to tinderbox - public visibility
  • Impacts for sec tests on reporting?
• Need a couple xservs - RefTest and JS tests <<== AI timr
  • Mac Intel xservers - to replace the minis - for the CoLo
• Run 1.8 on one machine and 1.8.0 and 1.9 on another machine
  • So they all complete within a day

Next steps for collecting /reporting results

• Litmus is not necessarily the best place for results
• Look at what buildbot displays
  • easy to extend
• Long running tests can still be run and reported with one of the late night builds
• Litmus advanced query would be useful for future data mining and non-tinderbox oriented reporting

Plans/ideas for test extension

• New tool available from Seneca College
  • Developed by David Hamp-Gonsalves
  • See Seneca College Project - Buggy Bar
  • Grabs next unconfirmed bug from bugzilla
  • Good as long as bug is formatted by the bug submitter tool
  • Waiting for AMO group to sign off on Buggy Bar
  • Updates bug with confirmed status and comments
    • Add to comment: "confirmed via buggy bar" or something similar so we can see how may bugs are being processed by this extension?
    • Tracy to check if David can add unique text in comment so we can see how this is working. <<== AI tracy
  • He has offered to create a litmus extension
    • tracy - to lead this <<== AI tracy

• RefTests - not good candidates for test ext

• Other testing extensions [bob]
  • Spider - general purpose
    • Fire off JS events
    • See Bob's blog - about attempting to drive browser from KB, Mouse events, fill in forms, etc.
    • Spider is currently very user un-friendly but a lot of potential
    • Next step: discuss in weekly JS test automation mtg

Extending search engine/trademark verifiers (search enginers, rss feeds, search strings, etc)

• Didn't get to this