From MozillaWiki
< QA‎ | TDAI‎ | Meetings
Jump to: navigation, search

index | next mtg »

Test Development, Automation, Infrastructure Discussion - Dec 18th, 2006


  • alice, rob, bc, coop, tracy, tony, timr, juanb, carsten (tomcat)


  • Status/ideas for buildbot
  • Plans/ideas for test extension
  • Extending search engine/trademark verifiers (search enginers, rss feeds, search strings, )
  • Next steps for collecting /reporting results


Status/ideas for buildbot

  • Start integrating JS tests
  • [bc] simplifying tree, test invocation scripts. Should work for all platforms
  • SSH working
  • getting VNC/RDC running
  • reporting status on tinderboxes
    • every build triggers test
  • Older branches might show progressively more failures
  • [robcee] Initially testing against trunk
  • [bc] JS on all three branches
    • 1.8, 1.8.0, 1.9
    • Why not buildbot for all branches?
  • [bc] Tests take different amounts of time to run
    • All may not be appropriate to run per build interation
    • JS tests take a couple hours to run - once per day
    • [robcee] Main buildbot tests now - RefTests - take 10-15 seconds
    • Fast cycles tests - run per build
      • one VM per platform
    • Long running tests - once per night
  • Build time
    • [robcee] take 8 minute - make-depend build
    • [bc] should probably do make-clobber builds - make-depend is not reliable
  • Buildbot should support debug builds, waytoomuch GC builds, or other special builds
  • Bob's tools:
    • Steps: checkout, clean (clobber), build
    • Parameters: for optomized or debug build, etc
  • Build FFx, TBird, XULRunner for all branches
  • Buildbot master is on private network
    • Reports to tinderbox - public visibility
    • Impacts for sec tests on reporting?
  • Need a couple xservs - RefTest and JS tests <<== AI timr
    • Mac Intel xservers - to replace the minis - for the CoLo
  • Run 1.8 on one machine and 1.8.0 and 1.9 on another machine
    • So they all complete within a day

Next steps for collecting /reporting results

  • Litmus is not necessarily the best place for results
  • Look at what buildbot displays
    • easy to extend
  • Long running tests can still be run and reported with one of the late night builds
  • Litmus advanced query would be useful for future data mining and non-tinderbox oriented reporting

Plans/ideas for test extension

  • New tool available from Seneca College
    • Developed by David Hamp-Gonsalves
    • See Seneca College Project - Buggy Bar
    • Grabs next unconfirmed bug from bugzilla
    • Good as long as bug is formatted by the bug submitter tool
    • Waiting for AMO group to sign off on Buggy Bar
    • Updates bug with confirmed status and comments
      • Add to comment: "confirmed via buggy bar" or something similar so we can see how may bugs are being processed by this extension?
      • Tracy to check if David can add unique text in comment so we can see how this is working. <<== AI tracy
    • He has offered to create a litmus extension
      • tracy - to lead this <<== AI tracy
  • RefTests - not good candidates for test ext
  • Other testing extensions [bob]
    • Spider - general purpose
      • Fire off JS events
      • See Bob's blog - about attempting to drive browser from KB, Mouse events, fill in forms, etc.
      • Spider is currently very user un-friendly but a lot of potential
      • Next step: discuss in weekly JS test automation mtg

Extending search engine/trademark verifiers (search enginers, rss feeds, search strings, etc)

  • Didn't get to this