QA/TDAI/Test Farm Roadmap

Q4 2006

• (DONE) create Windows XP SP2 test-specific reference platform
• (DONE) install RHEL and XP reference platform to internal test machines
• (DONE) assist Alice in setup of test harness, tests and graph server

Q1 2007

• create Vista, Windows server, Linux and Mac test-specific reference platforms
• identify and order batches of test hardware - at least two of each ref platform on each applicable hardware configuration
  • document hardware platform
• document procedure for installing new test machines e.g.
  • for IT:
    • installing ref image
    • installing client software
  • for build:
    • basic configuration
  • for QA:
    • decide on right location for test harness, tests and test data (CVS?), changes picked up automatically on test machines
• Use buildbot for driving test harness
  • trigger test based on build availability
    • (DONE) rhelmer working on tinderboxpoller change object
    • some tinderbox client and server bugs that will make this better:
      • https://bugzilla.mozilla.org/show_bug.cgi?id=291167
      • https://bugzilla.mozilla.org/show_bug.cgi?id=369449
  • report test run status to tinderbox
    • robcee working on setup
  • farm out jobs to pools of machines as-needed
• port performance test harness to Mac/Linux
• buildbot testing infrastructure
  • (DONE) setup linux master
  • (DONE) setup linux & win32 slave
  • (DONE) setup mac slave
  • (DONE) setup make check
  • (DONE) setup reftest on linux, win32 & mac
  • (DONE) document installation
  • (DONE) setup tinderbox & email notification (reporting to MozillaTest)
  • (DONE) add mochitest
  • (DONE) prettify tinderbox reports
  • add js tests
  • add/port leak tests
  • performance testing?