From MozillaWiki
< QA‎ | TDAI‎ | Meetings
Jump to: navigation, search

« previous mtg | index | next mtg »

TDAI - Test Development, Automation, Infrastructure Discussion

  • 10am, January 5th, 2007
  • Dial-in:
    • Dial Toll-free Number: 1-866-879-4799 (password: 369) or International Number: 650-903-0800 x91
    • Then dial my conference number 245#


  • Status/plans for buildbot infrastructure
  • Review reporting infrastructure
  • Status of Test Extension
  • Update on the Tp test tool
  • Discussion test suite/harness/infrastructure inventory
  • Discuss options for detecting server side bouncer problems like we had for 1509/2001


  • tracy, coop, marcia, juan, bc, martijn, alice, robcee, tony, timr, carsten


Status/plans for buildbot infrastructure

  • Linux VM doesn't have windowing env
    • Robcee had requested the buildbot to be configured as a "build" spec VM. He didn't realize this meant no windows
    • Options: use RPM?
    • tinderbox "build" spec needs windowing env to run tinderbox tests [coop]
    • Robcee and Coop with work on this with rhelmer <<== AI Robcee and Coop
  • Next steps once VM is ready:
    • Land buildbot config for Win and Linux from robcee's local config
    • Get Makecheck tests running
    • Setup Reftests - (results online on new tinderbox page)
    • Setup Mochikit tests
      • Currently these are rendering/display(?) regression tests
    • Then JS tests

Review reporting infrastructure

  • tinderbox sucks for double clicking on results
  • eggplant results might be a good as a next step to capture and report results with the ability to drilldown
  • Key things a database like litmus provides:
    • determining for which tests were run, when they were run, last pass or fail, what OS they were rub on, what versions of the product were tested
    • trending over time
    • red/yellow/green dashboards
  • Priority is on getting tests to report status to tinderbox page. Then collecting/presenting the data in more interesting ways
    • Next steps:
      • Tracy to get eggplant fully automated and tied into litmus <<== AI: tracy and coop

Status of Test Extension

  • Robcee sent David Joseph Hamp-Gonsalves list of requirements - not heard back
  • Requirements from Rob's email:
    • store user login (may not be necessary since litmus'll put a cookie on the user's machine anyway)
    • run tests
      • Select product,
      • Platform,
      • OS,
      • ...
    • testing group
    • subgroup
    • walk the user through each test in the group
      • iterate through the steps to perform or even just display them
      • present the expected results
      • let the user enter pass/fail/unclear, notes and associated bug ids.
    • submit the results
  • David seems enthusiastic to help extend his current extension concept
  • Tracy - have David meet us in Toronto? Good idea!
    • Who should follow-up with David? <<== AI timr

Update on the Tp2 test tool

  • graydon - asked Alice to run on trunk - broke
    • We assumed, if it worked on 1.8.0 and 1.8.1, it should run on 1.9
    • Bad assumption!
  • Tp2 Currently runs on Window and we have results for 1.8.0 and 1.8.1
  • OS X perf tests - [robcee]
    • Need Mac/linux timing utils. Robcee knows of python tools for this
  • Ideas from Schrep
    • (left handwritten notes at work - will provide better details and priority later)
    • Update current perf tests: Tp, Ts,
    • competitive benchmarking
    • platform specific
    • more granular (not just high level user based)

Discussion test suite/harness/infrastructure inventory

  • The purpose is to get better visibility into:
    • Which test harnesses and infrastructure do the current tests use
    • What is the status of development or consolidation of test harnesses
    • What is the status of setting up major infrastructures
  • Some important side purposes are
    • Get a sense of scope and coverage for the test suites we have. This is a side purpose as understanding scope and coverage requires different and more detailed information.
    • Get a sense of how usable and how well documented these harnesses are. This is a side purpose as this will be embedded in the status but really needs more details that should be placed elsewhere.
  • Some tests, harnesses, and other resources we discussed
    • necko harness
    • nss suite
    • nspr suite
    • xpfe suite
    • jssh harness - out of date - doesn't work on trunk
      • Introduces event timing issues - possible incorrect ersults [rob]
    • js/sec automation harness
      • sec not automated
        • timr and others seem to keep forgetting it is not automated  %-}
      • js framework is custom build by bc and is used for other things
        • It uses the shell scripting language to perform a set of very modular steps:
          • build/install browser on 3 platforms/ 3 branches
          • create profile
          • start browser
          • execute web pages
          • collect results
        • It is a set of building blocks
        • Plans
          • Get this into src tree testing dir soon for reuse by others
          • Need better reporting
          • Need better ability to add tests

Discuss options for detecting server side bouncer problems

  • This is prompted by the major last minute problem we had for the 1509/2001 release
  • Bug covering this: bug 364499
  • One option is to automate server side verification
    • We need a tool to get updates from AUS and verify the update is applied correctly
  • Davel's update checker just gets the mar file from an ftp server and applies it. This does not test the AUS mechanism not the client side mechanism.
  • A related issue is why people are getting multiple updates
  • Next steps for main issue: bug 364499
    • See how the 1509/2001 postmortem goes on Monday, Jan 8th. If this is still a high priority after that meeting we can see who else is willing to help create a test tool and then either lead or support this project.