QA/TDAI/Meetings/2007-01-05

TDAI - Test Development, Automation, Infrastructure Discussion

• 10am, January 5th, 2007

Dial-in

1. Dial Toll-free Number: 1-866-879-4799 (password: 369) or International Number: 650-903-0800 x91
2. Then dial my conference number 245#

Agenda

• Status/plans for buildbot infrastructure
• Review reporting infrastructure
• Status of Test Extension
• Update on the Tp test tool
• Discussion test suite/harness/infrastructure inventory
• Discuss options for detecting server side bouncer problems like we had for 1509/2001

Attendees

• tracy, coop, marcia, juan, bc, martijn, alice, robcee, tony, timr, carsten

Discussion

Status/plans for buildbot infrastructure

• Linux VM doesn't have windowing env
  • Robcee had requested the buildbot to be configured as a "build" spec VM. He didn't realize this meant no windows
  • Options: use RPM?
  • tinderbox "build" spec needs windowing env to run tinderbox tests [coop]
  • Robcee and Coop with work on this with rhelmer <<== AI Robcee and Coop
• Next steps once VM is ready:
  • Land buildbot config for Win and Linux from robcee's local config
  • Get Makecheck tests running
  • Setup Reftests - (results online on new tinderbox page)
  • Setup Mochikit tests
    • Currently these are rendering/display(?) regression tests
  • Then JS tests

Review reporting infrastructure

• tinderbox sucks for double clicking on results
• eggplant results might be a good as a next step to capture and report results with the ability to drilldown
• Key things a database like litmus provides:
  • determining for which tests were run, when they were run, last pass or fail, what OS they were rub on, what versions of the product were tested
  • trending over time
  • red/yellow/green dashboards
• Priority is on getting tests to report status to tinderbox page. Then collecting/presenting the data in more interesting ways
  • Next steps:
    • Tracy to get eggplant fully automated and tied into litmus <<== AI: tracy and coop

Status of Test Extension

• Robcee sent David Joseph Hamp-Gonsalves list of requirements - not heard back
• Requirements from Rob's email:
  • store user login (may not be necessary since litmus'll put a cookie on the user's machine anyway)
  • run tests
    • Select product,
    • Platform,
    • OS,
    • ...
  • testing group
  • subgroup
  • walk the user through each test in the group
    • iterate through the steps to perform or even just display them
    • present the expected results
    • let the user enter pass/fail/unclear, notes and associated bug ids.
  • submit the results
• David seems enthusiastic to help extend his current extension concept
• Tracy - have David meet us in Toronto? Good idea!
  • Who should follow-up with David? <<== AI timr

Update on the Tp test tool

• graydon - asked Alice to run on trunk - broke
  • We assumed if it worked on 1.8.0 and 1.8.1 it should run on 1.9 - bad assumption
• Tp Currently runs on Win and we have results for 1.8.0 and 1.8.1
• OS X perf tests - [robcee]
  • Need Mac/linux timing utils. Robcee knows of python tools for this
• Ideas from Schrep
  • (left handwritten notes at work - will provide better details and priority later)
  • Update current perf tests: Tp, Ts,
  • competitive benchmarking
  • platform specific
  • more granular (not just high level user based

Discussion test suite/harness/infrastructure inventory

• The purpose is to get better visibility into:
  • Which test harnesses and infrastructure do the current tests use
  • What is the status of development or consolidation of test harnesses
  • What is the status of setting up major infrastructures
• Some important side purposes are
  • Get a sense of scope and coverage for the test suites we have. This is a side purpose as understanding scope and coverage requires different and more detailed information.
  • Get a sense of how usable and how well documented these harnesses are. This is a side purpose as this will be embedded in the status but really needs more details that should be placed elsewhere.

• Initial draft of inventory can be found here

• Some tests, harnesses, and other resources we discussed
  • necko harness
  • nss suite
  • nspr suite
  • xpfe suite
  • jssh harness - out of date - doesn't work on trunk
    • Introduces event timing issues - possible incorrect ersults [rob]
  • js/sec automation harness
    • sec not automated
      • timr and others seem to keep forgetting it is not automated  %-}
    • js framework is custom build by bc and is used for other things
      • It uses the shell scripting language to perform a set of very modular steps:
        • build/install browser on 3 platforms/ 3 branches
        • create profile
        • start browser
        • execute web pages
        • collect results
      • It is a set of building blocks
      • Plans
        • Get this into src tree testing dir soon for reuse by others
        • Need better reporting
        • Need better ability to add tests

• Other lists of test tools and references:
  • from robcee:
    • wiki.mozilla.org/SoftwareTesting
    • developer.mozilla.org/en/docs/Mozilla_automated_testing
  • from marcia:
    • lxr.mozilla.org/seamonkey/source/embedding/qa/testembed/ - Gecko API work from Dave Epstein that may be able to be re-purposed

Discuss options for detecting server side bouncer problems

• This is prompted by the major last minute problem we had for the 1509/2001 release
• Bug covering this: bug 364499
• One option is to automate server side verification
  • We need a tool to get updates from AUS and verify the update is applied correctly
• Davel's update checker just gets the mar file from an ftp server and applies it. This does not test the AUS mechanism not the client side mechanism.
• A related issue is why people are getting multiple updates
• Next steps for main issue: bug 364499
  • See how the 1509/2001 postmortem goes on Monday, Jan 8th. If this is still a high priority after that meeting we can see who else is willing to help create a test tool and then either lead or support this project.