User:Rcampbell/Toronto Summit Notes

These are robcee's bulletized notes from the Toronto QA Development Summit on January 25th-26th, 2007. They are rough and there are large pieces missing. Feel free to fill in the gaps.

automated testing

buildbot functions

• pull nightly
• initiate test runs
• build given patch
  • ben hearsum doing work with build team on this
• select platforms
• reporting handled by performance testing framework
• litmus reporting
• clean up build directory/create cleaner mirror
  • get rid of kruft
• drive performance testing

Reporting Tools

Survey of reporting tools

• litmus
• buildbot
• tinderbox

Report aggregation

• RSS
• Atom
• Microformats
• Security
  • passworded version
  • email
  • irc
• report drill-down
  • history
  • test details
  • logs
  • store links in database for detailed analysis

Rollups

• graphserver
• buildbot?

History

• store some set period
• retain important historical milestones / benchmarks

JS Tests

• specialized builds
• buildbot vs tinderbox canned builds
  • requires js shell, xpcshell, can't use 'off-the-shelf' versions.

Metrics

• bug density
  • per component
  • map bugzilla modules
  • checkins churns
• lines of code
• performance thresholds
  • links on wiki
• top-crashers