FirefoxSummit/2006/ProposedSessions/CommunityTestingAndLitmus/Notes

From MozillaWiki
Jump to: navigation, search

Stats

  • Now nearly 3500 testcases
  • About 850 active testers out of the 2000 who have accounts.
  • Not seeing much overlap between testdays, only a few people come back.
  • Of the top testers, several community members keeping pace with MoCo staff testers.
  • Dip in number of testers Aug/Sep -- one was Thunderbird which historically has less interest, and we had a lot of testdays around the same time, community getting burned out?

Test days

  • Seneca testday 2006-11-03 with 35 testers. Harness that kind of support in an ongoing basis? Seneca students assigned to come, most probably won't come back.
  • Trying to reward people with shirts and such, but we can't keep doing it to attract people
  • School/university support
  • Scheduling
    • Make testing calendar more widely known, publicize dates better
    • Don't schedule them at the list minute
  • Send automatic email to new Litmus testers after they've done some testing to introduce Mozilla QA + provide resources to get more involved.
  • More creative testdays--task oriented tests. The fun smoketest.
  • Let people create their own testcases and groups specific to content and sites they interact with regularly. Mix and match existing testcases and new tests.
  • Email out testday notifications to people who came to previous test days.

Manual test execution/Ways Litmus can help

  • Partner testing
    • Security flags for testcases
  • One-off tests (emails, hendryx data, etc...)
    • "Whiteboard" to track these tests, get them turned into real testcases
  • Pull data from Bugzilla. List of bugs to be verified, etc... Make Litmus show other QA activities.
  • Be better at vetting results + getting bugs found/filed in Bugzilla
  • Display with checkboxes for admins to check off when results have been appropriately dealt with
  • Programmatic regression analysis--determine when tests began failing and find the delta
  • Daily digest email from Litmus, giving failures and stats from the day's testing -- Bug 360996
  • Testcases in other languages
    • Let the localizers take care of the results
    • Do localizers really need tests in their own language?
    • Create new l10n suite of tests that are sensitive to locale changes
  • Easy way to tag tests/make a small list of tests to point people to around a specific area

Admin Interface Fixes

  • Test group editing is foobar'd
  • Adding testcases as a one-step operation again, not going through the hierarchy

Test Runs

  • Right now a very freeform testing interface. Need a better way to target testers to what we need help with right now
  • Adding an extra Test Run layer doesn't really add much functionality
  • Homepage to point people towards what we want to test
  • Limiting - test a particular build id on certain platforms
  • How is a run different then a group?
  • Run is a bucket of tests with a limited lifetime or permanent
  • Make it simpler, point people to the build and the tests--pimp it to them up front
  • Nomenclature - "run tests" and "test runs"
  • Admin reporting
    • Generate breakdown by subgroup, coverage data, etc...
  • "What is Litmus" box on the side of pages, tips