FirefoxSummit/2006/ProposedSessions/CommunityTestingAndLitmus/Notes

Stats

• Now nearly 3500 testcases
• About 850 active testers out of the 2000 who have accounts.
• Not seeing much overlap between testdays, only a few people come back.
• Of the top testers, several community members keeping pace with MoCo staff testers.
• Dip in number of testers Aug/Sep -- one was Thunderbird which historically has less interest, and we had a lot of testdays around the same time, community getting burned out?

Test days

• Seneca testday 2006-11-03 with 35 testers. Harness that kind of support in an ongoing basis? Seneca students assigned to come, most probably won't come back.
• Trying to reward people with shirts and such, but we can't keep doing it to attract people
• School/university support
• Scheduling
  • Make testing calendar more widely known, publicize dates better
  • Don't schedule them at the list minute
• Send automatic email to new Litmus testers after they've done some testing to introduce Mozilla QA + provide resources to get more involved.
• More creative testdays--task oriented tests. The fun smoketest.
• Let people create their own testcases and groups specific to content and sites they interact with regularly. Mix and match existing testcases and new tests.
• Email out testday notifications to people who came to previous test days.

Manual test execution/Ways Litmus can help

• Partner testing
  • Security flags for testcases
• One-off tests (emails, hendryx data, etc...)
  • "Whiteboard" to track these tests, get them turned into real testcases
• Pull data from Bugzilla. List of bugs to be verified, etc... Make Litmus show other QA activities.
• Be better at vetting results + getting bugs found/filed in Bugzilla
• Display with checkboxes for admins to check off when results have been appropriately dealt with
• Programmatic regression analysis--determine when tests began failing and find the delta
• Daily digest email from Litmus, giving failures and stats from the day's testing -- Bug 360996

• Testcases in other languages
  • Let the localizers take care of the results
  • Do localizers really need tests in their own language?
  • Create new l10n suite of tests that are sensitive to locale changes

• Easy way to tag tests/make a small list of tests to point people to around a specific area

Admin Interface Fixes

• Test group editing is foobar'd
• Adding testcases as a one-step operation again, not going through the hierarchy

Test Runs

• Right now a very freeform testing interface. Need a better way to target testers to what we need help with right now
• Adding an extra Test Run layer doesn't really add much functionality
• Homepage to point people towards what we want to test
• Limiting - test a particular build id on certain platforms
• How is a run different then a group?
• Run is a bucket of tests with a limited lifetime or permanent
• Make it simpler, point people to the build and the tests--pimp it to them up front
• Nomenclature - "run tests" and "test runs"
• Admin reporting
  • Generate breakdown by subgroup, coverage data, etc...

• "What is Litmus" box on the side of pages, tips