FirefoxSummit/2006/ProposedSessions/CommunityTestingAndLitmus/Notes

Stats

Now nearly 3500 testcases
About 850 active testers out of the 2000 who have accounts.
Not seeing much overlap between testdays, only a few people come back.
Of the top testers, several community members keeping pace with MoCo staff testers.
Dip in number of testers Aug/Sep -- one was Thunderbird which historically has less interest, and we had a lot of testdays around the same time, community getting burned out?

Seneca testday 2006-11-03 with 35 testers. Harness that kind of support in an ongoing basis? Seneca students assigned to come, most probably won't come back.
Trying to reward people with shirts and such, but we can't keep doing it to attract people
School/university support
Scheduling
- Make testing calendar more widely known, publicize dates better
- Don't schedule them at the list minute
Send automatic email to new Litmus testers after they've done some testing to introduce Mozilla QA + provide resources to get more involved.
More creative testdays--task oriented tests. The fun smoketest.
Let people create their own testcases and groups specific to content and sites they interact with regularly. Mix and match existing testcases and new tests.
Email out testday notifications to people who came to previous test days.

Partner testing
- Security flags for testcases
One-off tests (emails, hendryx data, etc...)
- "Whiteboard" to track these tests, get them turned into real testcases
Pull data from Bugzilla. List of bugs to be verified, etc... Make Litmus show other QA activities.
Be better at vetting results + getting bugs found/filed in Bugzilla
Display with checkboxes for admins to check off when results have been appropriately dealt with
Programmatic regression analysis--determine when tests began failing and find the delta
Daily digest email from Litmus