FirefoxSummit/2006/ProposedSessions/CommunityTestingAndLitmus/Notes

Stats

Now nearly 3500 testcases
About 850 active testers out of the 2000 who have accounts.
Not seeing much overlap between testdays, only a few people come back.
Of the top testers, several community members keeping pace with MoCo staff testers.
Dip in number of testers Aug/Sep -- one was Thunderbird which historically has less interest, and we had a lot of testdays around the same time, community getting burned out?

Seneca testday 2006-11-03 with 35 testers. Harness that kind of support in an ongoing basis? Seneca students assigned to come, most probably won't come back.
Trying to reward people with shirts and such, but we can't keep doing it to attract people
School/university support
Scheduling
- Make testing calendar more widely known, publicize dates better
- Don't schedule them at the list minute
Send automatic email to new Litmus testers after they've done some testing to introduce Mozilla QA + provide resources to get more involved.
More creative testdays--task oriented tests. The fun smoketest.
Let people create their own testcases and groups specific to content and sites they interact with regularly. Mix and match existing testcases and new tests.
Email out testday notifications to people who came to previous test days.

Partner testing
- Security flags for testcases
One-off tests (emails, hendryx data, etc...)
- "Whiteboard" to track these tests, get them turned into real testcases
Pull data from Bugzilla. List of bugs to be verified, etc... Make Litmus show other QA activities.
Be better at vetting results + getting bugs found/filed in Bugzilla
Display with checkboxes for admins to check off when results have been appropriately dealt with
Programmatic regression analysis--determine when tests began failing and find the delta
Daily digest email from Litmus, giving failures and stats from the day's testing -- Bug 360996

Testcases in other languages
- Let the localizers take care of the results
- Do localizers really need tests in their own language?
- Create new l10n suite of tests that are sensitive to locale changes

Easy way to tag tests/make a small list of tests to point people to around a specific area

Right now a very freeform testing interface. Need a better way to target testers to what we need help with right now
Adding an extra Test Run layer doesn't really add much functionality
Homepage to point people towards what we want to test
Limiting - test a particular build id on certain platforms
How is a run different then a group?
Run is a bucket of tests with a limited lifetime or permanent
Make it simpler, point people to the build and the tests--pimp it to them up front
Nomenclature - "run tests" and "test runs"
Admin reporting
- Generate breakdown by subgroup, coverage data, etc...