2005Offsite/LitmusFutures

Topics for Discussion

• Vision of Litmus
• Current State
• Future milestones
• Open discussion of useful features
• How developers can help

Session Notes

• admin interface:
  • test events/runs built from testgroups
    • ability to select specific subgroups
      • must firm up granularity with which we want to be able to build test runs;
    • ability to tag runs with random tags (e.g. rc2)
  • test case versioning:
    • tagging
    • localization (l10n)
• test runs:
  • mandatory comments for FAILing results
  • report opt/cflags, obtainable from about:buildconfig (investigate the build ID scraping code that Zach already has in place)
• l10n
  • localize litmus itself
    • issues with test cases/results submitted in languages other than english is that the core MozQA staff cannot help parse results, however if we enlist localizers/locale coordinators and give them some ownership of locale-specific results, we may actually encourage more l10n testing (which is a good thing).
• search:
  • allow for personal saved searches (a la Bugzilla)
    • allow admins to publish personal saved searches to everyone/subsets
  • allow searching by user type
    • requires auth
    • allows for quality of results reporting
• auth:
  • we have to allow for anonymous testing, i.e. we cannot require a Bugzilla account for Litmus use. The perception is that this is too high a barrier to entry for casual testers. Suggestions to overcome this:
    • straight anonymous testing
    • password-based logins with no relation to Bugzilla (con: another account for users to remember/forget)
    • redirect users through Bugzilla on result submission
    • ability to associate results with a Bugzilla/Litmus account after the fact.
      • track some form of data about the user in the db (temp username, cookie) and backfill that once the user signs up with an account.
    • NOTE: these are not necessarily competing paradigms; we might adopt more than one
• general:
  • do we need RSS?
    • perhaps only for automated test results
• QMO
  • return to the idea of QMO -> similar to MDC in that QMO is the clearinghouse for testing-related information:
    • QA blog posts
    • testday info
    • intro to testing/Bugzilla info
    • Litmus tutorial/instructions
    • highlighted testing results
    • etc.
  • speak to dria about lessons learned from MDC
• short term:
  • in prep for Thunderbirrd 1.5 testing, add l10n property to test results (default to en-US). Update search tools to allowing limiting by locale.