QA/Test Days/Stories

From MozillaWiki
< QA‎ | Test Days
Jump to: navigation, search


The following is a repository of "stories" about testdays. What was tried, what was successful, what wasn't, what was learned, ideas new and old.

I recommend that you include the following information in your story as points of reference:

  • Date/Time of your testday
  • Subject/Goals of your testday
  • Links to testplan or any important documents
  • Your name and contact info (ex. email) so people can ask you follow up questions

WebQA 'Live' Test Day

On February 23, 2012 at 18:30, the WebQA team held a combined test day and meetup event. The purpose was to invite web testers from all skill levels to our San Fransisco space to learn hands-on how we test our websites.

This event was organized by Dave Hunt.

Desktop and Android Web Apps Test Day

Event Organizer and Date

Organizer: Jason Smith

Date: February 17th, 2012

Time: 1am - 5pm PST


  • Simulate an end-to-end customer scenario with the Web Apps product
  • Determine what manual test cases pass and fail
  • Verify fixed bugs
  • Exploratory test the product

What was tried

  • Starting test day organization a week and half ahead
  • Utilizing a template built by past test days
  • Following a checklist of items posted by Anthony
  • Publicizing the test day to many different avenues
  • Requesting moderators from waverly and QA staff
  • Building test accounts for using the product
  • Posting links to major builds, test sites, bug queries
  • Asking community members to join Mozillians
  • Posting a "how to" for non-obvious tasks
  • Indicating known issues with the product
  • Posting a key customer scenario to model a demo script with detailed steps
  • Asking community members to run documented manual test cases
  • Asking community members to verify fixed bugs
  • Proposing different areas to exploratory test the product
  • Asking community members to free write with including test results & bug numbers
  • Being highly active on the channel to answer any question that arises
  • Helping new people to testing get ramped up on how to get started on IRC
  • Getting major test day documents and processes reviewed by QA staff members

What was successful

  • Well-prepared mostly for the test day overall
  • Had more enough work for community members to do on the test day
  • Publicity was good - Lots of people knew about this test day
  • Had enough moderators to answer questions
  • Community members were active asking questions
  • Very few setup issues with community members in testing the product
  • Some community members added themselves to Mozillians
  • Many existing resolved bugs were verified on the test day (>10 bugs)
  • Free writing - Allowed community members to be more open about what they found, even if they were unsure
  • Reviewing the test day before starting - caught issues before the test day
  • New people seemed interested and helped out on the test day

What was not successful

  • Lots of issues brought up in free writing, but few bugs filed
  • Configuration issues with the build and deployment on test day
    • Some areas were unavailable to test since the code was not ready
  • Mostly desktop testers, not that many Android testers
  • Did not seem like community members saw that the manual test cases spreadsheet had multiple sheets to it
    • Most tests ran seemed to reflect one of the sheets typically
  • Even with documented known issues, community members hit them and did not realize it was a known issue
  • Facebook integration for test days was found to have some issues with QMO integration