Engagement/Social/Facebook/Firefox 4 Badges/retrospective

Mozilla?

   What went well?
       Webqa and Engagement were quickly and easily accessible
       Quick turn around when answering questions
       Test plan was well (mostly) scoped and well defined
           Testers had good familiarity with Facebook
           Didn’t need use cases
   Where were the trouble spots?
       We didn’t setup a pre-made form to file bugs through
       Towards the end of the project the developer went missing in action
       Didn’t properly access uTest’s familiarity with Bugzilla
       Took on too much responsibility of verifying bugs
           How much is enough?

uTest?

   What went well?
       Tim was accessible 24-7, quick to return emails
       Augmented test plan midcycle
       Lot’s of coverage/exploratory testing quickly
       uTest offered to have testers verify bug fixes at no extra cost
   Where were the trouble spots?
       Testing missed a significant SSL bug
       Concerns about bugs being entered in uTest Dashboard but not Bugzilla

       Bugzilla needs to be more closely tracked
           Often missing screenshots
           Clarity on reproduction
           Testers were slow to return queries for more information

Conclusions Mozilla

   I feel comfortable letting uTest manage test runs
   Work to ensure the developers are more responsive
   Add SSL into test plan scope
   Bugzilla
       Setup form for testers to submit bugs through
       Work with Tim Evans (or other stewards) on good Bugzilla practices
           How to track bugs more effectively

uTest

   Bugzilla
       Track bugs more closely
           Ensure screenshots/videos are attached to bugs
           Watch bugs for questions from the team
           Nudge testers accordingly
       Future test cycles (experiment)
           Testers will submit bugs as unconfirmed
           uTest will move bugs to new

   Regression testing (we’ll experiment with this)
       The tester who submitted the bug will be responsible for re-testing
       uTest will bump bug to verified
   Best channels for conversation
       For immediate response skype or phone
           timothyd.evans@skype
       Less immediate - email
       Establish a backup contact at uTest

Other

   Can we be doing something different to involve the community?

   Does the current uTest reward structure encourage not enough noise?
       Testers lose points when they submit duplicates
       Rewarded for being point person on a new bug
       Encouraged to submit quickly/first - sometimes this rewards a lack of clarity with the filed bug (too much noise).
   How many (Bugzilla)
       Total: 11
       Verified new/will fix: 2
       # of won’t fix: 5 -- many of these ended up being Facebook bugs
       # of invalid: 2
       # of duplicates: 2
   How many (uTest Dashboard)
       submitted bugs: 36