User:Ashughes/Proposals/Good-First-Bug-Verifications

From MozillaWiki
Jump to: navigation, search

Proposal for Bug Fix Verification Community Mentorship

Goal

The purpose of this proposal is to provide a more accessible way for new community members to learn about and contribute to the process of verifying bug fixes.

Current Process

  • QA Release Driver triages a list of status-firefoxN:fixed bugs
  • Any bug deemed important enough to test before releasing the next Firefox version is flagged for testing using the verifyme keyword
  • Similarly, the Driver will also triage the hg pushlog for a given milestone to flag important bugs with the verifyme keyword
  • The testing team is then advised to verify all the verifyme flagged bugs before QA signs off a release

The Problem

Poor visibility to outsiders of:

  • how to contribute to the qualification of a release
  • how to test and verify a bug fix
  • what it means to mark a bug verified
  • who to talk to and ask questions

Proposal

Until we can finalize an official proposal I recommend the following intermediate proposal to be implemented as of Firefox 27 Nightly and iterated upon as it's put into practice.

Responsibilities

  • QA Driver
    • triage fixes as the land on Nightly and/or uplifted to Aurora/Beta as per details below
    • ensure whiteboard tags, keywords, status flags, and resolution appropriately set
    • educate testers and developers to ease maintenance over time
    • find mentors for unassigned [good first verify] bugs
  • Mentors (ie. teaching newcomers how to verify fixes)
    • add "QA Mentor" to your name in Bugzilla
    • set yourself as QA Contact on any bug you wish to mentor someone on
    • engage with a person ASAP when they reach out on IRC, mailing list, or in Bugzilla
    • provide guidance, including step by step instructions if necessary, to verify a bug
    • provide assistance when they get stuck but try not to do the work for them
    • continue to provide mentorship and tasks (ie. more bugs) until they become self sufficient
  • Experienced Testers
    • test high risk, high importance, and/or time sensitive fixes as identified by the verifyme keyword
    • set status flags and resolution to VERIFIED FIXED once verified across all patched branches
    • assign themselves as mentors to [good first verify] bugs by using the QA Contact field
    • provide guidance, encouragement, and appreciation to people who identify themselves as wanting to help
    • push people who help to verify more fixes
  • Bug Reporter
    • test fixes we were unable to reproduce or which require an uncommon configuration; as identified by the need-info? flag
    • set status flags and resolution to VERIFIED FIXED once verified across all patched branches; QA Driver to assist as needed
  • Inexperienced Testers
    • introduce themselves in [good first verify] bugs they want to test by adding a comment to the bug
    • if a mentor is unassigned the QA Driver should find someone
    • if a mentor is assigned the mentor should provide the tester as much guidance as needed to verify the patch across all branches

Tags, Keywords, Flags, etc

  • verifyme (keyword) - fixes deemed high risk, important, and/or time sensitive; to be verified by experienced testers
  • [qa-] (whiteboard) - fixes deemed unnecessary or impossible to verify (ex. in-testsuite, new APIs, test failures, etc)
  • need-info? (flag) - fixes where the reporter's assistance is required (ex. uncommon configuration); to be verified by the reporter
  • [good first verify] (whiteboard) - fixes deemed low risk and straightforward to test, ideal for inexperienced testers to learn about and help with verifications

Evangelism and Education

  • integrate into a Get Involved section to Firefox test plans
  • reach out periodically to QMO and mailing lists to solicit for volunteers
  • educate developers and testers of the use of [qa-], verifyme, and [good first verify]
  • incorporate [good first verify] into bugday and testday events