User:Ashughes/Proposals/Good-First-Bug-Verifications
From MozillaWiki
Contents
Proposal for Bug Fix Verification Community Mentorship
Goal
The purpose of this proposal is to provide a more accessible way for new community members to learn about and contribute to the process of verifying bug fixes.
Current Process
- QA Release Driver triages a list of status-firefoxN:fixed bugs
- Any bug deemed important enough to test before releasing the next Firefox version is flagged for testing using the verifyme keyword
- Similarly, the Driver will also triage the hg pushlog for a given milestone to flag important bugs with the verifyme keyword
- The testing team is then advised to verify all the verifyme flagged bugs before QA signs off a release
The Problem
Poor visibility to outsiders of:
- how to contribute to the qualification of a release
- how to test and verify a bug fix
- what it means to mark a bug verified
- who to talk to and ask questions
Proposal
Until we can finalize an official proposal I recommend the following intermediate proposal to be implemented as of Firefox 27 Nightly and iterated upon as it's put into practice.
Responsibilities
- QA Driver
- triage fixes as the land on Nightly and/or uplifted to Aurora/Beta as per details below
- ensure whiteboard tags, keywords, status flags, and resolution appropriately set
- educate testers and developers to ease maintenance over time
- find mentors for unassigned [good first verify] bugs
- Mentors (ie. teaching newcomers how to verify fixes)
- add "QA Mentor" to your name in Bugzilla
- set yourself as QA Contact on any bug you wish to mentor someone on
- engage with a person ASAP when they reach out on IRC, mailing list, or in Bugzilla
- provide guidance, including step by step instructions if necessary, to verify a bug
- provide assistance when they get stuck but try not to do the work for them
- continue to provide mentorship and tasks (ie. more bugs) until they become self sufficient
- Experienced Testers
- test high risk, high importance, and/or time sensitive fixes as identified by the verifyme keyword
- set status flags and resolution to VERIFIED FIXED once verified across all patched branches
- assign themselves as mentors to [good first verify] bugs by using the QA Contact field
- provide guidance, encouragement, and appreciation to people who identify themselves as wanting to help
- push people who help to verify more fixes
- Bug Reporter
- test fixes we were unable to reproduce or which require an uncommon configuration; as identified by the need-info? flag
- set status flags and resolution to VERIFIED FIXED once verified across all patched branches; QA Driver to assist as needed
- Inexperienced Testers
- introduce themselves in [good first verify] bugs they want to test by adding a comment to the bug
- if a mentor is unassigned the QA Driver should find someone
- if a mentor is assigned the mentor should provide the tester as much guidance as needed to verify the patch across all branches
Tags, Keywords, Flags, etc
- verifyme (keyword) - fixes deemed high risk, important, and/or time sensitive; to be verified by experienced testers
- [qa-] (whiteboard) - fixes deemed unnecessary or impossible to verify (ex. in-testsuite, new APIs, test failures, etc)
- need-info? (flag) - fixes where the reporter's assistance is required (ex. uncommon configuration); to be verified by the reporter
- [good first verify] (whiteboard) - fixes deemed low risk and straightforward to test, ideal for inexperienced testers to learn about and help with verifications
Evangelism and Education
- integrate into a Get Involved section to Firefox test plans
- reach out periodically to QMO and mailing lists to solicit for volunteers
- educate developers and testers of the use of [qa-], verifyme, and [good first verify]
- incorporate [good first verify] into bugday and testday events