Confirmed users
1,595
edits
QA/Feature Testing v2: Difference between revisions
m
→History of changes: -
Andrei.vaida (talk | contribs) m (-) |
Andrei.vaida (talk | contribs) m (→History of changes: -) |
||
| (9 intermediate revisions by the same user not shown) | |||
| Line 41: | Line 41: | ||
* <u>A feature kick-off meeting is scheduled once technical documentation is received</u> | * <u>A feature kick-off meeting is scheduled once technical documentation is received</u> | ||
** ''' | ** '''Soft deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk5] | ||
** '''Responsible''': QA, Eng/Product | ** '''Responsible''': QA, Eng/Product | ||
** '''Description''': The kick-off meeting is the perfect place for QA to get a better understanding of the scope of the feature and clarify any questions related to it. | ** '''Description''': The kick-off meeting is the perfect place for QA to get a better understanding of the scope of the feature and clarify any questions related to it. | ||
| Line 49: | Line 49: | ||
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk5] | ** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk5] | ||
** Responsible: QA | ** Responsible: QA | ||
** '''Description''': Documents are created for the Test Plan and High-level Test Cases. QA will provide the links and ask for formal approval | ** '''Description''': Documents are created for the Test Plan and High-level Test Cases. QA will provide the links and ask for formal approval via email. | ||
*** Once the links are provided, QA will set the status of the Jira ticket to <code>WAITING FOR DEVELOPERS</code>. | *** Once the links are provided, QA will set the status of the Jira ticket to <code>WAITING FOR DEVELOPERS</code>. | ||
** '''References''': [https://docs.google.com/document/d/1Dj3IYlmRpBtC9sSAmuecbpOgme_n6zu6eWZok7bMs5E/edit#heading=h.z6ne0og04bp5 Test Plan template], [https://docs.google.com/spreadsheets/d/ | ** '''References''': [https://docs.google.com/document/d/1Dj3IYlmRpBtC9sSAmuecbpOgme_n6zu6eWZok7bMs5E/edit#heading=h.z6ne0og04bp5 Test Plan template], [https://docs.google.com/spreadsheets/d/1OleeB9WGjf5ufh_hot0TD91cWYFDY5C9uGAsLeRZKFw/edit#gid=0 Test Cases template] | ||
* <u>Test Plan and Test Cases are reviewed and formal approvals are made</u> | * <u>Test Plan and Test Cases are reviewed and formal approvals are made</u> | ||
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk7] | ** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk7] | ||
** '''Responsible''': Eng/Product | ** '''Responsible''': Eng/Product | ||
** '''Description''': | ** '''Description''': Feedback and formal approval should be given on the email thread started by QA on this matter. | ||
==== Phase III - Go/No-Go decision based on feature completeness ==== | ==== Phase III - Go/No-Go decision based on feature completeness ==== | ||
| Line 69: | Line 69: | ||
** '''Responsible''': QA | ** '''Responsible''': QA | ||
** '''Description''': In the case of features for which Nightly testing was requested, a test report will be sent out highlighting concerns. | ** '''Description''': In the case of features for which Nightly testing was requested, a test report will be sent out highlighting concerns. | ||
** '''Reference''': [https://docs.google.com/document/d/14GOeKqoNWBZFVlmbo1kqEJik3kvpvhhizVq0QJjmpkk/edit Nightly testing optional report template] | |||
* <u>Go/No-Go decision is made based on feature completeness, quality and risk</u> | * <u>Go/No-Go decision is made based on feature completeness, quality and risk</u> | ||
| Line 88: | Line 89: | ||
** '''Reference''': [https://docs.google.com/document/d/1iY25P1_AuIRagz1NBMJmDn_2XuzR_WEvHCaI7LUPCuY/edit Preliminary status report optional template] | ** '''Reference''': [https://docs.google.com/document/d/1iY25P1_AuIRagz1NBMJmDn_2XuzR_WEvHCaI7LUPCuY/edit Preliminary status report optional template] | ||
==== Phase V - Pre-Release test execution, sign off and follow-ups | ==== Phase V - Pre-Release test execution, sign off and follow-ups ==== | ||
* <u>QA provides a formal, Pre-Release feature sign off</u> | * <u>QA provides a formal, Pre-Release feature sign off</u> | ||
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Beta, wk7] | ** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Beta, wk7] | ||
| Line 335: | Line 336: | ||
** A continuous monitoring process should be in place for new bug fixes. | ** A continuous monitoring process should be in place for new bug fixes. | ||
*** This can be easily done by setting up Bugzilla queries, or something similar. | *** This can be easily done by setting up Bugzilla queries, or something similar. | ||
== Sign off overview == | |||
'''''The QA sign off is based on the quality status of the feature in question. The green, yellow or red quality status is a schedule-based conversation, NOT a general release-readiness status.''''' | |||
=== Quality status evaluation criteria === | |||
The following aspects are taken into account when evaluating the quality status of a feature: | |||
* percentage of completed implementation, based on Engineering’s commitment | |||
* unresolved bugs count and severity | |||
* failed tests count and failure reason | |||
* blocked tests percentage and blocking reason | |||
=== Quality statuses === | |||
==== Green ==== | |||
'''Status description'''<br/> | |||
A <font style="background-color:green;">GREEN</font> quality status indicates a feature that meets all of QA's expectations. | |||
'''Status reasons''' | |||
* Everything is going according to our plan. | |||
* Manual test results have met the acceptance criteria that was agreed upon with Engineering. | |||
* All or major and critical test objectives agreed upon with Engineering have been covered. | |||
* There are no issues severe enough to jeopardize the release of the feature. | |||
'''Example''' | |||
* [65] [desktop] [feature] Widevine 4.10.1196.0 - pre-Release QA sign off (GREEN) - [https://docs.google.com/document/d/1IG6VOQ3CN4rGESGYnyl71kZfaNRuXi90mIX0lwhUioU/edit link] | |||
==== Yellow ==== | |||
'''Status description'''<br/> | |||
A <mark>YELLOW</mark> quality status indicates a feature of questionable quality, meaning that there are issues or potential issues that need to be addressed by the Product, User Experience and Engineering teams as part of that feature’s Go/NoGo decision. | |||
'''Status reasons''' | |||
* Manual test results have ''mostly'' met the acceptance criteria that was agreed upon with Engineering, with a few minor to medium exceptions that need to be addressed. | |||
* Only major and critical test objectives that were agreed upon with Engineering have been covered, minor ones have been left out. | |||
* There are ''moderate issues'' that might jeopardize the release of the feature, but an input is required from Product, User Experience and/or Engineering to determine if they should block or not. | |||
* Testing progress is behind the planned timeline and we are not able to cover everything on time. | |||
'''Example''' | |||
* [65] [desktop] [feature] Language Switcher - Pre-Release QA sign off (YELLOW) - [https://docs.google.com/document/d/1a1YRvIf2ISSVeuZq-9ZWEGoviZqAPiJq7BAvExpELzI/edit link] | |||
==== Red ==== | |||
'''Status description'''<br/> | |||
A <font style="background-color:red">RED</font> quality status indicates a feature of low quality, meaning that there are severe issues that need to be addressed by the Product, User Experience and Engineering teams as part of that feature’s Go/NoGo decision. | |||
'''Status reasons''' | |||
* There’s a ''big mismatch'' between manual test results and the acceptance criteria that was agreed upon with Engineering. | |||
* Part of or all major and critical test objectives that were agreed upon with Engineering have not been covered. | |||
* There are ''severe issues'' jeopardizing the release of the feature or there’s a big mismatch between what was planned for implementation and what was actually implemented. | |||
* We are unable to cover important parts of the feature, due to resource constraints or hardware/environment requirements that we cannot meet, which means that those areas may have important issues that we don’t know about. | |||
'''Example''' | |||
* [60] [desktop] [feature] Normandy Telemetry - pre-Release QA sign off (RED) - [https://drive.google.com/open?id=1Wc26EOlZCIbgbEcfrStIE95qfSkTh-TbwHwm32Za1js link] | |||
==== Pref'd OFF ==== | |||
'''Description'''<br/> | |||
Features riding a specific Firefox version in a disabled state (via preference flip) will not be receiving a sign off report from QA. This is because there’s no real impact for the end user, and so an in-depth assessment of that feature’s quality is irrelevant for the Firefox version in question. | |||
Instead of a sign off, QA will send out a regular test report describing what was considered in scope/covered. | |||
'''Examples''' | |||
* [63] [desktop] [feature] Block Autoplay v2 - Test Results - [https://docs.google.com/document/d/1AYL0fwNldYPVl5ElZFNYDqIOC-e4CxVH7eqAp3Z6Ki0/edit link] | |||
=== Sign off template === | |||
* The mandatory sign off template for pref’d on features is available [https://docs.google.com/document/d/17WVL90Dl8wAQCs_9z4w8qtAhqQUbnBAw5oAurFd3AeI/edit#heading=h.l2sirhdp8k8s here]. | |||
* The mandatory sign off template for pref’d off features is available [https://docs.google.com/document/d/17WVL90Dl8wAQCs_9z4w8qtAhqQUbnBAw5oAurFd3AeI/edit#heading=h.y3lsu9pe3emj here]. | |||
* The guideline for preliminary test reports is available [https://docs.google.com/document/d/1iY25P1_AuIRagz1NBMJmDn_2XuzR_WEvHCaI7LUPCuY/edit here]. | |||
== References == | |||
* All QA documentation available for features, structured per Firefox version: [https://drive.google.com/open?id=0B-n79IWvmJYYQngxVjhiRUJ1bTA here]. | |||
* Feature QA status per Firefox version: | |||
** 2019: [https://docs.google.com/spreadsheets/u/2/d/10iNQQ2tmr0FP49bqmAi66-4qhSjKiW5e6wNQ47E-P9A/edit#gid=0 69], [https://docs.google.com/spreadsheets/d/1ygCmrDTQmoN6ecyNlkPHft6O2tayhn3S1RN5ii_ER_8/edit#gid=0 68], [https://docs.google.com/spreadsheets/d/1J83OlNavefkzcOiy79iVDh6FrHQkpRFejRZ9u76lIVk/edit#gid=0 67], [https://docs.google.com/spreadsheets/d/1vPyFxPmnQt5YZThJlFaeyriLkeXU6sE6jgEHgAryUbo/edit#gid=0 66], [https://docs.google.com/spreadsheets/d/1-zxhm8GJf9fZ0kFbrfB5tzpfCcTqUNHVhDJaCfIr89o/edit#gid=0 65] | |||
** 2018: [https://docs.google.com/spreadsheets/d/1z4_GjF-OHKihIeqfJGzH3NUC0hK3Ct8chyHOm8Nguo4/edit 64], [https://docs.google.com/spreadsheets/u/2/d/1IdIMWOrYymTVKgCPiPoxWQ3xlYBNbGX_UC2rAX1YRYY/edit 63], [https://docs.google.com/document/u/2/d/1rxisBtzIVZwSAuIIzQ5WCAUUYL2ZhtUQ9xiSBBAOCKY/edit 62], [https://docs.google.com/document/d/1IjualMcpheEbZsT7ujyEr0tg4XIpdaG4oRpu1MTUQw0/edit 61], [https://docs.google.com/document/d/1jLrir7BEF7nerypyqbbtdu_uL2IDRLjZqigSOeUMFho/edit 60], [https://docs.google.com/document/d/1HLCYgwnFFj3sIBybrFM19VhccSaLuqwXaRPaGHj1Ap0/edit 59] | |||
** 2017: [https://docs.google.com/document/d/1HLCYgwnFFj3sIBybrFM19VhccSaLuqwXaRPaGHj1Ap0/edit 58], [https://docs.google.com/document/u/2/d/1e840tyKTmhBdlYxtpWsGnyVXgyKVSJxxv9p98mMDqEU/edit 57] | |||
== History of changes == | |||
* '''Aug 6, 2019''' - (btot) Updated the High-Level Test Cases document template | |||
* '''April 23, 2019''' - (avaida) Added an optional test report template for features requiring Nightly testing. | |||