Confirmed users
1,595
edits
QA/Feature Testing: Difference between revisions
Jump to navigation
Jump to search
m
-
Andrei.vaida (talk | contribs) m (-) |
Andrei.vaida (talk | contribs) m (-) |
||
| Line 154: | Line 154: | ||
===== Example ===== | ===== Example ===== | ||
This is an email report example of a sign off. | This is an email report example of a sign off. | ||
<u>Subject</u>: | * <u>Subject</u>: | ||
<blockquote style="background: #F3F3F3; border-left: 4px solid #ccc; padding-left: 15px;">[57] [desktop] [feature] e10s with a11y - pre-Release QA sign off</blockquote> | <blockquote style="background: #F3F3F3; border-left: 4px solid #ccc; padding-left: 15px;">[57] [desktop] [feature] e10s with a11y - pre-Release QA sign off</blockquote> | ||
<u>Body</u>: | * <u>Body</u>: | ||
<blockquote style="background: #F3F3F3; border-left: 4px solid #ccc; padding-left: 15px;"> '''e10s with a11y''' | <blockquote style="background: #F3F3F3; border-left: 4px solid #ccc; padding-left: 15px;"> '''e10s with a11y''' | ||
Targeted GA: Firefox 57 - November 14, 2017. | Targeted GA: Firefox 57 - November 14, 2017.<br/> | ||
Sign off phase: pre-Release (Fx57). | Sign off phase: pre-Release (Fx57). | ||
<br/><br/> | <br/><br/> | ||
| Line 169: | Line 169: | ||
*** Other browsers (Edge, Chrome, Opera) seem to work fine with both of these versions. A screencast showing the behavior on Fx 57 is available here. | *** Other browsers (Edge, Chrome, Opera) seem to work fine with both of these versions. A screencast showing the behavior on Fx 57 is available here. | ||
*** Our assessment is that while this feature did not improve compatibility with JAWS on Fx 57, it didn’t decrease it either, as Fx 56.0.2 shows the exact same behavior. | *** Our assessment is that while this feature did not improve compatibility with JAWS on Fx 57, it didn’t decrease it either, as Fx 56.0.2 shows the exact same behavior. | ||
** Bug 1411188 is also a point of concern for NVDA in particular (see Comment 1 per jteh), although not severe enough to block this feature from shipping with Fx 57. | ** [https://bugzil.la/1411188 Bug 1411188] is also a point of concern for NVDA in particular (see [https://bugzil.la/1411188#c1 Comment 1] per jteh), although not severe enough to block this feature from shipping with Fx 57. | ||
* '''What needs to be done?''' | * '''What needs to be done?''' | ||
** Due Nov 03 - [aklotz] [rkothari] Take into account the JAWS compatibility issue and Bug 1411188 for the Go/NoGo decision. | ** <u>Due Nov 03</u> - [aklotz] [rkothari] Take into account the JAWS compatibility issue and [https://bugzil.la/1411188 Bug 1411188] for the Go/NoGo decision. | ||
<br/> | <br/> | ||
<u>Quality trend</u>: | <u>Quality trend</u>: | ||
* mid-Nightly: RED (signed off on Aug 30, 2017 - test report) | * mid-Nightly: RED (signed off on Aug 30, 2017 - [https://testrail.stage.mozaws.net/index.php?/plans/view/5795 test report]) | ||
* pre-Beta: YELLOW (signed off on Sep 16, 2017 - test report) | * pre-Beta: YELLOW (signed off on Sep 16, 2017 - [https://testrail.stage.mozaws.net/index.php?/plans/view/6196 test report]) | ||
<br/> | <br/> | ||
<u>Testing summary</u>: | <u>Testing summary</u>: | ||
* Test report: https://testrail.stage.mozaws.net/index.php?/plans/view/6749 | * Test report: https://testrail.stage.mozaws.net/index.php?/plans/view/6749 | ||
* New bugs: Bug 1411188 | * New bugs: [https://bugzil.la/1411188 Bug 1411188] | ||
* Known bugs: none | * Known bugs: none | ||
* Firefox 57 feature QA status: https://goo.gl/i6q3VA | * Firefox 57 feature QA status: https://goo.gl/i6q3VA | ||
<br/> | <br/> | ||
'''What is the release criteria for this feature?''' You can find it here.<br/> | '''What is the release criteria for this feature?''' You can find it here.<br/> | ||
'''What’s this report about?''' Find out here. | '''What’s this report about?''' Find out [https://wiki.mozilla.org/QA/Feature_Testing here]. | ||
Thanks, | Thanks, | ||
| Line 190: | Line 190: | ||
Desktop Release QA | Desktop Release QA | ||
</blockquote> | </blockquote> | ||
==== Red ==== | |||
===== Status description ===== | |||
A <font style="background: #FF0000;">RED</font> quality status indicates a feature of low quality, meaning that there are severe issues that need to be addressed by the Product, User Experience and Engineering teams as part of that feature’s Go/NoGo decision. | |||
===== Status reasons ===== | |||
* There’s a ''big mismatch'' between manual test results and the acceptance criteria that was agreed upon with Engineering. | |||
* Part of or all major and critical test objectives that were agreed upon with Engineering have not been covered. | |||
* There are ''severe issues'' jeopardizing the release of the feature or there’s a ''big mismatch'' between what was planned for implementation and what was actually implemented. | |||
* We are unable to cover important parts of the feature, due to resource constraints or hardware/environment requirements that we cannot meet, which means that those areas may have important issues that we don’t know about. | |||
===== Example ===== | |||
This is an email report example of a sign off. | |||
* <u>Subject</u>: | |||
<blockquote style="background: #F3F3F3; border-left: 4px solid #ccc; padding-left: 15px;">[57] [desktop] [feature] Form Autofill MVP with Credit Card support and Sync - pre-Beta QA sign off</blockquote> | |||
* <u>Body</u>: | |||
<blockquote style="background: #F3F3F3; border-left: 4px solid #ccc; padding-left: 15px;"> '''Form Autofill MVP with Credit Card support and Sync''' | |||
Targeted GA: Firefox 57 - November 14, 2017.<br/> | |||
Sign off phase: pre-Release (Fx57). | |||
<br/><br/> | |||
<u>Quality status</u>: <font style="background: #FF0000;>RED</font> | |||
* '''Why is this feature yellow?''' | |||
** Credit Card information sync has not been implemented yet (see [https://bugzil.la/1395028 Bug 1395028]). | |||
** Form Autofill does not work for some of the top shopping sites (see [https://bugzil.la/1398101 Bug 1398101]), such as Groupon and Newegg. | |||
* '''What needs to be done?''' | |||
** <u>Due Sep 22</u> - [mnoorenberghe] [vchen] [rkothari] To take into account [https://bugzil.la/1398101 Bug 1398101] for this feature’s Go/NoGo decision. | |||
** <u>Due Oct 20</u> - [mnoorenberghe] [vchen] To clarify whether Credit Card information sync is still part of this feature’s scope for Firefox 57. | |||
<br/> | |||
<u>Quality trend</u>: | |||
* no previous sign offs available | |||
<br/> | |||
<u>Testing summary</u>: | |||
* Test report: https://testrail.stage.mozaws.net/index.php?/plans/view/5872 | |||
* New bugs: [https://bugzil.la/1399065 Bug 1399065], [https://bugzil.la/1399402 Bug 1399402] | |||
* Known bugs: https://mzl.la/2CPaUBI | |||
* Firefox 57 feature QA status: https://goo.gl/i6q3VA | |||
<br/> | |||
'''What is the release criteria for this feature?''' You can find it here.<br/> | |||
'''What’s this report about?''' Find out [https://wiki.mozilla.org/QA/Feature_Testing here]. | |||
Thanks, | |||
Gabi Cheta<br/> | |||
Desktop Release QA | |||
</blockquote> | |||
=== Quality statuses === | |||
The feature sign off template is available [https://docs.google.com/document/d/11sCQTPO2F2xWObcBkbXh4VPp9ampujujVvO7XJh7sqA/edit# here]. | |||