QA/Feature Testing v2: Difference between revisions

Jump to navigation Jump to search

QA/Feature Testing v2 (view source)

Revision as of 08:46, 6 August 2019

27,443 bytes added ,  6 August 2019
m
→‎History of changes: -
(-)
 
 
(12 intermediate revisions by the same user not shown)
Line 18: Line 18:
=== Overview ===
=== Overview ===
==== Phase I - PI request submission and QA resource allocation ====
==== Phase I - PI request submission and QA resource allocation ====
1. QA testing is requested by the expected deadline in Jira Service Desk
* <u>QA testing is requested by the expected deadline in Jira Service Desk</u>
* Deadline: [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk2]
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk2]
* Responsible: Eng/Product
** '''Responsible''': Eng/Product
* Description: A testing requests need to be filed directly in [https://moz-pi-test.atlassian.net/servicedesk/customer/portal/9 Jira Service Desk]. By default the submitted PI requests are to be considered for QA testing in the Beta cycle.
** '''Description''': A testing requests need to be filed directly in [https://moz-pi-test.atlassian.net/servicedesk/customer/portal/9 Jira Service Desk]. By default the submitted PI requests are to be considered for QA testing in the Beta cycle.
**
*** Project team should highlight if they need [https://docs.google.com/document/d/12Fp_TsREix8yjj8Ly1XRG_0x5grZYwhKoin5My74PSM/edit Nightly testing] or not (optional but recommended).
*** The Exception path for late submissions can be found [https://docs.google.com/document/d/129IEYhx8wdBltgn3ErBEPSBrzsWyHobZsjruwlHhxfw/edit#heading=h.9al51v705u2f here].
 
* <u>A quality engineer is assigned to the request, based on available resources and priorities</u>
** '''Responsible''': QA
** '''Description''': Once a quality engineer has been assigned, the associated Jira ticket will be updated and the PI requestor gets notified automatically.
 
==== Phase II - Feature documentation and pre-execution approvals ====
* <u>Feature technical documentation is provided by the expected deadline</u>
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk4]
*** Past due features will be called out and deferred by default to the next release.
*** Request for exception must have approvals from head of engineering AND product.
** '''Responsible''': Eng/Product
** '''Description''': The technical documentation should be provided based on this [https://docs.google.com/document/d/1h6F5A_PW4X-20zBmrftDtZwp0jdsl9oIyjaqGNFm8YQ/edit template] and should be complete and relevant to the work requested.
*** Once documentation is received, QA will set the status of the Jira ticket status to <code>TEST PLANNING</code>.
*** Any delay in submitting technical documentation will significantly impact QA’s ability to plan and test during the Beta cycle.
** '''Reference''': [https://docs.google.com/document/d/1h6F5A_PW4X-20zBmrftDtZwp0jdsl9oIyjaqGNFm8YQ/edit Guideline for Technical Documentation]
 
* <u>A feature kick-off meeting is scheduled once technical documentation is received</u>
** '''Soft deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk5]
** '''Responsible''': QA, Eng/Product
** '''Description''': The kick-off meeting is the perfect place for QA to get a better understanding of the scope of the feature and clarify any questions related to it.
*** Following the kick-off meeting, QA will set the status of the Jira ticket to <code>TEST CREATION</code>.
 
* <u>Test Plan and Test Cases are drafted and formal approvals requested</u>
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk5]
** Responsible: QA
** '''Description''': Documents are created for the Test Plan and High-level Test Cases. QA will provide the links and ask for formal approval via email.
*** Once the links are provided, QA will set the status of the Jira ticket to <code>WAITING FOR DEVELOPERS</code>.
** '''References''': [https://docs.google.com/document/d/1Dj3IYlmRpBtC9sSAmuecbpOgme_n6zu6eWZok7bMs5E/edit#heading=h.z6ne0og04bp5 Test Plan template], [https://docs.google.com/spreadsheets/d/1OleeB9WGjf5ufh_hot0TD91cWYFDY5C9uGAsLeRZKFw/edit#gid=0 Test Cases template]
 
* <u>Test Plan and Test Cases are reviewed and formal approvals are made</u>
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk7]
** '''Responsible''': Eng/Product
** '''Description''': Feedback and formal approval should be given on the email thread started by QA on this matter.
 
==== Phase III - Go/No-Go decision based on feature completeness ====
* <u>Feature achieves completeness by the expected deadline</u>
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk6]
** '''Responsible''': Eng/Product
** '''Description''': [https://docs.google.com/document/d/129IEYhx8wdBltgn3ErBEPSBrzsWyHobZsjruwlHhxfw/edit#heading=h.h4c44sntnkxl here]
*** The Exception path for late submissions can be found [https://docs.google.com/document/d/129IEYhx8wdBltgn3ErBEPSBrzsWyHobZsjruwlHhxfw/edit#heading=h.9al51v705u2f here].
 
* <u>QA provides a test report for the features for which Nightly testing was requested</u>
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk7]
** '''Responsible''': QA
** '''Description''': In the case of features for which Nightly testing was requested, a test report will be sent out highlighting concerns.
** '''Reference''': [https://docs.google.com/document/d/14GOeKqoNWBZFVlmbo1kqEJik3kvpvhhizVq0QJjmpkk/edit Nightly testing optional report template]
 
* <u>Go/No-Go decision is made based on feature completeness, quality and risk</u>
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Nightly, wk7]
** '''Responsible''': Eng/Product, Relman
** '''Description''': Go/no-go will be based on (input from) feature completeness (Eng/Product), quality (QA), and risk (Relman).
 
==== Phase IV - Preliminary feature quality assessment ====
* <u>QA testing begins in Beta</u>
** '''Responsible''': QA
** '''Description''': Test execution begins on the first builds available in [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Beta, wk1].
*** QA will also set the status of the Jira ticket to <code>TEST EXECUTION</code>, and provide regular updates regarding testing progress and new issues found as comments.
 
* <u>QA provides a pre-Release preliminary status report</u>
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Beta, wk5]
** '''Responsible''': QA
** '''Description''': Two weeks before the formal Pre-Release sign off, QA will provide a preliminary report to highlight concerns early on. The report will be sent out as an email, but it will also be posted as a comment in the Jira ticket.
*** If there are major concerns reported, a meeting with Engineering will likely be required to determine the next steps.
** '''Reference''': [https://docs.google.com/document/d/1iY25P1_AuIRagz1NBMJmDn_2XuzR_WEvHCaI7LUPCuY/edit Preliminary status report optional template]
 
==== Phase V - Pre-Release test execution, sign off and follow-ups ====
* <u>QA provides a formal, Pre-Release feature sign off</u>
** '''Deadline''': [https://docs.google.com/document/d/1yP4u_zyjdquqbsEDJoHbuDor1FnRWmTESL0KS-TP6u8/edit#heading=h.88b8irlyxcai Beta, wk7]
** '''Responsible''': QA
** '''Description''': The Pre-Release sign off represents the final in-depth testing conducted by QA before the merge to Release. The sign off will highlight concerns (if any) and provide a quality status that will be factored into the Go/No-Go decision that will be made by Release Management.
*** Once the Pre-Release test execution is done, QA will set the status of the Jira ticket to <code>SIGN-OFF EMAIL SENT</code>, and a summary of the report will be posted in it as well.
** '''Reference''': [https://docs.google.com/document/d/11sCQTPO2F2xWObcBkbXh4VPp9ampujujVvO7XJh7sqA/edit# Feature sign off mandatory template]
 
* <u>QA will continue to assist following the sign off, if follow-ups are required</u>
 
* <u>QA will set the status of the Jira ticket to <code>RESOLVED</code>, when the work is done</u>
 
=== Schedule and milestones ===
==== 6-week release cycle ====
{| class="wikitable"
! rowspan="2" style="font-weight:bold; background-color:#efefef;" | CYCLE
! rowspan="2" style="font-weight:bold; background-color:#efefef;" | MILESTONE
! colspan="6" style="font-weight:bold; background-color:#efefef;" | WEEK NO
|-
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 1
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 2
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 3
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 4
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 5
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 6
|-
| rowspan="4" style="font-weight:bold; background-color:#efefef; color:#6200c9;" | NIGHTLY
| PI request deadline
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| Feature technical documentation
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" | 
|-
| Feature kick-offs
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| Test Plan & Test Case formal approvals
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| rowspan="2" style="font-weight:bold; background-color:#efefef; color:#ce6301;" | BETA
| Pre-Release preliminary status report
| width="20px" style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
|-
| Pre-Release feature sign off
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" | 
|}
 
==== 7-week release cycle ====
{| class="wikitable"
! rowspan="2" style="font-weight:bold; background-color:#efefef;" | CYCLE
! rowspan="2" style="font-weight:bold; background-color:#efefef;" | MILESTONE
! colspan="7" style="font-weight:bold; background-color:#efefef;" | WEEK NO
|-
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 1
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 2
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 3
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 4
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 5
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 6
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 7
|-
| rowspan="4" style="font-weight:bold; background-color:#efefef; color:#6200c9;" | NIGHTLY
| PI request deadline
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| Feature technical documentation
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| Feature kick-offs
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| Test Plan & Test Case formal approvals
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| rowspan="2" style="font-weight:bold; background-color:#efefef; color:#ce6301;" | BETA
| Pre-Release preliminary status report
| width="20px" style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
|-
| Pre-Release feature sign off
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
|}
 
==== 8-week release cycle ====
{| class="wikitable"
! rowspan="2" style="font-weight:bold; background-color:#efefef;" | CYCLE
! rowspan="2" style="font-weight:bold; background-color:#efefef;" | MILESTONE
! colspan="8" style="font-weight:bold; background-color:#efefef;" | WEEK NO
|-
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 1
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 2
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 3
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 4
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 5
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 6
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 7
| width="20px"  style="text-align: center; font-weight:bold; background-color:#efefef; color:#656565;" | 8
|-
| rowspan="4" style="font-weight:bold; background-color:#efefef; color:#6200c9;" | NIGHTLY
| PI request deadline
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" | ⬤
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| Feature technical documentation
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" | ⬤
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| Feature kick-offs
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" | ⬤
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| Test Plan & Test Case formal approvals
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" | ⬤
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
| width="20px"  style="text-align: center; color:#6200c9;" |
|-
| rowspan="2" style="font-weight:bold; background-color:#efefef; color:#ce6301;" | BETA
| Pre-Release preliminary status report
| width="20px" style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" | ⬤
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
|-
| Pre-Release feature sign off
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" |
| width="20px"  style="text-align: center; color:#ce6301;" | ⬤
| width="20px"  style="text-align: center; color:#ce6301;" |
|}
 
=== Best practices ===
* <u>Getting help</u>
** Each feature’s QA owner should have a peer (helper) assigned to help.
*** Larger, more complex features can justify more than one QA peer.
 
* <u>Maintaining documentation</u>
** Internal Test Plan reviews and updates should occur periodically.
*** Feature Test Plans should be updated at least once a week to keep them relevant.
** Feature status updates should be provided periodically in the QA status documents associated to each Firefox version.
 
* <u>Bug tracking</u>
** Weekly checks should be made for the bugs reported in the wild.
*** Since there are so many environment variations (due to various software and hardware pairings), some bugs might only be uncovered by users that have very specific environment setups.
** In the case of highly complex features, a meta bug should be created to track all the issue reported by QA.
*** Having a separate meta bug for the issues reported by QA ensures a more efficient tracking, referencing and reporting.
** Highly severe bugs (critical, blockers) affecting a feature should be flagged using the <code>qablocker</code> keyword.
*** Using this keyword in addition to setting <code>needinfo?</code> flags for the right people is the most efficient way of raising major concerns.
 
* <u>Bug verification</u>
** A continuous monitoring process should be in place for new bug fixes.
*** This can be easily done by setting up Bugzilla queries, or something similar.
 
== Sign off overview ==
'''''The QA sign off is based on the quality status of the feature in question. The green, yellow or red quality status is a schedule-based conversation, NOT a general release-readiness status.'''''
 
=== Quality status evaluation criteria ===
The following aspects are taken into account when evaluating the quality status of a feature:
* percentage of completed implementation, based on Engineering’s commitment
* unresolved bugs count and severity
* failed tests count and failure reason
* blocked tests percentage and blocking reason
 
=== Quality statuses ===
==== Green ====
'''Status description'''<br/>
A <font style="background-color:green;">GREEN</font> quality status indicates a feature that meets all of QA's expectations.
 
'''Status reasons'''
* Everything is going according to our plan.
* Manual test results have met the acceptance criteria that was agreed upon with Engineering.
* All or major and critical test objectives agreed upon with Engineering have been covered.
* There are no issues severe enough to jeopardize the release of the feature.
 
'''Example'''
* [65] [desktop] [feature] Widevine 4.10.1196.0 - pre-Release QA sign off (GREEN) - [https://docs.google.com/document/d/1IG6VOQ3CN4rGESGYnyl71kZfaNRuXi90mIX0lwhUioU/edit link]
 
==== Yellow ====
'''Status description'''<br/>
A <mark>YELLOW</mark> quality status indicates a feature of questionable quality, meaning that there are issues or potential issues that need to be addressed by the Product, User Experience and Engineering teams as part of that feature’s Go/NoGo decision.
 
'''Status reasons'''
* Manual test results have ''mostly'' met the acceptance criteria that was agreed upon with Engineering, with a few minor to medium exceptions that need to be addressed.
* Only major and critical test objectives that were agreed upon with Engineering have been covered, minor ones have been left out.
* There are ''moderate issues'' that might jeopardize the release of the feature, but an input is required from Product, User Experience and/or Engineering to determine if they should block or not.
* Testing progress is behind the planned timeline and we are not able to cover everything on time.
 
'''Example'''
* [65] [desktop] [feature] Language Switcher - Pre-Release QA sign off (YELLOW) - [https://docs.google.com/document/d/1a1YRvIf2ISSVeuZq-9ZWEGoviZqAPiJq7BAvExpELzI/edit link]
 
==== Red ====
'''Status description'''<br/>
A <font style="background-color:red">RED</font> quality status indicates a feature of low quality, meaning that there are severe issues that need to be addressed by the Product, User Experience and Engineering teams as part of that feature’s Go/NoGo decision.
 
'''Status reasons'''
* There’s a ''big mismatch'' between manual test results and the acceptance criteria that was agreed upon with Engineering.
* Part of or all major and critical test objectives that were agreed upon with Engineering have not been covered.
* There are ''severe issues'' jeopardizing the release of the feature or there’s a big mismatch between what was planned for implementation and what was actually implemented.
* We are unable to cover important parts of the feature, due to resource constraints or hardware/environment requirements that we cannot meet, which means that those areas may have important issues that we don’t know about.
 
'''Example'''
* [60] [desktop] [feature] Normandy Telemetry - pre-Release QA sign off (RED) - [https://drive.google.com/open?id=1Wc26EOlZCIbgbEcfrStIE95qfSkTh-TbwHwm32Za1js link]
 
==== Pref'd OFF ====
'''Description'''<br/>
Features riding a specific Firefox version in a disabled state (via preference flip) will not be receiving a sign off report from QA. This is because there’s no real impact for the end user, and so an in-depth assessment of that feature’s quality is irrelevant for the Firefox version in question.
 
Instead of a sign off, QA will send out a regular test report describing what was considered in scope/covered.
 
'''Examples'''
* [63] [desktop] [feature] Block Autoplay v2 - Test Results - [https://docs.google.com/document/d/1AYL0fwNldYPVl5ElZFNYDqIOC-e4CxVH7eqAp3Z6Ki0/edit link]
 
=== Sign off template ===
* The mandatory sign off template for pref’d on features is available [https://docs.google.com/document/d/17WVL90Dl8wAQCs_9z4w8qtAhqQUbnBAw5oAurFd3AeI/edit#heading=h.l2sirhdp8k8s here].
* The mandatory sign off template for pref’d off features is available [https://docs.google.com/document/d/17WVL90Dl8wAQCs_9z4w8qtAhqQUbnBAw5oAurFd3AeI/edit#heading=h.y3lsu9pe3emj here].
* The guideline for preliminary test reports is available [https://docs.google.com/document/d/1iY25P1_AuIRagz1NBMJmDn_2XuzR_WEvHCaI7LUPCuY/edit here].
 
== References ==
* All QA documentation available for features, structured per Firefox version: [https://drive.google.com/open?id=0B-n79IWvmJYYQngxVjhiRUJ1bTA here].
* Feature QA status per Firefox version:
** 2019: [https://docs.google.com/spreadsheets/u/2/d/10iNQQ2tmr0FP49bqmAi66-4qhSjKiW5e6wNQ47E-P9A/edit#gid=0 69], [https://docs.google.com/spreadsheets/d/1ygCmrDTQmoN6ecyNlkPHft6O2tayhn3S1RN5ii_ER_8/edit#gid=0 68], [https://docs.google.com/spreadsheets/d/1J83OlNavefkzcOiy79iVDh6FrHQkpRFejRZ9u76lIVk/edit#gid=0 67], [https://docs.google.com/spreadsheets/d/1vPyFxPmnQt5YZThJlFaeyriLkeXU6sE6jgEHgAryUbo/edit#gid=0 66], [https://docs.google.com/spreadsheets/d/1-zxhm8GJf9fZ0kFbrfB5tzpfCcTqUNHVhDJaCfIr89o/edit#gid=0 65]
** 2018: [https://docs.google.com/spreadsheets/d/1z4_GjF-OHKihIeqfJGzH3NUC0hK3Ct8chyHOm8Nguo4/edit 64], [https://docs.google.com/spreadsheets/u/2/d/1IdIMWOrYymTVKgCPiPoxWQ3xlYBNbGX_UC2rAX1YRYY/edit 63], [https://docs.google.com/document/u/2/d/1rxisBtzIVZwSAuIIzQ5WCAUUYL2ZhtUQ9xiSBBAOCKY/edit 62], [https://docs.google.com/document/d/1IjualMcpheEbZsT7ujyEr0tg4XIpdaG4oRpu1MTUQw0/edit 61], [https://docs.google.com/document/d/1jLrir7BEF7nerypyqbbtdu_uL2IDRLjZqigSOeUMFho/edit 60], [https://docs.google.com/document/d/1HLCYgwnFFj3sIBybrFM19VhccSaLuqwXaRPaGHj1Ap0/edit 59]
** 2017: [https://docs.google.com/document/d/1HLCYgwnFFj3sIBybrFM19VhccSaLuqwXaRPaGHj1Ap0/edit 58], [https://docs.google.com/document/u/2/d/1e840tyKTmhBdlYxtpWsGnyVXgyKVSJxxv9p98mMDqEU/edit 57]
 
== History of changes ==
* '''Aug 6, 2019''' - (btot) Updated the High-Level Test Cases document template
* '''April 23, 2019''' - (avaida) Added an optional test report template for features requiring Nightly testing.
Andrei.vaida
Confirmed users
1,595

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1209069...1216000"

Navigation menu