MozillaWiki

QA/Shield: Difference between revisions

Page Discussion

QA/Shield (view source)

Revision as of 17:00, 20 April 2017

3,995 bytes added ,  20 April 2017
→‎Checklist
 
(40 intermediate revisions by the same user not shown)
Line 10: Line 10:
|-
|-
| 10/10/2016 || 1.1 || Adrian Florinescu || Updates
| 10/10/2016 || 1.1 || Adrian Florinescu || Updates
|-
| 11/10/2016 || 1.2 || Adrian Florinescu || Updates
|-
| 17/10/2016 || 1.3 || Adrian Florinescu || Updates
|-
| 02/21/2017 || 2.0 || Adrian Florinescu || Updates
|-
| 03/27/2017 || 2.1 || Adrian Florinescu || Updates
|}
|}


Line 15: Line 23:


== Scope ==
== Scope ==
This wiki details the testing that will be performed by the QA-Eng team for the SHIELD project. SHIELD is a system addon that addresses user attrition and satisfaction in Firefox. User facing products that are included in SHIELD and on which the current test plan will focus are:
This wiki details the testing that will be performed by the QA-Eng team for the SHIELD project. SHIELD is a system addon that addresses user attrition and satisfaction in Firefox.  
* '''Shield Studies''' - Recruits users to participate in testing new features and ideas that help us validate product decisions.
QA focus shall be given mainly to:
* '''Heartbeat''' - Gathers feedback from users for us to better understand what users find helpful and useful
* '''Self Repair''' - Detects problems with Firefox and suggest fixes to the user
* '''Contextual Feature Recommendation''' - Allows users to discover features and addons based on how they use the product
 
One of the main focuses of the QA team will be the user facing products listed above, but QA focus shall be given to:
* SHIELD Recipe management  
* SHIELD Recipe management  
* SHIELD Control center
* SHIELD Control center (recipe administration center)
* Delivery system for the Shield recipes: how and if the filter expressions are targeting a certain niche of the FF users.
* Delivery system for the Shield recipes: how and if the filter expressions are targeting a certain niche of the FF users.
 
 


== Ownership ==
== Ownership ==
Line 36: Line 37:


PM, QA Engineeting Team - [mailto:rares.bologa@softvisioninc.eu Rares Bologa]<br />
PM, QA Engineeting Team - [mailto:rares.bologa@softvisioninc.eu Rares Bologa]<br />
QA Engineering Team - [mailto:adrian.florinescu@softvisioninc.eu Adrian Florinescu] <br />
QA Engineering Team - [mailto:adrian.florinescu@softvision.ro Adrian Florinescu] <br />
QA Engineering Team - [mailto:ovidiu.boca@softvision.ro Ovidiu Boca] <br />


= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
Detail what is in scope from a testing perspective for the project team.
Detail what is in scope from a testing perspective for the project team:
 
* Testing the admin interface : control center -> basic functionality of creating/updating a recipe from the functional point of view
* Minimal recipe delivery: should cover integration with the Firefox client
* Basic recipe filters: the testing will cover that the recipes are delivered and executed to the intended bucket of people.
* Basic performance variations on FF browser: the testing shall cover basic scenarios to identify that broken or extreme recipes do not affect negatively FF


=== Out of Scope ===
=== Out of Scope ===
Out of the SHIELD two main components: Client Side and Server Side, the server side named Normandy does not fall under the scope of the current Test Plan.
* Control Center
* Peer Review
* Preferences


= Requirements for testing =
= Requirements for testing =
Line 57: Line 66:
=== Nightly ===
=== Nightly ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
The initial tests that shall be conducted in the incipient testing phases will require a Normandy local setup (docker compose), Normandy-addon and several FF Client settings as follows. We expect these to be replaced by configurations that are delivered by Shield as a System Add-on.
The System Add-on is to be tested with two configurations:
 
*1. Mock test recipes - emulation of recipes
 
Documentation on: https://normandy-mock.dev.mozaws.net/
 
#Obtain a copy of Firefox with the SHIELD recipe client system add-on installed. You can check about:support to ensure that you have it.
#Set the extensions.shield-recipe-client.dev_mode preference to true to run recipes immediately on startup.
#Set the extensions.shield-recipe-client.logging.level preference to 0 to enable more logging.
#Set the security.content.signature.root_hash preference to 4C:35:B1:C3:E3:12:D9:55:E7:78:ED:D0:A7:E7:8A:38:83:04:EF:01:BF:FA:03:29:B2:46:9F:3C:C5:EC:36:04. This is the public hash used for verifying content signatures of the test data.
#The preference value for extensions.shield-recipe-client.api_url should be set for the mock recipe you want to run.




# Normandy - docker compose  (for setting and configuring the environment see [http://normandy.readthedocs.io/en/latest/qa/setup.html QAsetup])
*2. Staging server - using Shield System addon and the staging Control Center
## Hosts the control center
 
## Delivers recipes to target clients
# You need VPN access.
#Obtain a copy of Firefox with the SHIELD recipe client system add-on installed. You can check about:support to ensure that you have it.
#Set the extensions.shield-recipe-client.dev_mode preference to true to run recipes immediately on startup.
#Set the extensions.shield-recipe-client.logging.level preference to 0 to enable more logging.
#Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
#Set extensions.shield-recipe-client.api_url to https://normandy-admin.stage.mozaws.net/api/v1
#To access control center, https://normandy-admin.stage.mozaws.net/control/.


# FireFox Client
## Normandy add-on is required (the git for it is [https://github.com/mozilla/normandy-addon Git_normandy-addon ])  which must be installed on the client FF
## Several preferences must also be set on the Client FF as follows where localhost must be replaced with the ip of the docker-compose Normandy:
### browser.selfsupport.url = "https://localhost:8443/%LOCALE%/repair"
### browser.uitour.testingOrigins = "https://localhost:8443"
### extensions.recipeclient.api_url = "https://localhost:8443/api/v1"
### security.content.signature.root_hash = "4C:35:B1:C3:E3:12:D9:55:E7:78:ED:D0:A7:E7:8A:38:83:04:EF:01:BF:FA:03:29:B2:46:9F:3C:C5:EC:36:04"


</div>
</div>
Line 81: Line 99:
=== Beta ===
=== Beta ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
1. Set the preference value for extensions.systemaddon.update.url set to https://aus5.mozilla.org/update/3/SystemAddons/53.0.0/20160922113459/default/en-US/beta-sysaddon/default/default/default/update.xml
2. Go to Tools -> Developer Tools -> Web Console -> Open "Toolbox Options" from "Advanced settings" mark as checked the last 3 available check boxes. (browser chrome....; remote debugging; worker debugging)
3. Open Browser Console and run this script: Components.utils.import("resource://gre/modules/AddonManager.jsm"); AddonManagerPrivate.backgroundUpdateCheck();
4. Set the preference value for extensions.shield-recipe-client.api_url set to https://normandy.stage.mozaws.net/api/v1
5. Set the extensions.shield-recipe-client.dev_mode preference to true to run recipes immediately on startup.
6. Set the extensions.shield-recipe-client.logging.level preference to 0 to enable more logging.
7. Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
8. Set the preference value for extensions.shield-recipe-client.api_url set to https://normandy.stage.mozaws.net/api/v1
 
For mock recipes:
 
8'. Set the security.content.signature.root_hash preference to 4C:35:B1:C3:E3:12:D9:55:E7:78:ED:D0:A7:E7:8A:38:83:04:EF:01:BF:FA:03:29:B2:46:9F:3C:C5:EC:36:04. This is the public hash used for verifying content signatures of the test data.
</div>
</div>


Line 97: Line 126:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners  
! Ref !! Function !! Test Objective !! Test Type !! Risk Assessment and Coverage(RAC) !! Owners  
|-
| TO-1 || Control Center || To verify basic functionality of the admin interface  || Manual || RAC-3 ||  Eng Team
|-
| TO-2 || Recipe fetching and execution || To verify that recipes are fetched and executed when required || Manual ||  ||  Eng Team
|-
|-
| 1 || Control Center || Admin interface allows admin users to add/edit recipes  || Basic configuration actions are possible  || Manual/Usability || Eng Team
| TO-3 || Filter validation || To verify that recipes are executed only when filter expression returns true || Manual || RAC-1 || Eng Team
|-
|-
| 2 || Recipe execution || Recipes created in the admin interface are delivered to the Client FF || The FF Client receives and executes the recipe || Manual || Eng Team
| TO-4 || Heartbeat basic functionality || To verify the basic functionality of a heartbeat  || Manual || RAC-4, RAC-5 || Eng Team
|-
|-
| 3 || Peer approval || Recipes created in the admin interface need peer approval to be enabled || Each recipe created in the admin interface cannot be run without peer approval || Manual || Eng Team
| TO-5 || Recipes/Heartbeat && FF integration || To verify that recipe execution and heartbeat display doesn't affect performance and visual expectations || Manual || RAC-4 ||  Eng Team
|-
|-
| 4 || Filter Expressions || Each recipe targets a particular bucket defined in filter expressions area in admin || Only FF matching the filter expressions will run the recipe || Manual || Eng Team
| TO-6 || Show heartbeat defaults || Checks: one heartbebat shown per 24hrs, display heartbeat again after 7 days + the other admin recipes display settings  || Manual || - || Eng Team
|-
|-
| 5 || Sampling || Basic check on client running the recipe ratio || Heuristic approach to do a basic checking of the sampling set in the recipe|| Manual || Eng Team
| TO-7 || Sampling || Basic functionality of recipe ratio/buckets || Manual || - ||  Eng Team
|-
| TO-8 || Test heartbeat/recipes edge cases - mock recipes || Covers mock testing of edge case scenarios || Manual || RAC-4 ||  Eng Team
|}
 
== Risk Assessment and Coverage ==


{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Recipes are not ran when filter == true || TO-03 || '''3-High''' || '''3-High''' || '''3-High''' || '''27'''
|-
| RAC-2 || Filter conditions are not returning the expected results || TO-3 || '''3-High''' || '''3-High''' || '''3-High''' || '''27'''
|-
| RAC-3 || Admin validation allows recipes values that break recipe workflow || TO-01 || '''3-High''' || '''3-High''' || 1- Low || ''9''
|-
| RAC-4 || Edge cases might break FF/Shield || TO-04, TO-05, TO-8 || ''2-Moderate'' || '''3-Almost Certain''' || ''2-Medium'' || ''12''
|-
| RAC-5 || about: pages cannot be used as Heartbeat links || TO-04 || ''2-Moderate'' || ''2-Possible'' || ''2-Medium'' || 8
|}
|}
'''Values:'''
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


== Builds ==
== Builds ==
Line 124: Line 188:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | 10.05.2016 ||  
|style="text-align:center;" | 10.05.2016 || 04.20.2017
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" | 10.05.2016 ||  
|style="text-align:center;" | 10.05.2016 || 04.20.2017
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 12.05.2016  ||  
|style="text-align:center;" | 12.05.2016  || 04.20.2017
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | 23.06.2016 ||  
|style="text-align:center;" | 06.23.2016 || 20.04.2017
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 11.20.2016 || 01.30.2016
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
|style="text-align:center;" | ||
|style="text-align:center;" | 01.30.2016 || 03.25.2017
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 03.25.2017 || 04.20.2017
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | ||  
|style="text-align:center;" | TBD || TBD
|}
|}


Line 160: Line 224:
| Test case execution || Test Rail
| Test case execution || Test Rail
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla / Git
|}
|}


= Status =
== Overview ==
Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta
== Risk analysis ==
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix
= References =
= References =
* List and links for specs
* List and links for specs
Line 184: Line 237:
= Testcases =  
= Testcases =  
== Overview ==
== Overview ==
  Summary of testing scenarios
  Summary of testing scenarios - TBD


== Test Areas ==
== Test Areas ==
{| class="wikitable" style="width:80%"
{| class="wikitable" style="width:80%"
|-
|-
! Test Areas !! Covered !! Details
! Test Areas !! Covered !! Details !! Reviewer
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || || Osmose, Mythmon
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || || Osmose, Mythmon
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || || Osmose, Mythmon
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || || Osmose, Mythmon
|-
|-
| '''UI'''  
| '''UI'''  
||  ||  
||  || ||
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || || Osmose, Mythmon
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes || || Osmose, Mythmon
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || the heartbeat bar shouldn't zoom (same behavior as the url bar) || Osmose, Mythmon
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | Yes || Is this feature user friendly || Osmose, Mythmon


|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
||  ||  
||  || ||
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | No || future plans to have something related to this, but not in the first versions || Osmose, Mythmon
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | Yes || Make sure support documents are written and are correct. || Osmose, Mythmon


|-
|-
| '''Install/Upgrade'''  
| '''Install/Upgrade'''  
||  ||  
||  || ||
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | No || System addon || Osmose, Mythmon
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | Yes || at least some tcs ran for first run || Osmose, Mythmon
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" |  || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | Yes ||  || Osmose, Mythmon


|-
|-
| ''' Enterprise '''  
| ''' Enterprise '''  
||  ||  Raise up the topic to developers to see if they are expecting to work different on ESR builds
||  ||  Raise up the topic to developers to see if they are expecting to work different on ESR builds ||
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | ||  
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | ||  
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | ||
|style="text-align:center;" | No || || Osmose, Mythmon


|-
|-
| ''' Data Monitoring '''  
| ''' Data Monitoring '''  
||  ||  
||  || ||
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" |  || List of error conditions to monitor
|style="text-align:center;" | Yes ||  || Osmose, Mythmon
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || || Osmose, Mythmon
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || Chris H handles it || Osmose, Mythmon
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes || Chris H handles it || Osmose, Mythmon
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" |  ||
|style="text-align:center;" | No || || Osmose, Mythmon


|-
|-
| ''' Add-ons '''  
| ''' Add-ons '''  
||  || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
||  || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. ||
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  || Osmose, Mythmon
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || Pop-up blocker, privacy addons, profile json, test pilot integration testing: talk with BM team about effort || Osmose, Mythmon


|-
|-
| ''' Security '''  
| ''' Security '''  
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. || Osmose, Mythmon
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | TBD || osmoses to advice further on this || Osmose, Mythmon
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | TBD || osmoses to advice further on this || Osmose, Mythmon


|-
|-
| ''' Web Compatibility '''  
| ''' Web Compatibility '''  
||  || depends on the feature
||  || depends on the feature ||
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  || Osmose, Mythmon
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" |  ||  
|style="text-align:center;" | Yes || scenarios with heartbeat redirect with addon blocker and blocked site || Osmose, Mythmon


|-
|-
Line 332: Line 385:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  || Osmose, Mythmon
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No || || Osmose, Mythmon
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | yes || test if it's possible to send to the about:accounts as a survey page || Osmose, Mythmon
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/192BDmxxPirNZMxaH8a9bN4ak1m0sp3U1PXT-G-THpWk/edit?usp=sharing link]
  Full Test suite - Link within TestRail [https://testrail.stage.mozaws.net/index.php?/suites/view/232&group_by=cases:section_id&group_order=asc link]
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/192BDmxxPirNZMxaH8a9bN4ak1m0sp3U1PXT-G-THpWk/edit?usp=sharing link]
Regression Test suite - Link with the gdoc - if available/needed.


= Bug Work =
= Bug Work =
Tracking bug – [https://bugzilla.mozilla.org/show_bug.cgi?id=1275996 Bug 1275996 - [Tracker<nowiki>]</nowiki> SHIELD System Add-on v1.0]
Tracking bugs:
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1275996 Bug 1275996 - [Tracker<nowiki>]</nowiki> SHIELD System Add-on v1.0]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1308656 Bug 1308656 - [Land shield-recipe-client as system addon]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1342014 Bug 1342014 - [Shield<nowiki>]</nowiki> QA tracker for manual Testing Bug 1308656 - Land shield-recipe-client as system addon]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">


====== Bug fix verification ======
<div class="mw-collapsible-content">
{| class="wikitable" style="width:50%"
|-
! Bug No !! Summary !! Status !! Firefox Verion
|-
|  123 || bug summary || NEW || Nighly 45
|-
| || || ||
|}
</div></div>


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">


====== Logged bugs ======
====== Bug work ======
<div class="mw-collapsible-content">
*  {{Bug|1342014}} - QA tracker for manual Testing Bug 1308656 - Land shield-recipe-client as system addon]
Bug 111111<br/>
<bugzilla>
Bug 211111<br/>
    {
</div>
        "blocks":[1342014],
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>
</div>
</div>


Line 378: Line 423:
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
== Results ==
'''Nightly testing'''<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Regression Test suite, if needed/available
'''Merge to Aurora Sign-off'''
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite


== Checklist ==
== Checklist ==
Line 399: Line 430:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |  
| style="text-align:center;" | done 
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | done  ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | done ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | done ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  done ||
|style="text-align:center;" |
 
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" |  
|style="text-align:center;" |  yes ||
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | no ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | no || defects logging split between Bugzilla and Git
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | yes ||  
|-
|-
|  Daily Status Report (email/etherpad statuses/ gdoc with results) || ||  
|  Daily Status Report (email/etherpad statuses/ gdoc with results)  
|style="text-align:center;" | Yes || when required
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry
|style="text-align:center;" | N/A ||
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Nightly Release||
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent
|style="text-align:center;" | N/A ||  System Add-on
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A || System Add-on
|-
|-
|  QA Signoff  - Aurora Release ||
|  QA Signoff  - Aurora Release  
|style="text-align:center;"| Email to be sent
|style="text-align:center;" | N/A ||  System Add-on
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A || System Add-on
|-
|-
|  QA Signoff  - Beta Release ||
|  QA Signoff  - Beta Release  
|style="text-align:center;" | Email to be sent
|style="text-align:center;" | N/A ||  System Add-on
|}
|}
AdiFlorinescu
584

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1151022...1168876"