MozillaWiki

QA/Shield: Difference between revisions

Page Discussion

QA/Shield (view source)

Revision as of 17:00, 20 April 2017

2,574 bytes added ,  20 April 2017
→‎Checklist
 
(21 intermediate revisions by the same user not shown)
Line 15: Line 15:
| 17/10/2016 || 1.3 || Adrian Florinescu || Updates
| 17/10/2016 || 1.3 || Adrian Florinescu || Updates
|-
|-
| 21/02/2017 || 2.0 || Adrian Florinescu || Updates
| 02/21/2017 || 2.0 || Adrian Florinescu || Updates
|-
| 03/27/2017 || 2.1 || Adrian Florinescu || Updates
|}
|}


Line 49: Line 51:


=== Out of Scope ===
=== Out of Scope ===
-
* Control Center
* Peer Review
* Preferences


= Requirements for testing =
= Requirements for testing =
Line 95: Line 99:
=== Beta ===
=== Beta ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
1. Set the preference value for extensions.systemaddon.update.url set to https://aus5.mozilla.org/update/3/SystemAddons/53.0.0/20160922113459/default/en-US/beta-sysaddon/default/default/default/update.xml
2. Go to Tools -> Developer Tools -> Web Console -> Open "Toolbox Options" from "Advanced settings" mark as checked the last 3 available check boxes. (browser chrome....; remote debugging; worker debugging)
3. Open Browser Console and run this script: Components.utils.import("resource://gre/modules/AddonManager.jsm"); AddonManagerPrivate.backgroundUpdateCheck();
4. Set the preference value for extensions.shield-recipe-client.api_url set to https://normandy.stage.mozaws.net/api/v1
5. Set the extensions.shield-recipe-client.dev_mode preference to true to run recipes immediately on startup.
6. Set the extensions.shield-recipe-client.logging.level preference to 0 to enable more logging.
7. Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
8. Set the preference value for extensions.shield-recipe-client.api_url set to https://normandy.stage.mozaws.net/api/v1
 
For mock recipes:
 
8'. Set the security.content.signature.root_hash preference to 4C:35:B1:C3:E3:12:D9:55:E7:78:ED:D0:A7:E7:8A:38:83:04:EF:01:BF:FA:03:29:B2:46:9F:3C:C5:EC:36:04. This is the public hash used for verifying content signatures of the test data.
</div>
</div>


Line 111: Line 126:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners  
! Ref !! Function !! Test Objective !! Test Type !! Risk Assessment and Coverage(RAC) !! Owners  
|-
| TO-1 || Control Center || To verify basic functionality of the admin interface  || Manual || RAC-3 ||  Eng Team
|-
| TO-2 || Recipe fetching and execution || To verify that recipes are fetched and executed when required || Manual ||  ||  Eng Team
|-
|-
| 1 || Control Center || Admin interface allows admin users to add/edit recipes  || Basic configuration actions are possible  || Manual/Usability || Eng Team
| TO-3 || Filter validation || To verify that recipes are executed only when filter expression returns true || Manual || RAC-1 || Eng Team
|-
|-
| 2 || Recipe execution || Recipes created in the admin interface are delivered to the Client FF || The FF Client receives and executes the recipe || Manual || Eng Team
| TO-4 || Heartbeat basic functionality || To verify the basic functionality of a heartbeat  || Manual || RAC-4, RAC-5 || Eng Team
|-
|-
| 3 || Peer approval || Recipes created in the admin interface need peer approval to be enabled || Each recipe created in the admin interface cannot be run without peer approval || Manual || Eng Team
| TO-5 || Recipes/Heartbeat && FF integration || To verify that recipe execution and heartbeat display doesn't affect performance and visual expectations || Manual || RAC-4 ||  Eng Team
|-
|-
| 4 || Filter Expressions || Each recipe targets a particular bucket defined in filter expressions area in admin || Only FF matching the filter expressions will run the recipe || Manual || Eng Team
| TO-6 || Show heartbeat defaults || Checks: one heartbebat shown per 24hrs, display heartbeat again after 7 days + the other admin recipes display settings  || Manual || - || Eng Team
|-
|-
| 5 || Sampling || Basic check on client running the recipe ratio || Heuristic approach to do a basic checking of the sampling set in the recipe|| Manual || Eng Team
| TO-7 || Sampling || Basic functionality of recipe ratio/buckets || Manual || - ||  Eng Team
|-
| TO-8 || Test heartbeat/recipes edge cases - mock recipes || Covers mock testing of edge case scenarios || Manual || RAC-4 ||  Eng Team
|}


== Risk Assessment and Coverage ==
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Recipes are not ran when filter == true || TO-03 || '''3-High''' || '''3-High''' || '''3-High''' || '''27'''
|-
| RAC-2 || Filter conditions are not returning the expected results || TO-3 || '''3-High''' || '''3-High''' || '''3-High''' || '''27'''
|-
| RAC-3 || Admin validation allows recipes values that break recipe workflow || TO-01 || '''3-High''' || '''3-High''' || 1- Low || ''9''
|-
| RAC-4 || Edge cases might break FF/Shield || TO-04, TO-05, TO-8 || ''2-Moderate'' || '''3-Almost Certain''' || ''2-Medium'' || ''12''
|-
| RAC-5 || about: pages cannot be used as Heartbeat links || TO-04 || ''2-Moderate'' || ''2-Possible'' || ''2-Medium'' || 8
|}
|}
'''Values:'''
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


== Builds ==
== Builds ==
Line 138: Line 188:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | 10.05.2016 ||  
|style="text-align:center;" | 10.05.2016 || 04.20.2017
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" | 10.05.2016 ||  
|style="text-align:center;" | 10.05.2016 || 04.20.2017
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 12.05.2016  ||  
|style="text-align:center;" | 12.05.2016  || 04.20.2017
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | 23.06.2016 ||  
|style="text-align:center;" | 06.23.2016 || 20.04.2017
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 11.20.2016 || 01.30.2016
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
|style="text-align:center;" | ||
|style="text-align:center;" | 01.30.2016 || 03.25.2017
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 03.25.2017 || 04.20.2017
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | ||  
|style="text-align:center;" | TBD || TBD
|}
|}


Line 174: Line 224:
| Test case execution || Test Rail
| Test case execution || Test Rail
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla / Git
|}
|}


= Status =
== Overview ==
Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta
== Risk analysis ==
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix
= References =
= References =
* List and links for specs
* List and links for specs
Line 356: Line 395:


== Test suite ==
== Test suite ==
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/192BDmxxPirNZMxaH8a9bN4ak1m0sp3U1PXT-G-THpWk/edit?usp=sharing link]
  Full Test suite - Link within TestRail [https://testrail.stage.mozaws.net/index.php?/suites/view/232&group_by=cases:section_id&group_order=asc link]
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/192BDmxxPirNZMxaH8a9bN4ak1m0sp3U1PXT-G-THpWk/edit?usp=sharing link]
Regression Test suite - Link with the gdoc - if available/needed.


= Bug Work =
= Bug Work =
Tracking bug – [https://bugzilla.mozilla.org/show_bug.cgi?id=1275996 Bug 1275996 - [Tracker<nowiki>]</nowiki> SHIELD System Add-on v1.0]
Tracking bugs:
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1275996 Bug 1275996 - [Tracker<nowiki>]</nowiki> SHIELD System Add-on v1.0]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1308656 Bug 1308656 - [Land shield-recipe-client as system addon]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1342014 Bug 1342014 - [Shield<nowiki>]</nowiki> QA tracker for manual Testing Bug 1308656 - Land shield-recipe-client as system addon]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">


====== Bug fix verification ======
<div class="mw-collapsible-content">
{| class="wikitable" style="width:50%"
|-
! Bug No !! Summary !! Status !! Firefox Verion
|-
|  123 || bug summary || NEW || Nighly 45
|-
| || || ||
|}
</div></div>


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">


====== Logged bugs ======
====== Bug work ======
<div class="mw-collapsible-content">
*  {{Bug|1342014}} - QA tracker for manual Testing Bug 1308656 - Land shield-recipe-client as system addon]
Bug 111111<br/>
<bugzilla>
Bug 211111<br/>
    {
</div>
        "blocks":[1342014],
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>
</div>
</div>


Line 392: Line 423:
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
== Results ==
'''Nightly testing'''<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Regression Test suite, if needed/available
'''Merge to Aurora Sign-off'''
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite


== Checklist ==
== Checklist ==
Line 413: Line 430:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |  
| style="text-align:center;" | done 
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | done  ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | done ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | done ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  done ||
|style="text-align:center;" |
 
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" |  
|style="text-align:center;" |  yes ||
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | no ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | no || defects logging split between Bugzilla and Git
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | yes ||  
|-
|-
|  Daily Status Report (email/etherpad statuses/ gdoc with results) || ||  
|  Daily Status Report (email/etherpad statuses/ gdoc with results)  
|style="text-align:center;" | Yes || when required
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry
|style="text-align:center;" | N/A ||
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Nightly Release||
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent
|style="text-align:center;" | N/A ||  System Add-on
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A || System Add-on
|-
|-
|  QA Signoff  - Aurora Release ||
|  QA Signoff  - Aurora Release  
|style="text-align:center;"| Email to be sent
|style="text-align:center;" | N/A ||  System Add-on
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A || System Add-on
|-
|-
|  QA Signoff  - Beta Release ||
|  QA Signoff  - Beta Release  
|style="text-align:center;" | Email to be sent
|style="text-align:center;" | N/A ||  System Add-on
|}
|}
AdiFlorinescu
584

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1163534...1168876"