QA/New DevTools Debugger Panel: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
(Created page with "'''Revision History''' This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is...")
 
(-)
 
(41 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
| Andrei Vaida || QA Manager || Product Integrity || n/a || [mailto:andrei.vaida@softvisioninc.eu Email]
|-
| Jason Laster|| Software Engineer || Engineering || n/a || [mailto:jlaster@mozilla.com Email]
|-
| Bryan Clark || EPM || Product Management || n/a || [mailto:clarkbw@mozilla.com Email]
|}
'''Revision History'''
'''Revision History'''


Line 7: Line 22:
! Date !! Version !! Author !! Description  
! Date !! Version !! Author !! Description  
|-
|-
| 11/23/2015 || 1.0 || Brindusa Tot || Created first draft
| 04/24/2017 || 1.0 || Hani Yacoub || Created first draft
|-
| 10/05/2016|| 1.1 || Adrian Florinescu ||  Added point : 3.2 Channel dependent settings (configs) and environment setup
|-
|-
| 02/28/2017|| 1.2 || Brindusa Tot || Move Risk Assessment and Coverage upper in the test plan, under the Test Strategy chapter
| 08/14/2017 || 1.1 || Emil Ghitta || Updates
|}
|}


= Overview =
= Overview =
This feature purpose is to convert to a brand new debugger front-end, with a treeview of source files, tabs for displaying the source content, a better file search, and it can work everywhere the web can! It’s just a web app. It’s able to talk both Firefox and Chrome protocol and can debug all sorts of targets.
== Purpose ==
== Purpose ==
Detail the purpose of this document. For example:
This document describes the following topics:
* The test scope, focus areas and objectives
* The test scope, focus areas and objectives
* The test responsibilities
* The test responsibilities
Line 27: Line 42:


== Scope ==
== Scope ==
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
This wiki details the testing that will be performed by the project team for the New '''DevTools Debugger Panel''' project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document <u>what will be tested</u> and <u>how testing will be performed</u>.
* What will be tested
* How testing will be performed


== Ownership ==
== Ownership ==
What resources are needed, and when (developers, QA, PMs working on the feature)
 
*<big>Development Team</big>''':<br />
**Product Manager: [mailto:clarkbw@mozilla.com Bryan Clark] (irc: clarkbw)<br />
**Engineering Manager: [mailto:jlaster@mozilla.com Jason Laster] <br />
 
*<big>Engineering Team</big> - Nightly Testing''':<br />
**[mailto:rares.bologa@softvisioninc.eu Rares Bologa] (irc: RaresB) - PM for QA team <br />
**[mailto:brindusa.tot@softvisioninc.eu Brindusa Tot] (irc: brindusat) - leading QA efforts <br />
**[mailto:hani.yacoub@softvision.ro Hani Yacoub] (irc: hani) - QA <br />
**[mailto:ovidiu.boca@softvision.ro Ovidiu Boca] (irc: Ovidiu) - QA <br />
 
*<big>Release Team</big> - Beta/Release Testing''':<br />
** [mailto:andrei.vaida@softvisioninc.eu Andrei Vaida] (irc: avaida) - TL for QA team
** [mailto:emil.ghitta@softvision.ro Emil Ghitta] (irc: emilghitta) - QA
** [mailto:cristian.comorasu@softvision.ro Cristian Comorasu] (irc: CristiComo) - QA peer
 
= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
Detail what is in scope from a testing perspective for the project team.
The testing effort for the New DevTools Debugger Panel will be invested on the following areas:
 
* functionality: basic and advanced functionality to be verified according to the existing requirements;
* usability: new devtools debugger panel must be easy to use and straightforward;
* accessibility: new devtools debugger panel can be used with High Contrast Theme, Keyboard only or Screen Reader;
 
In the scope of current testing are following critical areas:
 
* Breakpoints
* Watch Expressions
* Call Stack
* Scopes
* Editor Search
* Toolbar


=== Out of Scope ===
=== Out of Scope ===
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.
Testing will <u>not</u> check:
 
* Security
* Telemetry data correctness
= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Specify OSes that need to be covered
*'''Operating Systems'''
Specify Devices that need to be covered
** Windows 10 x64, Windows 7 x32
Specify other configuration/environmental setup needed
** Mac OS X 10.12
 
** Ubuntu 16.04
* '''Devices''' : n/a
*'''Other configuration/environmental setup needed''': n/a


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
There are no special settings needed to test this feature.
 
=== Nightly ===
<div class="mw-collapsible-content">
text
</div>
 
=== Aurora ===
<div class="mw-collapsible-content">
text
</div>
 
=== Beta ===
<div class="mw-collapsible-content">
text
</div>
 
=== Post Beta / Release ===
<div class="mw-collapsible-content">
text
</div>
</div>


= Test Strategy =  
= Test Strategy =  
Line 80: Line 103:
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
|-
|-
| RAC-1 || Risk description 1 || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
| RAC-2 || Adding or removing a breakpoint might not be done correctly || TO-1 || 3-High || 2-Possible || 3-High || 18
|-
|-
| RAC-2 || Risk description 2 || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
| RAC-3 || To search for files or to search within a file might not be done correctly || TO-2 || 2-Moderate || 2-Possible || 3-High || 12
|-
|-
| RAC-3 || Risk description 3 || TO-2 || 2-Moderate || 2-Possible || 3-High || 12
| RAC-4 || Toolbar buttons might have different behavior || TO-2 || 2-Moderate || 2-Possible || 3-High || 12
|}
|}


Line 99: Line 124:
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
'''RAC''': Refers to the "Risk Assessment and Coverage" item that is covering.


== Test Objectives ==
== Test Objectives ==
Line 106: Line 133:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
! Ref !! Function !! Evaluation Criteria !! Test Objective !! Test Type !! RAC !! Owners  
|-
| TO-1 || Breakpoints || The breakpoint suspends the execution of the program || Verifying that the breakpoints successfully suspend the program's execution || Manual || RAC-1, RAC-2,||  Eng Team
|-
| TO-2 || Watch Expressions || The debugger will watch the expression and return any results when executions are paused|| Verifying that the debugger successfully returns the watch expressions  || Manual ||  RAC-1, RAC-2 || Eng Team
|-
| TO-3 || Call Stack || Each level of the call stack gets a line, with the name of the function, the filename and the line number. Clicking the line opens that source in the source pane. || Verifying that the call stack displays the name,filename, line number of the function and if it opens the source in the source pane || Manual || RAC-1, RAC-2 ||  Eng Team
|-
| TO-4 || Scopes  || At pause, the debugger give the option to expand the section in order to see all objects that are in scope at that point in the program || Verifying that the option to expand the section in order to see the all the objects is available and to check if the most local appears first and if the global scope appears last || Manual || RAC-1, RAC-2 ||  Eng Team
|-
|-
| 1 || Name of the feature or sub-function being tested || The objective the test is trying to demonstrate || The criteria that will be evaluated to demonstrate the test is successful || Manual/ Automation/ Regression/ Performance/ Usability/ Security/ Telemetry || RAC-1, RAC-2, RAC-3 || Eng Team
| TO-5 || Editor Search  || The debugger gives has the option to search for certain files or to search withing a certain file || Verifying that searches can be successfully performed for both files or for elements found inside certain files  || Manual || RAC-1, RAC-2, RAC-3 || Eng Team
|-
|-
| 2 || Repeat for each feature/sub-function || || || || ||
| TO-6 || Toolbar  || The debugger gives the possibility of different functions (Play/pause, Step over, Step in, Step out) when the debugger has stopped at a brakepoint || Verifying that the Play/pause, Step over, Step in , Step out functions work as intended  || Manual || RAC-1, RAC-2, RAC-4 || Eng Team
|-
|-
| 3 || || || || || ||
|}
|}


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly builds
* Links for Nightly builds - [http://archive.mozilla.org/pub/firefox/nightly/2017/04/2017-04-25-03-02-21-mozilla-central// link]
* Links for Aurora builds
* Links for Beta builds - [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ link]
* Links for Beta builds


== Test Execution Schedule ==
== Test Execution Schedule ==  
The following table identifies the anticipated testing period available for test execution.
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
Line 128: Line 161:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | ||  
|style="text-align:center;" | 04/24/2017 ||  
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" | ||  
|style="text-align:center;" | 04/24/2017 || 07/07/2017
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | ||  
|style="text-align:center;" | 04/24/2017 || 04/26/2017
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | ||  
|style="text-align:center;" | 05/08/2017 || 06/21/2017
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 05/10/2017 || 07/28/2017
|-
| QA - Aurora Testing
|style="text-align:center;" |  || 
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 08.07.2017 || 09.26.2017
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | ||
|style="text-align:center;" colspan="2" | 09.26.2017
|}
|}


Line 169: Line 199:
= Status =  
= Status =  
== Overview ==
== Overview ==
Track the dates and build number where feature was released to Nightly
* Feature was released to Nightly - [https://archive.mozilla.org/pub/firefox/nightly/2016/09/2016-09-13-03-04-25-mozilla-central/ (2016-09-13)]
Track the dates and build number where feature was merged to Aurora
* Feature was merged to Beta - [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ 56.0b1 (2017-08-07)]
Track the dates and build number where feature was merged to Release/Beta
* Feature was merged to Release - [https://archive.mozilla.org/pub/firefox/candidates/56.0-candidates/build1/win64/en-US/ (2017-09-19)]




= References =
= References =
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1294139 1294139] - [debugger.html] Turn new debugger on by default for all channels
* List and links for specs
* List and links for specs
  List and links for available specs - documents, user stories, specifications
**[https://projects.invisionapp.com/share/AY8OEG3C6#/screens/117937554_Debugger Debugger Design]
* Meta bug
 
= Test Cases =
= Testcases =  
Full Test Suites can be found [https://testrail.stage.mozaws.net/index.php?/suites/view/924&group_by=cases:section_id&group_order=asc TestRail] or in the PDF doc.
== Test Areas ==
== Test Areas ==
{| class="wikitable" style="width:80%"
{| class="wikitable" style="width:80%"
Line 186: Line 217:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 201: Line 232:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction (scroll, zoom)  
| Interaction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | Yes || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | Yes || Is this feature user friendly
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes ||
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 225: Line 256:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | Yes || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | Yes || Make sure support documents are written and are correct.


|-
|-
Line 235: Line 266:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | No || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | No || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | No || yes/no options, add comment with details about who will lead testing


|-
|-
Line 254: Line 285:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 270: Line 301:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | No || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 289: Line 320:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 305: Line 336:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 318: Line 349:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 328: Line 359:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link to test rail - tascases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]
  Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/924 Test Rail]
Smoke Test suite - Link with the tests - if available/needed.
Regression Test suite - Link with the tests - if available/needed.


= Bug Work =
= Bug Work =
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=1294139 1294139 - [debugger.html] Turn new debugger on by default for all channels]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1294139 1294139 ] )======


<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[12345],
         "blocks":[1294139],
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
     }
     }
Line 360: Line 389:


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bug fix verification ======
 
====== Verified bug fixes ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[12345],
         "blocks":[1294139],
         "resolution":"FIXED",
         "resolution":"FIXED",
         "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
Line 380: Line 410:


== Results ==
== Results ==
'''Nightly testing'''<br />
*[ Link for the tests run ]
 
** [https://testrail.stage.mozaws.net/index.php?/plans/view/4064 Mid-Nightly56]  
List of OSes that will be covered by testing<br />
**[https://testrail.stage.mozaws.net/index.php?/plans/view/4064 Pre-Beta56]
*Link for the tests run
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]
** Daily Smoke, lif needed/available
** Regression Test suite, if needed/available
 
'''Merge to Aurora Sign-off'''
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite


== Checklist ==
== Checklist ==
Line 399: Line 420:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |   
| style="text-align:center;" |  {{mdone|}} ||  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | N/A ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  {{mdone|}} ||   
|style="text-align:center;" |   
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" |  
|style="text-align:center;" | N/A ||
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | <b>In Progress</b> ||  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | <b>In Progress</b> ||  
|-
|-
|  Metrics/Telemetry||
|  Metrics/Telemetry  
|style="text-align:center;" |  
|style="text-align:center;" | N/A ||
|-
|-
|  QA Signoff  - Nightly Release||
|  QA Beta - Full Testing
|style="text-align:center;" | Email to be sent
|style="text-align:center;" | {{mdone|}} ||  
|-
|  QA Aurora - Full Testing  
|style="text-align:center;" | ||  
|-
|-
|  QA Signoff  - Aurora Release ||
|  QA Signoff  - Pre Beta 
|style="text-align:center;"| Email to be sent
|style="text-align:center;" | {{mdone|}} || [[https://drive.google.com/open?id=0B94iuIVMr-TEX1l3Vnh4RzNhNnc Email sent in 7/28/2017]]
|-
| QA Beta - Full Testing
|style="text-align:center;" |  ||
|-
|-
| QA Signoff - Beta Release ||
| QA Signoff - Pre Release  
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | {{mdone|}} || [[https://drive.google.com/open?id=0B94iuIVMr-TERW5zUERuUk0tRkE Email sent in 9/8/2017]]
|}
|}

Latest revision as of 11:46, 29 September 2017

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Andrei Vaida QA Manager Product Integrity n/a Email
Jason Laster Software Engineer Engineering n/a Email
Bryan Clark EPM Product Management n/a Email

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
04/24/2017 1.0 Hani Yacoub Created first draft
08/14/2017 1.1 Emil Ghitta Updates

Overview

This feature purpose is to convert to a brand new debugger front-end, with a treeview of source files, tabs for displaying the source content, a better file search, and it can work everywhere the web can! It’s just a web app. It’s able to talk both Firefox and Chrome protocol and can debug all sorts of targets.

Purpose

This document describes the following topics:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the New DevTools Debugger Panel project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document what will be tested and how testing will be performed.

Ownership

Testing summary

Scope of Testing

In Scope

The testing effort for the New DevTools Debugger Panel will be invested on the following areas:

  • functionality: basic and advanced functionality to be verified according to the existing requirements;
  • usability: new devtools debugger panel must be easy to use and straightforward;
  • accessibility: new devtools debugger panel can be used with High Contrast Theme, Keyboard only or Screen Reader;

In the scope of current testing are following critical areas:

  • Breakpoints
  • Watch Expressions
  • Call Stack
  • Scopes
  • Editor Search
  • Toolbar

Out of Scope

Testing will not check:

  • Security
  • Telemetry data correctness

Requirements for testing

Environments

  • Operating Systems
    • Windows 10 x64, Windows 7 x32
    • Mac OS X 10.12
    • Ubuntu 16.04
  • Devices : n/a
  • Other configuration/environmental setup needed: n/a

Channel dependent settings (configs) and environment setups

There are no special settings needed to test this feature.

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Mouse and keyboard actions may not be recognized TO-1 2-Moderate 1-Unlikely 3-High 6
RAC-2 Adding or removing a breakpoint might not be done correctly TO-1 3-High 2-Possible 3-High 18
RAC-3 To search for files or to search within a file might not be done correctly TO-2 2-Moderate 2-Possible 3-High 12
RAC-4 Toolbar buttons might have different behavior TO-2 2-Moderate 2-Possible 3-High 12

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

RAC: Refers to the "Risk Assessment and Coverage" item that is covering.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Evaluation Criteria Test Objective Test Type RAC Owners
TO-1 Breakpoints The breakpoint suspends the execution of the program Verifying that the breakpoints successfully suspend the program's execution Manual RAC-1, RAC-2, Eng Team
TO-2 Watch Expressions The debugger will watch the expression and return any results when executions are paused Verifying that the debugger successfully returns the watch expressions Manual RAC-1, RAC-2 Eng Team
TO-3 Call Stack Each level of the call stack gets a line, with the name of the function, the filename and the line number. Clicking the line opens that source in the source pane. Verifying that the call stack displays the name,filename, line number of the function and if it opens the source in the source pane Manual RAC-1, RAC-2 Eng Team
TO-4 Scopes At pause, the debugger give the option to expand the section in order to see all objects that are in scope at that point in the program Verifying that the option to expand the section in order to see the all the objects is available and to check if the most local appears first and if the global scope appears last Manual RAC-1, RAC-2 Eng Team
TO-5 Editor Search The debugger gives has the option to search for certain files or to search withing a certain file Verifying that searches can be successfully performed for both files or for elements found inside certain files Manual RAC-1, RAC-2, RAC-3 Eng Team
TO-6 Toolbar The debugger gives the possibility of different functions (Play/pause, Step over, Step in, Step out) when the debugger has stopped at a brakepoint Verifying that the Play/pause, Step over, Step in , Step out functions work as intended Manual RAC-1, RAC-2, RAC-4 Eng Team

Builds

This section should contain links for builds with the feature -

  • Links for Nightly builds - link
  • Links for Beta builds - link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 04/24/2017
Study documentation/specs received from developers 04/24/2017 07/07/2017
QA - Test plan creation 04/24/2017 04/26/2017
QA - Test cases/Env preparation 05/08/2017 06/21/2017
QA - Nightly Testing 05/10/2017 07/28/2017
QA - Beta Testing 08.07.2017 09.26.2017
Release Date 09.26.2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview


References

  • Meta bug 1294139 - [debugger.html] Turn new debugger on by default for all channels
  • List and links for specs

Test Cases

Full Test Suites can be found TestRail or in the PDF doc.

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) No
Interaction (scroll, zoom) No
Usable with a screen reader Yes e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
RTL build testing Yes
Help/Support
Help/support interface required Yes Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Yes Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected No
Does sync work across upgrades No
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding No Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing No yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration No
Network proxies/autoconfig No
ESR behavior changes No
Locked preferences No
Data Monitoring
Temporary or permanent telemetry monitoring No List of error conditions to monitor
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites No
Survey of many sites for compatibility No
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test suite

Full Test suite - Test Rail

Bug Work

Meta bug: 1294139 - [debugger.html Turn new debugger on by default for all channels]

Logged bugs ( blocking 1294139 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1294103 -- Debugger James Long (:jlongster) Load source-map and pretty-print workers on the client-side in new debugger RESOLVED Firefox 51
1294686 P1 Debugger James Long (:jlongster) Mochitests for new debugger RESOLVED Firefox 51
1294749 P1 Debugger James Long (:jlongster) Share modules with the rest of devtools RESOLVED ---
1295321 P2 Debugger New debugger should work with light/dark theme RESOLVED ---
1295753 -- Debugger Jason Laster [:jlast] New debugger can break console RESOLVED Firefox 51
1296048 -- Debugger James Long (:jlongster) Scope mozilla-specific CodeMirror CSS RESOLVED Firefox 51
1299602 P2 Debugger Debugger fails to emit tabNavigated event when paused RESOLVED Firefox 51
1300820 P1 Debugger Brian Grinstead [:bgrins] Always use 'old' debugger frontend in Browser Toolbox RESOLVED Firefox 51
1300861 P1 Debugger Brian Grinstead [:bgrins] Enable new debugger frontend in nightly builds RESOLVED Firefox 51
1300865 -- Debugger Always use 'old' debugger frontend when addon debugging RESOLVED ---
1300866 P1 Debugger James Long (:jlongster) Add a `devtoolsRequire` hook to expose native modules RESOLVED Firefox 51
1301147 -- Debugger James Long (:jlongster) Use BrowserLoader require in new debugger so we can share frontend modules RESOLVED Firefox 51
1301207 P1 Debugger James Long (:jlongster) Load theme switcher in new debugger RESOLVED Firefox 51
1301719 P1 Debugger Jason Laster [:jlast] Fix new debugger in remote toolboxes RESOLVED Firefox 51
1302833 P1 Debugger James Long (:jlongster) Fix CodeMirror height issue in new debugger RESOLVED Firefox 51
1302862 P2 Debugger James Long (:jlongster) Open the panel when the sources have loaded RESOLVED Firefox 52
1302872 P1 Debugger James Long (:jlongster) Update new debugger frontend (9/14/2016) RESOLVED Firefox 51
1303045 P1 Debugger James Long (:jlongster) Fix `viewSourceInDebugger` to highlight specific line RESOLVED Firefox 51
1303177 P1 Debugger James Long (:jlongster) Turn off sourcemaps if using the new debugger frontend RESOLVED Firefox 51
1303862 P1 Debugger James Long (:jlongster) Update new debugger frontend (9/19/2016) RESOLVED Firefox 52
1303866 P1 Debugger James Long (:jlongster) Don't track sourcemapped sources if sourcemaps are disabled RESOLVED Firefox 52
1304416 P1 Debugger James Long (:jlongster) [debugger.html] github tracked release requirements RESOLVED ---
1305079 P1 Debugger James Long (:jlongster) Add `noSliding` option to SourceActor.prototype.setBreakpoint RESOLVED Firefox 52
1306459 P1 Debugger James Long (:jlongster) Update new debugger frontend (9/29/2016) RESOLVED Firefox 52
1310330 -- Debugger Javascript source pane in debugger has double scrollbar. RESOLVED ---
1313197 P1 Debugger James Long (:jlongster) Open source in new tab if it doesn't exist in debugger RESOLVED Firefox 52
1319743 P2 Debugger Jason Laster [:jlast] Scrolling with the mouse for highlighting script in the new Debugger is jerky and consumes CPU VERIFIED ---
1332114 -- Debugger Jason Laster [:jlast] [debugger.html] Hold the new debugger in Aurora RESOLVED Firefox 53
1335901 P2 Debugger Yury Delendik (:yury) [debugger.html] cannot debug asm.js code RESOLVED ---
1346902 P3 Debugger Alexandre Poirot [:ochameau] [PTO back Jan 5th] Enable new debugger UI for Browser Toolbox RESOLVED Firefox 55
1365565 -- Debugger The position of the caret should be after the text RESOLVED ---
1365571 -- Debugger Slide the breakpoint to the next line of code when adding a breakpoint to a comment or to an empty line RESOLVED ---
1365603 P3 Debugger Search results should have a yellow background RESOLVED ---
1366745 P3 Debugger The order of the breakpoints added changes after reloading the page RESOLVED ---
1366754 P2 Debugger Hubert B Manilla After reloading the page closed JS files are still displayed VERIFIED Firefox 56
1366800 P1 Debugger Jason Laster [:jlast] Two search fields are displayed on the new debugger VERIFIED Firefox 56
1367777 -- Debugger Flicker in breakpoint section when adding a breakpoint to a comment or to an empty line RESOLVED ---
1376266 -- Debugger Out of scope functions should appear faded RESOLVED ---
1376727 -- Debugger A frame is displayed around the buttons in debugger toolbar VERIFIED ---
1379958 P1 Debugger Jason Laster [:jlast] The position of the cursor does not change in "watch expression" using the mouse VERIFIED Firefox 56
1379978 -- Debugger When you resume, the watch expression value should be cleared RESOLVED ---
1380007 -- Debugger Some areas in the debugger can't be accessed with keyboard only RESOLVED ---
1384061 -- Debugger After you step out from a breakpoint the number from the breakpoint is not visible RESOLVED ---
1384536 -- Debugger Firefox becomes unresponsive if I start a search RESOLVED ---
1384933 -- Debugger Can't iterate through search results after switching to a different file RESOLVED ---
1388798 P5 Debugger Source files are not accessible from the list view if the right hand pane is resized RESOLVED ---
1389013 -- Debugger Source files are displayed outside the dropdown list view RESOLVED ---
1389075 -- Debugger "Results" text from the search bar overlaps the search results RESOLVED ---
1389091 -- Debugger Search Bar results disappears when resizing Firefox's window VERIFIED ---
1389522 -- Debugger Error message is displayed outside the search results panel VERIFIED ---
1391240 -- Debugger Another, unwanted, breakpoint is created while holding ctrl + B VERIFIED ---
1391267 -- Debugger Conditional breakpoints can't replace disabled breakpoints VERIFIED ---
1392277 -- Debugger File location is not always revealed in the Tree menu VERIFIED ---
1395851 -- Debugger Watch Expressions are added and displayed only after restarting Firefox RESOLVED ---
1396523 -- Debugger Source file doesn't gain focus after being selected from the list view dropdown VERIFIED ---
1396841 -- Debugger Some breakpoints remain disabled after choosing the "Enable Breakpoints" option from the breakpoints pane VERIFIED ---
1397678 -- Debugger Horizontal scrollbar is not displayed at all times in the Scopes panel RESOLVED ---
1397697 -- Debugger Methods tooltip panel is dismissed while scrolling inside of it VERIFIED ---
1398064 -- Debugger Toolbar buttons are not visible while using High Contrast #1 and High Contrast Black themes VERIFIED ---

59 Total; 0 Open (0%); 45 Resolved (76.27%); 14 Verified (23.73%);


Verified bug fixes
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
1294103 -- Debugger James Long (:jlongster) Load source-map and pretty-print workers on the client-side in new debugger RESOLVED FIXED Firefox 51
1294686 P1 Debugger James Long (:jlongster) Mochitests for new debugger RESOLVED FIXED Firefox 51
1295753 -- Debugger Jason Laster [:jlast] New debugger can break console RESOLVED FIXED Firefox 51
1296048 -- Debugger James Long (:jlongster) Scope mozilla-specific CodeMirror CSS RESOLVED FIXED Firefox 51
1299602 P2 Debugger Debugger fails to emit tabNavigated event when paused RESOLVED FIXED Firefox 51
1300820 P1 Debugger Brian Grinstead [:bgrins] Always use 'old' debugger frontend in Browser Toolbox RESOLVED FIXED Firefox 51
1300861 P1 Debugger Brian Grinstead [:bgrins] Enable new debugger frontend in nightly builds RESOLVED FIXED Firefox 51
1300866 P1 Debugger James Long (:jlongster) Add a `devtoolsRequire` hook to expose native modules RESOLVED FIXED Firefox 51
1301147 -- Debugger James Long (:jlongster) Use BrowserLoader require in new debugger so we can share frontend modules RESOLVED FIXED Firefox 51
1301207 P1 Debugger James Long (:jlongster) Load theme switcher in new debugger RESOLVED FIXED Firefox 51
1301719 P1 Debugger Jason Laster [:jlast] Fix new debugger in remote toolboxes RESOLVED FIXED Firefox 51
1302833 P1 Debugger James Long (:jlongster) Fix CodeMirror height issue in new debugger RESOLVED FIXED Firefox 51
1302862 P2 Debugger James Long (:jlongster) Open the panel when the sources have loaded RESOLVED FIXED Firefox 52
1302872 P1 Debugger James Long (:jlongster) Update new debugger frontend (9/14/2016) RESOLVED FIXED Firefox 51
1303045 P1 Debugger James Long (:jlongster) Fix `viewSourceInDebugger` to highlight specific line RESOLVED FIXED Firefox 51
1303177 P1 Debugger James Long (:jlongster) Turn off sourcemaps if using the new debugger frontend RESOLVED FIXED Firefox 51
1303862 P1 Debugger James Long (:jlongster) Update new debugger frontend (9/19/2016) RESOLVED FIXED Firefox 52
1303866 P1 Debugger James Long (:jlongster) Don't track sourcemapped sources if sourcemaps are disabled RESOLVED FIXED Firefox 52
1304416 P1 Debugger James Long (:jlongster) [debugger.html] github tracked release requirements RESOLVED FIXED ---
1305079 P1 Debugger James Long (:jlongster) Add `noSliding` option to SourceActor.prototype.setBreakpoint RESOLVED FIXED Firefox 52
1306459 P1 Debugger James Long (:jlongster) Update new debugger frontend (9/29/2016) RESOLVED FIXED Firefox 52
1310330 -- Debugger Javascript source pane in debugger has double scrollbar. RESOLVED FIXED ---
1313197 P1 Debugger James Long (:jlongster) Open source in new tab if it doesn't exist in debugger RESOLVED FIXED Firefox 52
1319743 P2 Debugger Jason Laster [:jlast] Scrolling with the mouse for highlighting script in the new Debugger is jerky and consumes CPU VERIFIED FIXED ---
1332114 -- Debugger Jason Laster [:jlast] [debugger.html] Hold the new debugger in Aurora RESOLVED FIXED Firefox 53
1346902 P3 Debugger Alexandre Poirot [:ochameau] [PTO back Jan 5th] Enable new debugger UI for Browser Toolbox RESOLVED FIXED Firefox 55
1365603 P3 Debugger Search results should have a yellow background RESOLVED FIXED ---
1366745 P3 Debugger The order of the breakpoints added changes after reloading the page RESOLVED FIXED ---
1366754 P2 Debugger Hubert B Manilla After reloading the page closed JS files are still displayed VERIFIED FIXED Firefox 56
1366800 P1 Debugger Jason Laster [:jlast] Two search fields are displayed on the new debugger VERIFIED FIXED Firefox 56
1367777 -- Debugger Flicker in breakpoint section when adding a breakpoint to a comment or to an empty line RESOLVED FIXED ---
1379958 P1 Debugger Jason Laster [:jlast] The position of the cursor does not change in "watch expression" using the mouse VERIFIED FIXED Firefox 56
1379978 -- Debugger When you resume, the watch expression value should be cleared RESOLVED FIXED ---
1380007 -- Debugger Some areas in the debugger can't be accessed with keyboard only RESOLVED FIXED ---
1384536 -- Debugger Firefox becomes unresponsive if I start a search RESOLVED FIXED ---
1384933 -- Debugger Can't iterate through search results after switching to a different file RESOLVED FIXED ---
1389013 -- Debugger Source files are displayed outside the dropdown list view RESOLVED FIXED ---
1389075 -- Debugger "Results" text from the search bar overlaps the search results RESOLVED FIXED ---
1389091 -- Debugger Search Bar results disappears when resizing Firefox's window VERIFIED FIXED ---
1389522 -- Debugger Error message is displayed outside the search results panel VERIFIED FIXED ---
1391240 -- Debugger Another, unwanted, breakpoint is created while holding ctrl + B VERIFIED FIXED ---
1391267 -- Debugger Conditional breakpoints can't replace disabled breakpoints VERIFIED FIXED ---
1392277 -- Debugger File location is not always revealed in the Tree menu VERIFIED FIXED ---
1395851 -- Debugger Watch Expressions are added and displayed only after restarting Firefox RESOLVED FIXED ---
1396523 -- Debugger Source file doesn't gain focus after being selected from the list view dropdown VERIFIED FIXED ---
1396841 -- Debugger Some breakpoints remain disabled after choosing the "Enable Breakpoints" option from the breakpoints pane VERIFIED FIXED ---
1397678 -- Debugger Horizontal scrollbar is not displayed at all times in the Scopes panel RESOLVED FIXED ---
1397697 -- Debugger Methods tooltip panel is dismissed while scrolling inside of it VERIFIED FIXED ---
1398064 -- Debugger Toolbar buttons are not visible while using High Contrast #1 and High Contrast Black themes VERIFIED FIXED ---

49 Total; 0 Open (0%); 36 Resolved (73.47%); 13 Verified (26.53%);

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup N/A
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage N/A
Performance Testing N/A
All Defects Logged In Progress
Critical/Blockers Fixed and Verified In Progress
Metrics/Telemetry N/A
QA Beta - Full Testing [DONE]
QA Signoff - Pre Beta [DONE] [Email sent in 7/28/2017]
QA Signoff - Pre Release [DONE] [Email sent in 9/8/2017]
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/New_DevTools_Debugger_Panel&oldid=1181443"