MozillaWiki

QA/Netmonitor Phase II (Launchpad): Difference between revisions

Page Discussion

QA/Netmonitor Phase II (Launchpad) (view source)

Revision as of 07:55, 14 September 2017

259 bytes added ,  14 September 2017
→‎Checklist: vbnV
 
(12 intermediate revisions by the same user not shown)
Line 23: Line 23:
|-
|-
| 14/06/2017 || 1.0 || Ciprian Georgiu || Created first draft
| 14/06/2017 || 1.0 || Ciprian Georgiu || Created first draft
|-
|-
| 28/08/2017 || 1.1 || Ciprian Georgiu || Updates
|-
|-
|}
|}
Line 255: Line 258:


== Overview ==
== Overview ==
** Feature was released to Nightly: [https://archive.mozilla.org/pub/firefox/nightly/2017/04/2017-04-10-03-02-21-mozilla-central/ 55.0a1 (2017-04-10)]
* Feature was released to Nightly: [https://archive.mozilla.org/pub/firefox/nightly/2017/04/2017-04-10-03-02-21-mozilla-central/ 55.0a1 (2017-04-10)]
** Feature was merged to Beta: 56.0b1 [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ (2017-08-07)]
* Feature was merged to Beta: [https://archive.mozilla.org/pub/firefox/candidates/56.0b1-candidates/build1/ 56.0b1  (2017-08-07)]
** Feature was merged to Release: TBD
* Feature was merged to Release: TBD


= References =
= References =
* List and links for specs
* List and links for specs
  List and links for available specs - http://searchfox.org/mozilla-central/source/devtools/client/netmonitor/README.md
** [http://searchfox.org/mozilla-central/source/devtools/client/netmonitor/README.md run Netmonitor in Launchpad]
** [https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor MDN article]
 
* Meta bug
* Meta bug
(netmonitor-phaseII) [meta] Load Net Panel In Launchpad) [https://bugzilla.mozilla.org/show_bug.cgi?id=1348737]
** {{bug|1348737}} - (netmonitor-phaseII) [meta] Load Net Panel In Launchpad


= Testcases =  
= Testcases =  
Line 272: Line 277:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" |  ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 287: Line 292:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" |  ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction (scroll, zoom)  
| Interaction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | No || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | Yes || Is this feature user friendly
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 311: Line 316:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" |  || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | Yes || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" |  || Make sure support documents are written and are correct.
|style="text-align:center;" | Yes || Make sure support documents are written and are correct.


|-
|-
Line 321: Line 326:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | No || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | No || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | No || yes/no options, add comment with details about who will lead testing


|-
|-
Line 340: Line 345:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" |  ||
|style="text-align:center;" | No ||


|-
|-
Line 356: Line 361:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | No || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" |  ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" |  ||
|style="text-align:center;" | No ||


|-
|-
Line 375: Line 380:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" |  ||  
|style="text-align:center;" |  No ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  


|-
|-
Line 391: Line 396:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  


|-
|-
Line 404: Line 409:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  


|-
|-
Line 414: Line 419:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" |  ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  
|}
|}


== Test suite ==
== Test suite ==
Full Test suite - Link to test rail - tascases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/view/1108&group_by=cases:section_id&group_order=asc link]
* Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/1108&group_by=cases:section_id&group_order=asc TestRail link]
Smoke Test suite - Link with the tests - if available/needed.
* Smoke Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/1108&group_by=cases:section_id&group_order=asc TestRail link]
Regression Test suite - Link with the tests - if available/needed.
* Regression Test suite - n/a.


= Bug Work =
= Bug Work =
Line 468: Line 473:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


List of OSes that will be covered by testing<br />
* List of OSes that will be covered by testing:<br />
*Link for the tests run
** Windows 10 x64
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]
** Windows 7 x86
** Daily Smoke, if needed/available
** Ubuntu 16.04 x64
** Regression Test suite, if needed/available
** Mac OS X 10.11.6
<br />
* Link for the tests run - [https://testrail.stage.mozaws.net/index.php?/reports/view/411 link]


'''Merge to Beta Sign-off'''<br />
'''Merge to Beta Sign-Off'''
List of OSes that will be covered by testing<br />
* List of OSes that will be covered by testing<br />
*Link for the tests run
** Windows 10 x64
** Full Test suite
** Windows 7 x86
** Mac OS X 10.11.6
** Ubuntu 16.04 x64
* Link for the tests run - [https://testrail.stage.mozaws.net/index.php?/reports/view/473 link]


= Checklist =
== Checklist ==
{| class="wikitable" style="width:60%"
{| class="wikitable"
|-
|-
! Exit Criteria !! Status !! Notes/Details
! Exit Criteria !! Status !! Notes/Details
|-
|-
| Testing Prerequisites (specs, use cases)  
| Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |  Done
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
| Testing Infrastructure setup  
| Testing Infrastructure setup  
|style="text-align:center;" |  Done ||  
|style="text-align:center;" |  {{mdone|}} ||  
|-
|-
| Test Plan Creation  
| Test Plan Creation  
| style="text-align:center;" | Done ||  
| style="text-align:center;" |  {{mdone|}} ||  
|-
|-
| Test Cases Creation  
| Test Cases Creation  
|style="text-align:center;" |  Done ||  
|style="text-align:center;" |  {{mdone|}} ||  
|-
|-
|  Automation Coverage ||  
| Full Functional Tests Execution
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
| Performance Testing
| Automation Coverage
|style="text-align:center;" | ||
|style="text-align:center;" | N/A ||  
|-
 
|  All Defects Logged || ||  
|-
|  Critical/Blockers Fixed and Verified || ||
|-
|-
| Metrics/Telemetry||
| Performance Testing
|style="text-align:center;" |  
|style="text-align:center;" | <font style="font-weight:bold;font-size:80%;">[NOT APPLICABLE]</font> ||  
|-
|-
| Basic/Core functionality Nightly testing
| All Defects Logged
|style="text-align:center;" |
|style="text-align:center;" | {{mdone|}} ||  
|style="text-align:center;" |
|-
|-
| QA mid-Nightly Signoff||
| Critical/Blockers Fixed and Verified
|style="text-align:center;" | Done (YELLOW) [https://testrail.stage.mozaws.net/index.php?/reports/view/411 see testrail report]
|style="text-align:center;" | {{mdone|}}  ||
|-
|-
| QA Nightly - Full Testing
| Metrics/Telemetry
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||
|-
|-
|  QA pre-Beta Signoff||
| QA Signoff - Nightly Release
|style="text-align:center;"| Done (YELLOW) [https://testrail.stage.mozaws.net/index.php?/reports/view/473 see testrail report]
|style="text-align:center;" | {{mdone|}}
|| [https://testrail.stage.mozaws.net/index.php?/reports/view/473 Email sent - YELLOW]
|-
|-
| QA Beta - Full Testing
| QA Beta - Full Testing
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  QA pre-Release Signoff ||
| QA Signoff - Beta Release
|style="text-align:center;" | Email to be sent  
| style="text-align:center;" | {{mdone|}} || [https://testrail.stage.mozaws.net/index.php?/reports/view/544 Email sent - GREEN]
|}
|}
Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1179431...1180425"