MozillaWiki

QA/Multi-Addon Pref: Difference between revisions

Page Discussion

QA/Multi-Addon Pref (view source)

Revision as of 15:41, 18 July 2017

768 bytes added ,  18 July 2017
no edit summary
No edit summary
 
(10 intermediate revisions by the same user not shown)
Line 9: Line 9:
| Ryan VanderMeulen || QA Manager || Product Integrity || Date || Email
| Ryan VanderMeulen || QA Manager || Product Integrity || Date || Email
|-
|-
| Felipe Gomez || Software Engineer || Engineering || Date || Email
| Blake Kaplan || Software Engineer || Engineering || Date || Email
|-
|-
| TBD || EPM || Product Management || Date || Email
| TBD || EPM || Product Management || Date || Email
Line 41: Line 41:


== Ownership ==
== Ownership ==
What resources are needed, and when (developers, QA, PMs working on the feature)
Product contact:<br />
 
 
 
Engineering contact:<br />
[mailto:bkaplan@mozilla.com Blake Kaplan] (IRC: mrbkap) <br />
 
 
QA contact:<br />
[mailto:rvandermeulen@mozilla.com Ryan VanderMeulen] (IRC: RyanVM)<br />
 
QA:<br />
PM for QA team - [mailto:rares.bologa@softvisioninc.eu Rares Bologa] (IRC: RaresB)<br />
QA Engineering Contact - [mailto:gwimberly@softvision.com Grover Wimberly IV] (IRC: Grover-QA)<br />
QA - [mailto:kkumari@softvision.com Kanchan Kumari] (IRC: Kanchan_QA)<br />
QA - [mailto:jwilliams@softvision.com Justin Williams] (IRC: JW_SoftvisionQA)<br />
QA - [mailto:stefan.georgiev@softvision.com Stefan Georgiev] (IRC: StefanG_QA)<br />
QA - [mailto:amasresha@softvision.com Abe Masresha] (IRC: Abe_LV)<br />
 
= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
Line 53: Line 70:
= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Specify OSes that need to be covered
* Windows 7/8/8.1/10
Specify Devices that need to be covered
* Linux (Ubuntu 16.04)
Specify other configuration/environmental setup needed
* Mac OS X
 


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
Line 112: Line 128:
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
|-
|-
| 1 || Name of the feature or sub-function being tested || The objective the test is trying to demonstrate || The criteria that will be evaluated to demonstrate the test is successful || Manual/ Automation/ Regression/ Performance/ Usability/ Security/ Telemetry || RAC-1, RAC-2, RAC-3 || Eng Team
| 1 || Checking preference value of dom.ipc.processCount.web || Verify the pref || Value is set to integer instead of a string || Manual Testing (Bug Verification) || RAC-1, RAC-2, RAC-3 || Eng Team
|-
|-
| 2 || Repeat for each feature/sub-function || || || || ||
| 2 || Ensure updates of existing users of older versions have the pref flipped || Verify the pref || Value is set to integer instead of a string for existing Firefox users|| Manual Testing (Bug Verification)  || RAC-1, RAC-2, RAC-3 || Eng Team
|-
|-
| 3 || || || || || ||
| 3 || Monitoring the process tabs of the operating system || Verify multiple content processes are running per the pref || Multiple content processes are running simultaneously || Manual Testing (Bug Verification)  || RAC-1, RAC-2, RAC-3 || Eng Team
|}
|}


Line 185: Line 201:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 200: Line 216:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction (scroll, zoom)  
| Interaction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | No || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | No || Is this feature user friendly
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 224: Line 240:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | No || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | No || Make sure support documents are written and are correct.


|-
|-
Line 234: Line 250:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | ||  
|style="text-align:center;" | ??? ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | Yes || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | Yes || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | No || yes/no options, add comment with details about who will lead testing


|-
|-
Line 253: Line 269:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | ||  
|style="text-align:center;" | ??? ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 269: Line 285:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | ??? || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | ??? ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 288: Line 304:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | ??? ||  


|-
|-
Line 304: Line 320:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 317: Line 333:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 327: Line 343:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes (e10s multi-process) ||  
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]
  Full Test suite - TBA
Smoke Test suite - Link with the tests - if available/needed.
Regression Test suite - Link with the tests - if available/needed.


= Bug Work =
= Bug Work =
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=1380725 1380725 - release and beta channels are only running 1 content process even when in the 4 process cohort]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1380725 1380725] )======


<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[12345],
         "blocks":[1380725],
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
     }
     }
Line 363: Line 377:
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[12345],
         "blocks":[1380725],
         "resolution":"FIXED",
         "resolution":"FIXED",
         "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
Line 375: Line 389:
Checklist
Checklist
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* Preferences are set in the browser properly.
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/Release Management/QA)


== Results ==
== Results ==
Gwimberly
172

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1175892...1175970"