Symbol Exporting: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
 
(69 intermediate revisions by 3 users not shown)
Line 1: Line 1:
''' QA/Symbols Exporting'''<br />
'''Revision History'''
'''Revision History'''


This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded).  The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
 


{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
Line 7: Line 10:
! Date !! Version !! Author !! Description  
! Date !! Version !! Author !! Description  
|-
|-
| 10/27/2016 || 1.0 || Abe Masresha || Created first draft
| 11/10/2016 || 1.0 || Abe Masresha || Created first draft
|-
| 11/10/2016            ||  1.1  ||    Grover Wimberly IV        || Fixed some wording in several sub-sections 
|-
|-
|             ||   ||    Tester         ||  
| 1/13/2017            || 1.2  ||    Abe Masresha         || Updated some links and added info on testing on android.
|-
|-
|}
|}


= Overview =
= Overview =
== Purpose ==
== Purpose ==
The goal of this work is to make Firefox less crashy for release users by making it impossible for 3rd-party code to hook into XPCOM.
The goal of this work is to make Firefox crash less for release users.


== Scope ==
== Scope ==
Line 25: Line 29:
::: Sandboxing (content, GMP, plugin).
::: Sandboxing (content, GMP, plugin).
::: Exploratory testing for existing 3rd-party addons
::: Exploratory testing for existing 3rd-party addons
::: Android basic functionality testing


* How testing will be performed
* How testing will be performed
Line 32: Line 37:
  '''Mozilla Development'''<br />
  '''Mozilla Development'''<br />
  Benjamin Smedberg<br />
  Benjamin Smedberg<br />
Jim Mathies <br />
 


'''QA Eng. Softvision Las Vegas'''<br />
'''QA Eng. Softvision Las Vegas'''<br />
[mailto:mfunches@softvision.com Michelle Funches - Team Lead ]]<br />
[mailto:stefan.georgiev@softvision.com Michelle ->Stefan - Team Lead ]]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
Line 44: Line 49:
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
Functionality and permission for Screen Sharing
This feature will be tested manually on Windows, OSX, Linux operating systems and Android devices.
Windows 7
Windows 8.1
Windows 10
Ubuntu/Linux


=== Out of Scope ===
=== Out of Scope ===
Windows XP<br />
iOS Devices (Not affected by this work)<br />
Android & iOS Devices
Automation will be handled by the developer (if needed)<br />
Automation will be handled by the developer


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments: Operating Systems ==
Windows 7<br />
Windows 8.1
Windows 8.1<br />
Windows 10
Windows 10<br />
Windows 7
Ubuntu/Linux<br />
Ubuntu/Linux
Macintosh 10.12<br />
OSX 10.12
Webcam & Microphone<br />
Android
Screen Sharing Software


== Firefox Channel and environment setups ==


== Channel dependent settings (configs) and environment setups ==
=== Build ===
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
[https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa1e7161dbb0701360fc7703a6083db289703f38 Windows]<br />


=== Nightly ===
[https://treeherder.mozilla.org/#/jobs?repo=try&revision=e46703f15271becc535a10a35b3bcf9e77c1c3b3&selectedJob=31157378 Mac/Linux]<br />
<div class="mw-collapsible-content">
media.getusermedia.screensharing.allow_on_old_platforms <br>
media.getusermedia.screensharing.allowed_domains <br>
media.getusermedia.screensharing.enabled <br>
</div>
 
=== Aurora ===
<div class="mw-collapsible-content">
Not Applicable
</div>
 
=== Beta ===
<div class="mw-collapsible-content">
Not Applicable
</div>
 
=== Post Beta / Release ===
<div class="mw-collapsible-content">
Not Applicable
</div>
</div>


= Test Strategy =  
= Test Strategy =  
== Test Objectives ==
== Test Objectives ==
Verify Camera and Microphone<br />
Verify feature works correctly with Firefox launch and shutdowns<br />
Verify Screen Sharing (Single/Dual Monitors) [Cisco Spark]<br />
Verify feature works correctly with Firefox child processes<br />
Verify Application Sharing<br />
Verify feature works correctly with sandboxing (content, GMP, plugin)<br />
Verify Window Sharing<br />
Verify feature works fine with existing 3rd-party addons<br />
 
Verify basic functionalities are working properly on Android
== Builds ==
 
* Links for Nightly builds:
#Latest Mozilla Central Nightly Builds [[http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/]]
 


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 115: Line 89:
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" |10/24/2016  ||  
|style="text-align:center;" |11/10/2016  || 11/10/2016
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 10/27/2016 ||  
|style="text-align:center;" | 11/10/2016 || 11/10/2016
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | 10/25/2016  ||  
|style="text-align:center;" | 11/10/2016  || 11/16/2016
|-
|-
| QA - Nightly Testing
| QA - tryBuild
|style="text-align:center;" | ||  
|style="text-align:center;" | 11/15/2016 ||  
|-
|-
| QA - Aurora Testing
| QA - Nightly
|style="text-align:center;" | ||
|style="text-align:center;" | 1/20/2017 || 1/20/2017
|-
|-
| QA - Beta Testing
|style="text-align:center;" |  ||
|-
| Release Date
|style="text-align:center;" | ||
|}
|}


Line 144: Line 113:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/294&group_by=cases:section_id&group_order=asc TestRail]/ [https://docs.google.com/spreadsheets/d/1Ca2FquIuVdxw1yOidlQIMmY3MO3FoLVwyNnPuLAfMKs/edit#gid=1787174713 Google docs]
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/1380 TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla
|}
|}


= Status =  
= Status =  
== Overview ==
== Overview ==
Track the dates and build number where feature was released to Nightly
<strong>Try Builds:</strong><br />
TBD
 
[https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa1e7161dbb0701360fc7703a6083db289703f38 Windows]<br />
 
[https://treeherder.mozilla.org/#/jobs?repo=try&revision=e46703f15271becc535a10a35b3bcf9e77c1c3b3&selectedJob=31157378 Mac/Linux]<br />
 
Testing on Android, retesting and bug verification: Nightly


== Risk analysis ==
== Risk analysis ==
Accidental exposure -> sharing information on the desktop not intended to share
 
= References =
= References =
#Screen Sharing [[https://docs.google.com/document/d/1m1arSi9mr2qrOuzP9B-Oz1qG51Sy1xfPjBpHekpGJNc/edit?pref=2&pli=1]]
Test cases: [https://docs.google.com/spreadsheets/d/19OHlTaibS59AbZUcD1RwNqzEv50WcBayuIfnJZwyX24/edit#gid=1098947246 gdoc] or [https://testrail.stage.mozaws.net/index.php?/suites/view/294&group_by=cases:section_id&group_order=desc TestRail]
#Show Preview Bug 1284877#c0 [[https://bugzilla.mozilla.org/show_bug.cgi?id=1284877#c0]]
         
#Mock up [[https://mozilla.invisionapp.com/share/AF71R266U#/screens/152725629]]
Meta bug:  Bug-[https://bugzilla.mozilla.org/show_bug.cgi?id=1299187 1299187]<br />


* Meta bug
= Testcases =  
= Testcases =  
== Overview ==
== Overview ==
Line 182: Line 154:
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" |  || Only tested if there are bugs found in testing
|style="text-align:center;" |  || To be tested
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
Line 197: Line 169:
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | || To be tested
|style="text-align:center;" | || Not applicable
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom)  
Line 203: Line 175:
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" |  || e.g. with NVDA
|style="text-align:center;" |  || Not applicable
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
Line 224: Line 196:
||  ||  
||  ||  
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature installs as expected   
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || To be tested
|-
|-
| Does sync work across upgrades    
| Does sync work across Install    
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || To be tested
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" |  || Default installation with Nightly
|style="text-align:center;" |  || To be tested
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" |  || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" |  || Not applicable
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" |  || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" |  || Not applicable


|-
|-
| ''' Enterprise '''  
| ''' Enterprise '''  
||  || Raise up the topic to developers to see if they are expecting to work different on ESR builds
||  ||  
|-
|-
| Enterprise administration   
| Enterprise administration   
Line 276: Line 248:
|-
|-
| ''' Add-ons '''  
| ''' Add-ons '''  
||  || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
||  ||
|-
|-
| Addon API required?   
| Addon API required?   
Line 285: Line 257:
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" |  || Permission manager for sharing microphone, camera, and screen/monitor.
|style="text-align:center;" |  || Not applicable
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || To be tested


|-
|-
| ''' Security '''  
| ''' Security '''  
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
||  ||
|-
|-
| 3rd-party security review   
| 3rd-party security review   
Line 308: Line 280:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" |  || CiscoSpark, Skype, GoToMeeting, free web communication services
|style="text-align:center;" |  || To be Tested
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
Line 317: Line 289:
||  || depends on the feature
||  || depends on the feature
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.  
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.  
|style="text-align:center;" |  || Some tests require 2+ people to be able to do
|style="text-align:center;" |  || Not Applicable
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
Line 324: Line 296:
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" |  || Possible testing with other browsers
|style="text-align:center;" |  || Not Applicable
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
  Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/294&group_by=cases:section_id&group_order=asc TestRail]/ [https://docs.google.com/spreadsheets/d/1Ca2FquIuVdxw1yOidlQIMmY3MO3FoLVwyNnPuLAfMKs/edit#gid=1787174713 Google docs]<br />
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
Regression Test suite - Link with the gdoc - if available/needed.


= Bug Work =
= Bug Work =
Tracking bug – meta bug
Tracking bug – [https://bugzilla.mozilla.org/show_bug.cgi?id=1318149  meta bug] <br />


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bugs ======
 
====== Bug fix verification ======
<div class="mw-collapsible-content">
{| class="wikitable" style="width:50%"
{| class="wikitable" style="width:50%"
|-
|-
! Bug No !! Summary !! Status !! Firefox Verion
! Bug No !! Summary !! Status !! Firefox Verion
|-
|-
|  [TBA] || bug summary || NEW || Nightly
|-
|-
| || || ||  
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1318149  Bug 1318149] ||Crash  ||Works for Me  || TryBuild/Nightly
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1318175 Bug 1318175]||Crash ||Works for Me ||TryBuild/Nightly <br />
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1317196 Bug 1317196]||Netflix not working on Linux||Known Issue||Nightly
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1318430 Bug 1318430]||Norton Security Toolbar does not install in Firefox||Known Issue||Nightly
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1317995 Bug 1317995]||Firefox 50 64-bit Stable channel is unable to play Farmville 2 Facebook Flash game ||Known Issue||Nightly
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1308677 Bug 1308677]||Flash warnings in Nightly are out of control ||Known Issue||Nightly
|}
|}
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ======
<div class="mw-collapsible-content">
[https://bugzilla.mozilla.org/show_bug.cgi?id=1284877 Bug 1284877]<br/>
<br/>
</div>
</div>


= Sign off =
= Sign off =
Line 366: Line 332:


== Results ==
== Results ==
'''Nightly testing'''<br />
'''Feature testing'''<br />


List of OSes that will be covered by testing<br />
*Links for the tests run
*Link for the tests run
** Full Test suite: [https://testrail.stage.mozaws.net/index.php?/plans/view/2119 Link]  
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Regression Test suite: [link] if needed/available
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Regression Test suite, if needed/available
 
'''Merge to Aurora Sign-off'''
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite


== Checklist ==
== Checklist ==
Line 415: Line 374:
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release||  
|style="text-align:center;" | Email to be sent
|style="text-align:center;" |  
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
Line 421: Line 380:
|-
|-
|  QA Signoff  - Aurora Release ||  
|  QA Signoff  - Aurora Release ||  
|style="text-align:center;"| Email to be sent  
|style="text-align:center;"| Email sent  
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
Line 427: Line 386:
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release ||  
|style="text-align:center;" | Email to be sent
|style="text-align:center;" |
|}
|}

Latest revision as of 18:41, 25 January 2017

QA/Symbols Exporting


Revision History


Date Version Author Description
11/10/2016 1.0 Abe Masresha Created first draft
11/10/2016 1.1 Grover Wimberly IV Fixed some wording in several sub-sections
1/13/2017 1.2 Abe Masresha Updated some links and added info on testing on android.

Overview

Purpose

The goal of this work is to make Firefox crash less for release users.

Scope

This wiki details the testing that will be performed by the project team for the Symbol Exporting project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • Testing will focus on:
Firefox launch and shutdown
Firefox child processes.
Sandboxing (content, GMP, plugin).
Exploratory testing for existing 3rd-party addons
Android basic functionality testing
  • How testing will be performed
Manual

Ownership

Mozilla Development

Benjamin Smedberg


QA Eng. Softvision Las Vegas
Michelle ->Stefan - Team Lead ]
Abe Masresha - Test Lead
Grover Wimberly IV - Tester
Kanchan Kumari - Tester
Justin Williams - Tester

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, OSX, Linux operating systems and Android devices.

Out of Scope

iOS Devices (Not affected by this work)
Automation will be handled by the developer (if needed)

Requirements for testing

Environments: Operating Systems

Windows 8.1
Windows 10
Windows 7
Ubuntu/Linux
OSX 10.12
Android

Firefox Channel and environment setups

Build

Windows

Mac/Linux

Test Strategy

Test Objectives

Verify feature works correctly with Firefox launch and shutdowns
Verify feature works correctly with Firefox child processes
Verify feature works correctly with sandboxing (content, GMP, plugin)
Verify feature works fine with existing 3rd-party addons
Verify basic functionalities are working properly on Android

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project
Study documentation/specs received from developers 11/10/2016 11/10/2016
QA - Test plan creation 11/10/2016 11/10/2016
QA - Test cases/Env preparation 11/10/2016 11/16/2016
QA - tryBuild 11/15/2016
QA - Nightly 1/20/2017 1/20/2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Try Builds:

Windows

Mac/Linux

Testing on Android, retesting and bug verification: Nightly

Risk analysis

References

Test cases: gdoc or TestRail

Meta bug: Bug-1299187

Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Certain tests are performed in a private window.
Multi-Process Enabled Enabled by default
Multi-process Disabled To be tested
Theme (high contrast) Not applicable
UI
Mouse-only operation To be tested
Keyboard-only operation To be tested
Display (HiDPI) Not applicable
Interraction (scroll, zoom) To be tested
Usable with a screen reader Not applicable
Usability and/or discoverability testing Is this feature user friendly
RTL build testing To be tested
Help/Support
Help/support interface required Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Make sure support documents are written and are correct.
Install/Upgrade
Feature installs as expected To be tested
Does sync work across Install To be tested
Requires install testing To be tested
Affects first-run or onboarding Not applicable
Does this affect partner builds? Partner build testing Not applicable
Enterprise
Enterprise administration Not applicable
Network proxies/autoconfig Not applicable
ESR behavior changes Not applicable
Locked preferences Not applicable
Data Monitoring
Temporary or permanent telemetry monitoring List of error conditions to monitor
Telemetry correctness testing Not applicable
Server integration testing Not applicable
Offline and server failure testing Not applicable
Load testing Not applicable
Add-ons
Addon API required? Not applicable
Comprehensive API testing Not applicable
Permissions Not applicable
Testing with existing/popular addons To be tested
Security
3rd-party security review Not applicable
Privilege escalation testing Not applicable
Fuzzing Not applicable
Web Compatibility depends on the feature
Testing against target sites To be Tested
Survey of many sites for compatibility Not applicable
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. Not Applicable
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS Desktop only
Interaction of this feature with other browser features Not Applicable

Test suite

Full Test suite - TestRail/ Google docs

Bug Work

Tracking bug – meta bug

Bugs
Bug No Summary Status Firefox Verion
Bug 1318149 Crash Works for Me TryBuild/Nightly
Bug 1318175 Crash Works for Me TryBuild/Nightly
Bug 1317196 Netflix not working on Linux Known Issue Nightly
Bug 1318430 Norton Security Toolbar does not install in Firefox Known Issue Nightly
Bug 1317995 Firefox 50 64-bit Stable channel is unable to play Farmville 2 Facebook Flash game Known Issue Nightly
Bug 1308677 Flash warnings in Nightly are out of control Known Issue Nightly

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Feature testing

  • Links for the tests run
    • Full Test suite: Link
    • Regression Test suite: [link] if needed/available

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release
QA Aurora - Full Testing
QA Signoff - Aurora Release Email sent
QA Beta - Full Testing
QA Signoff - Beta Release
Retrieved from "https://wiki.mozilla.org/index.php?title=Symbol_Exporting&oldid=1160971"