Symbol Exporting: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
 
(47 intermediate revisions by 3 users not shown)
Line 1: Line 1:
''' QA/Symbols Exporting'''<br />
'''Revision History'''
'''Revision History'''


This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded).  The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
 


{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
Line 9: Line 12:
| 11/10/2016 || 1.0 || Abe Masresha || Created first draft
| 11/10/2016 || 1.0 || Abe Masresha || Created first draft
|-
|-
|             ||   ||    Tester         ||  
| 11/10/2016            || 1.1  ||    Grover Wimberly IV         || Fixed some wording in several sub-sections 
|-
| 1/13/2017            ||  1.2  ||    Abe Masresha        || Updated some links and added info on testing on android.
|-
|-
|}
|}


= Overview =
= Overview =
== Purpose ==
== Purpose ==
The goal of this work is to make Firefox less crashy for release users.
The goal of this work is to make Firefox crash less for release users.


== Scope ==
== Scope ==
Line 25: Line 29:
::: Sandboxing (content, GMP, plugin).
::: Sandboxing (content, GMP, plugin).
::: Exploratory testing for existing 3rd-party addons
::: Exploratory testing for existing 3rd-party addons
::: Android basic functionality testing


* How testing will be performed
* How testing will be performed
Line 32: Line 37:
  '''Mozilla Development'''<br />
  '''Mozilla Development'''<br />
  Benjamin Smedberg<br />
  Benjamin Smedberg<br />
Jim Mathies <br />
 


'''QA Eng. Softvision Las Vegas'''<br />
'''QA Eng. Softvision Las Vegas'''<br />
[mailto:mfunches@softvision.com Michelle Funches - Team Lead ]]<br />
[mailto:stefan.georgiev@softvision.com Michelle ->Stefan - Team Lead ]]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
Line 44: Line 49:
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
This feature will be tested manually on windows, OSX and Linux operating systems.
This feature will be tested manually on Windows, OSX, Linux operating systems and Android devices.


=== Out of Scope ===
=== Out of Scope ===
Android & iOS Devices<br />
iOS Devices (Not affected by this work)<br />
Automation will be handled by the developer<br />
Automation will be handled by the developer (if needed)<br />


= Requirements for testing =
= Requirements for testing =
Line 55: Line 60:
  Windows 10
  Windows 10
  Windows 7
  Windows 7
Windows XP (?)
  Ubuntu/Linux
  Ubuntu/Linux
  OSX 10.11
  OSX 10.12
Android


== Firefox Channel and environment setups ==
== Firefox Channel and environment setups ==


=== Build ===
=== Build ===
::: trybuild will be used to test this feature.
[https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa1e7161dbb0701360fc7703a6083db289703f38 Windows]<br />
::: Build Link: [link]
 
[https://treeherder.mozilla.org/#/jobs?repo=try&revision=e46703f15271becc535a10a35b3bcf9e77c1c3b3&selectedJob=31157378 Mac/Linux]<br />


= Test Strategy =  
= Test Strategy =  
Line 71: Line 77:
Verify feature works correctly with sandboxing (content, GMP, plugin)<br />
Verify feature works correctly with sandboxing (content, GMP, plugin)<br />
Verify feature works fine with existing 3rd-party addons<br />
Verify feature works fine with existing 3rd-party addons<br />
Verify basic functionalities are working properly on Android


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 82: Line 89:
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" |11/10/2016  ||  
|style="text-align:center;" |11/10/2016  || 11/10/2016
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 11/10/2016 ||  
|style="text-align:center;" | 11/10/2016 || 11/10/2016
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | 11/10-11/2016  ||  
|style="text-align:center;" | 11/10/2016  || 11/16/2016
|-
|-
| QA - tryBuild
| QA - tryBuild
|style="text-align:center;" | when ready ||  
|style="text-align:center;" | 11/15/2016 ||  
|-
| QA - Nightly
|style="text-align:center;" | 1/20/2017 || 1/20/2017
|-
|-
|}
|}
Line 103: Line 113:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/294&group_by=cases:section_id&group_order=asc TestRail]/ [https://docs.google.com/spreadsheets/d/1Ca2FquIuVdxw1yOidlQIMmY3MO3FoLVwyNnPuLAfMKs/edit#gid=1787174713 Google docs]
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/1380 TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla
|}
|}


= Status =  
= Status =  
== Overview ==
== Overview ==
Track the dates and build number where feature was released
<strong>Try Builds:</strong><br />
TBD
 
[https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa1e7161dbb0701360fc7703a6083db289703f38 Windows]<br />
 
[https://treeherder.mozilla.org/#/jobs?repo=try&revision=e46703f15271becc535a10a35b3bcf9e77c1c3b3&selectedJob=31157378 Mac/Linux]<br />
 
Testing on Android, retesting and bug verification: Nightly


== Risk analysis ==
== Risk analysis ==


= References =
= References =
Test cases: [link]<br />
Test cases: [https://docs.google.com/spreadsheets/d/19OHlTaibS59AbZUcD1RwNqzEv50WcBayuIfnJZwyX24/edit#gid=1098947246 gdoc] or [https://testrail.stage.mozaws.net/index.php?/suites/view/294&group_by=cases:section_id&group_order=desc TestRail]
         
Meta bug:  Bug-[https://bugzilla.mozilla.org/show_bug.cgi?id=1299187 1299187]<br />
Meta bug:  Bug-[https://bugzilla.mozilla.org/show_bug.cgi?id=1299187 1299187]<br />


Line 187: Line 204:
| Requires install testing     
| Requires install testing     
|style="text-align:center;" |  || To be tested
|style="text-align:center;" |  || To be tested
 
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || Not applicable
Line 272: Line 289:
||  || depends on the feature
||  || depends on the feature
|-
|-
|
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
|style="text-align:center;" |  || Not Applicable
|style="text-align:center;" |  || Not Applicable
|-
|-
Line 283: Line 300:


== Test suite ==
== Test suite ==
  Full Test suite - [Link]  
  Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/294&group_by=cases:section_id&group_order=asc TestRail]/ [https://docs.google.com/spreadsheets/d/1Ca2FquIuVdxw1yOidlQIMmY3MO3FoLVwyNnPuLAfMKs/edit#gid=1787174713 Google docs]<br />
Smoke Test suite - [Link]
Regression Test suite - [Link] - if available/needed.


= Bug Work =
= Bug Work =
Tracking bug – meta bug
Tracking bug – [https://bugzilla.mozilla.org/show_bug.cgi?id=1318149  meta bug] <br />
 
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">


====== Bug fix verification ======
====== Bugs ======
<div class="mw-collapsible-content">
{| class="wikitable" style="width:50%"
{| class="wikitable" style="width:50%"
|-
|-
! Bug No !! Summary !! Status !! Firefox Verion
! Bug No !! Summary !! Status !! Firefox Verion
|-
|-
|  [TBA] || bug summary || NEW || Nightly
|-
|-
| || || ||  
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1318149  Bug 1318149] ||Crash  ||Works for Me  || TryBuild/Nightly
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1318175 Bug 1318175]||Crash ||Works for Me ||TryBuild/Nightly <br />
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1317196 Bug 1317196]||Netflix not working on Linux||Known Issue||Nightly
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1318430 Bug 1318430]||Norton Security Toolbar does not install in Firefox||Known Issue||Nightly
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1317995 Bug 1317995]||Firefox 50 64-bit Stable channel is unable to play Farmville 2 Facebook Flash game ||Known Issue||Nightly
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1308677 Bug 1308677]||Flash warnings in Nightly are out of control ||Known Issue||Nightly
|}
|}
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ======
<div class="mw-collapsible-content">
[https://bugzilla.mozilla.org/show_bug.cgi?id=1284877 Bug 1284877]<br/>
<br/>
</div>
</div>


= Sign off =
= Sign off =
Line 324: Line 335:


*Links for the tests run
*Links for the tests run
** Daily Smoke, [link]
** Full Test suite: [https://testrail.stage.mozaws.net/index.php?/plans/view/2119 Link]  
** Full Test suite: [Link]  
** Regression Test suite: [link] if needed/available
** Regression Test suite: [link] if needed/available


Line 364: Line 374:
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release||  
|style="text-align:center;" | Email to be sent
|style="text-align:center;" |  
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
Line 370: Line 380:
|-
|-
|  QA Signoff  - Aurora Release ||  
|  QA Signoff  - Aurora Release ||  
|style="text-align:center;"| Email to be sent  
|style="text-align:center;"| Email sent  
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
Line 376: Line 386:
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release ||  
|style="text-align:center;" | Email to be sent
|style="text-align:center;" |
|}
|}

Latest revision as of 18:41, 25 January 2017

QA/Symbols Exporting


Revision History


Date Version Author Description
11/10/2016 1.0 Abe Masresha Created first draft
11/10/2016 1.1 Grover Wimberly IV Fixed some wording in several sub-sections
1/13/2017 1.2 Abe Masresha Updated some links and added info on testing on android.

Overview

Purpose

The goal of this work is to make Firefox crash less for release users.

Scope

This wiki details the testing that will be performed by the project team for the Symbol Exporting project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • Testing will focus on:
Firefox launch and shutdown
Firefox child processes.
Sandboxing (content, GMP, plugin).
Exploratory testing for existing 3rd-party addons
Android basic functionality testing
  • How testing will be performed
Manual

Ownership

Mozilla Development

Benjamin Smedberg


QA Eng. Softvision Las Vegas
Michelle ->Stefan - Team Lead ]
Abe Masresha - Test Lead
Grover Wimberly IV - Tester
Kanchan Kumari - Tester
Justin Williams - Tester

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, OSX, Linux operating systems and Android devices.

Out of Scope

iOS Devices (Not affected by this work)
Automation will be handled by the developer (if needed)

Requirements for testing

Environments: Operating Systems

Windows 8.1
Windows 10
Windows 7
Ubuntu/Linux
OSX 10.12
Android

Firefox Channel and environment setups

Build

Windows

Mac/Linux

Test Strategy

Test Objectives

Verify feature works correctly with Firefox launch and shutdowns
Verify feature works correctly with Firefox child processes
Verify feature works correctly with sandboxing (content, GMP, plugin)
Verify feature works fine with existing 3rd-party addons
Verify basic functionalities are working properly on Android

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project
Study documentation/specs received from developers 11/10/2016 11/10/2016
QA - Test plan creation 11/10/2016 11/10/2016
QA - Test cases/Env preparation 11/10/2016 11/16/2016
QA - tryBuild 11/15/2016
QA - Nightly 1/20/2017 1/20/2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Try Builds:

Windows

Mac/Linux

Testing on Android, retesting and bug verification: Nightly

Risk analysis

References

Test cases: gdoc or TestRail

Meta bug: Bug-1299187

Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Certain tests are performed in a private window.
Multi-Process Enabled Enabled by default
Multi-process Disabled To be tested
Theme (high contrast) Not applicable
UI
Mouse-only operation To be tested
Keyboard-only operation To be tested
Display (HiDPI) Not applicable
Interraction (scroll, zoom) To be tested
Usable with a screen reader Not applicable
Usability and/or discoverability testing Is this feature user friendly
RTL build testing To be tested
Help/Support
Help/support interface required Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Make sure support documents are written and are correct.
Install/Upgrade
Feature installs as expected To be tested
Does sync work across Install To be tested
Requires install testing To be tested
Affects first-run or onboarding Not applicable
Does this affect partner builds? Partner build testing Not applicable
Enterprise
Enterprise administration Not applicable
Network proxies/autoconfig Not applicable
ESR behavior changes Not applicable
Locked preferences Not applicable
Data Monitoring
Temporary or permanent telemetry monitoring List of error conditions to monitor
Telemetry correctness testing Not applicable
Server integration testing Not applicable
Offline and server failure testing Not applicable
Load testing Not applicable
Add-ons
Addon API required? Not applicable
Comprehensive API testing Not applicable
Permissions Not applicable
Testing with existing/popular addons To be tested
Security
3rd-party security review Not applicable
Privilege escalation testing Not applicable
Fuzzing Not applicable
Web Compatibility depends on the feature
Testing against target sites To be Tested
Survey of many sites for compatibility Not applicable
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. Not Applicable
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS Desktop only
Interaction of this feature with other browser features Not Applicable

Test suite

Full Test suite - TestRail/ Google docs

Bug Work

Tracking bug – meta bug

Bugs
Bug No Summary Status Firefox Verion
Bug 1318149 Crash Works for Me TryBuild/Nightly
Bug 1318175 Crash Works for Me TryBuild/Nightly
Bug 1317196 Netflix not working on Linux Known Issue Nightly
Bug 1318430 Norton Security Toolbar does not install in Firefox Known Issue Nightly
Bug 1317995 Firefox 50 64-bit Stable channel is unable to play Farmville 2 Facebook Flash game Known Issue Nightly
Bug 1308677 Flash warnings in Nightly are out of control Known Issue Nightly

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Feature testing

  • Links for the tests run
    • Full Test suite: Link
    • Regression Test suite: [link] if needed/available

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release
QA Aurora - Full Testing
QA Signoff - Aurora Release Email sent
QA Beta - Full Testing
QA Signoff - Beta Release
Retrieved from "https://wiki.mozilla.org/index.php?title=Symbol_Exporting&oldid=1160971"