Symbol Exporting: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
 
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
''' QA/Symbol Exporting'''<br />
''' QA/Symbols Exporting'''<br />




Line 29: Line 29:
::: Sandboxing (content, GMP, plugin).
::: Sandboxing (content, GMP, plugin).
::: Exploratory testing for existing 3rd-party addons
::: Exploratory testing for existing 3rd-party addons
::: Android basic functionality testing


* How testing will be performed
* How testing will be performed
Line 39: Line 40:


'''QA Eng. Softvision Las Vegas'''<br />
'''QA Eng. Softvision Las Vegas'''<br />
[mailto:mfunches@softvision.com Michelle Funches - Team Lead ]]<br />
[mailto:stefan.georgiev@softvision.com Michelle ->Stefan - Team Lead ]]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
Line 48: Line 49:
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
This feature will be tested manually on Windows, OSX and Linux operating systems.
This feature will be tested manually on Windows, OSX, Linux operating systems and Android devices.


=== Out of Scope ===
=== Out of Scope ===
Android & iOS Devices<br />
iOS Devices (Not affected by this work)<br />
Automation will be handled by the developer<br />
Automation will be handled by the developer (if needed)<br />


= Requirements for testing =
= Requirements for testing =
Line 98: Line 99:
| QA - tryBuild
| QA - tryBuild
|style="text-align:center;" | 11/15/2016 ||  
|style="text-align:center;" | 11/15/2016 ||  
|-
| QA - Nightly
|style="text-align:center;" | 1/20/2017 || 1/20/2017
|-
|-
|}
|}
Line 124: Line 128:


[https://treeherder.mozilla.org/#/jobs?repo=try&revision=e46703f15271becc535a10a35b3bcf9e77c1c3b3&selectedJob=31157378 Mac/Linux]<br />
[https://treeherder.mozilla.org/#/jobs?repo=try&revision=e46703f15271becc535a10a35b3bcf9e77c1c3b3&selectedJob=31157378 Mac/Linux]<br />
Testing on Android, retesting and bug verification: Nightly


== Risk analysis ==
== Risk analysis ==


= References =
= References =
Test cases: [link]<br />
Test cases: [https://docs.google.com/spreadsheets/d/19OHlTaibS59AbZUcD1RwNqzEv50WcBayuIfnJZwyX24/edit#gid=1098947246 gdoc] or [https://testrail.stage.mozaws.net/index.php?/suites/view/294&group_by=cases:section_id&group_order=desc TestRail]
         
Meta bug:  Bug-[https://bugzilla.mozilla.org/show_bug.cgi?id=1299187 1299187]<br />
Meta bug:  Bug-[https://bugzilla.mozilla.org/show_bug.cgi?id=1299187 1299187]<br />


Line 282: Line 289:
||  || depends on the feature
||  || depends on the feature
|-
|-
|
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
|style="text-align:center;" |  || Not Applicable
|style="text-align:center;" |  || Not Applicable
|-
|-
Line 304: Line 311:
|-
|-
|-
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1318149  Bug 1318149] ||Crash  ||New   ||  TryBuild
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1318149  Bug 1318149] ||Crash  ||Works for Me   ||  TryBuild/Nightly
|-
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1318175 Bug 1318175]||Crash ||New ||TryBuild <br />
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1318175 Bug 1318175]||Crash ||Works for Me ||TryBuild/Nightly <br />
|-
|-
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1317196 Bug 1317196]||Netflix not working on Linux||Known Issue||Nightly
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1317196 Bug 1317196]||Netflix not working on Linux||Known Issue||Nightly
Line 316: Line 323:
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1308677 Bug 1308677]||Flash warnings in Nightly are out of control ||Known Issue||Nightly
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1308677 Bug 1308677]||Flash warnings in Nightly are out of control ||Known Issue||Nightly
|}
|}
</div>


= Sign off =
= Sign off =
Line 329: Line 335:


*Links for the tests run
*Links for the tests run
** Daily Smoke: [link]
** Full Test suite: [https://testrail.stage.mozaws.net/index.php?/plans/view/2119 Link]  
** Full Test suite: [Link]  
** Regression Test suite: [link] if needed/available
** Regression Test suite: [link] if needed/available


Line 369: Line 374:
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release||  
|style="text-align:center;" | Email to be sent
|style="text-align:center;" |  
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
Line 375: Line 380:
|-
|-
|  QA Signoff  - Aurora Release ||  
|  QA Signoff  - Aurora Release ||  
|style="text-align:center;"| Email to be sent  
|style="text-align:center;"| Email sent  
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
Line 381: Line 386:
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release ||  
|style="text-align:center;" | Email to be sent
|style="text-align:center;" |
|}
|}

Latest revision as of 18:41, 25 January 2017

QA/Symbols Exporting


Revision History


Date Version Author Description
11/10/2016 1.0 Abe Masresha Created first draft
11/10/2016 1.1 Grover Wimberly IV Fixed some wording in several sub-sections
1/13/2017 1.2 Abe Masresha Updated some links and added info on testing on android.

Overview

Purpose

The goal of this work is to make Firefox crash less for release users.

Scope

This wiki details the testing that will be performed by the project team for the Symbol Exporting project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • Testing will focus on:
Firefox launch and shutdown
Firefox child processes.
Sandboxing (content, GMP, plugin).
Exploratory testing for existing 3rd-party addons
Android basic functionality testing
  • How testing will be performed
Manual

Ownership

Mozilla Development

Benjamin Smedberg


QA Eng. Softvision Las Vegas
Michelle ->Stefan - Team Lead ]
Abe Masresha - Test Lead
Grover Wimberly IV - Tester
Kanchan Kumari - Tester
Justin Williams - Tester

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, OSX, Linux operating systems and Android devices.

Out of Scope

iOS Devices (Not affected by this work)
Automation will be handled by the developer (if needed)

Requirements for testing

Environments: Operating Systems

Windows 8.1
Windows 10
Windows 7
Ubuntu/Linux
OSX 10.12
Android

Firefox Channel and environment setups

Build

Windows

Mac/Linux

Test Strategy

Test Objectives

Verify feature works correctly with Firefox launch and shutdowns
Verify feature works correctly with Firefox child processes
Verify feature works correctly with sandboxing (content, GMP, plugin)
Verify feature works fine with existing 3rd-party addons
Verify basic functionalities are working properly on Android

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project
Study documentation/specs received from developers 11/10/2016 11/10/2016
QA - Test plan creation 11/10/2016 11/10/2016
QA - Test cases/Env preparation 11/10/2016 11/16/2016
QA - tryBuild 11/15/2016
QA - Nightly 1/20/2017 1/20/2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Try Builds:

Windows

Mac/Linux

Testing on Android, retesting and bug verification: Nightly

Risk analysis

References

Test cases: gdoc or TestRail

Meta bug: Bug-1299187

Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Certain tests are performed in a private window.
Multi-Process Enabled Enabled by default
Multi-process Disabled To be tested
Theme (high contrast) Not applicable
UI
Mouse-only operation To be tested
Keyboard-only operation To be tested
Display (HiDPI) Not applicable
Interraction (scroll, zoom) To be tested
Usable with a screen reader Not applicable
Usability and/or discoverability testing Is this feature user friendly
RTL build testing To be tested
Help/Support
Help/support interface required Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Make sure support documents are written and are correct.
Install/Upgrade
Feature installs as expected To be tested
Does sync work across Install To be tested
Requires install testing To be tested
Affects first-run or onboarding Not applicable
Does this affect partner builds? Partner build testing Not applicable
Enterprise
Enterprise administration Not applicable
Network proxies/autoconfig Not applicable
ESR behavior changes Not applicable
Locked preferences Not applicable
Data Monitoring
Temporary or permanent telemetry monitoring List of error conditions to monitor
Telemetry correctness testing Not applicable
Server integration testing Not applicable
Offline and server failure testing Not applicable
Load testing Not applicable
Add-ons
Addon API required? Not applicable
Comprehensive API testing Not applicable
Permissions Not applicable
Testing with existing/popular addons To be tested
Security
3rd-party security review Not applicable
Privilege escalation testing Not applicable
Fuzzing Not applicable
Web Compatibility depends on the feature
Testing against target sites To be Tested
Survey of many sites for compatibility Not applicable
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. Not Applicable
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS Desktop only
Interaction of this feature with other browser features Not Applicable

Test suite

Full Test suite - TestRail/ Google docs

Bug Work

Tracking bug – meta bug

Bugs
Bug No Summary Status Firefox Verion
Bug 1318149 Crash Works for Me TryBuild/Nightly
Bug 1318175 Crash Works for Me TryBuild/Nightly
Bug 1317196 Netflix not working on Linux Known Issue Nightly
Bug 1318430 Norton Security Toolbar does not install in Firefox Known Issue Nightly
Bug 1317995 Firefox 50 64-bit Stable channel is unable to play Farmville 2 Facebook Flash game Known Issue Nightly
Bug 1308677 Flash warnings in Nightly are out of control Known Issue Nightly

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Feature testing

  • Links for the tests run
    • Full Test suite: Link
    • Regression Test suite: [link] if needed/available

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release
QA Aurora - Full Testing
QA Signoff - Aurora Release Email sent
QA Beta - Full Testing
QA Signoff - Beta Release
Retrieved from "https://wiki.mozilla.org/index.php?title=Symbol_Exporting&oldid=1160971"