This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
|07/21/2016||1.0||Vlad Bacia||Created first draft|
|02/28/2017||1.1||Cosmin Muntean||General Updates|
|03/06/2017||1.2||Peter deHaan||Added new FF 54 Plans|
|04/25/2017||1.3||Cosmin Muntean & Paul Oiegas||Updates for Screenshots system add-on|
- 1 Overview
- 2 Testing summary
- 3 Requirements for testing
- 4 Test Strategy
- 5 Status
- 6 References
- 7 Testcases
- 8 Bug Work
- 9 Sign off
Firefox Screenshots (Project Name: Page Shot) is an in-browser screenshoting application that lets you save and share permanent copies of anything on the web. It was originally deployed through Test Pilot as an experiment.
This wiki details the testing that will be performed by the project team for the Screenshots project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- What will be tested
- How testing will be performed
- Engineering - Ian Bicking
- Engineering - Donovan Preston
- Engineering - Jared "6a68" Hirsch
- Engineering - Danny Coates
- UX - Bram Pitoyo
- Ops - Daniel "relud" Thorn (backup is Miles)
- PM for QA team - Rares Bologa
- Leading QA efforts - Paul Oiegas
- QA - Cosmin Muntean
- QA - Peter deHaan
- Product Manager - John Gruen
- Sr. Engineering Manager - Wil Clouser
- Engineering Program Manager - Cory Price
IRC - #screenshots
Contact email - firstname.lastname@example.org (goes to Screenshots staff)
Scope of Testing
Screenshots lets you save anything on the web, including pages that requires a login or content that changes or is personalized. Once you've created a shot, anyone you give the link to can view it; never worry about logins or disappearing content. Screenshots is built and hosted by Mozilla, so you know it's safe to use.
The scope of our testing is the Screenshots-desktop system add-on and its functionality. The testing effort for Screenshots will be invested on the following areas:
- integration: verify the system add-on integration with the current browser functionalities and UI;
- functionality: basic and advanced functionality to be verified according to the existing requirements;
- usability: Screenshots features must be easy to use and straightforward;
- load testing: Verify that the server can handle expected load; pageshot-loadtests
Out of Scope
The mobile implementation and testing.
Requirements for testing
Testing will be performed on the following OSes (both x86 & x64 infrastructures):
- Windows 7
- Windows 8.1
- Windows 10
- Ubuntu 14.04
- Ubuntu 16.04
- Mac OS X 10.10
- Mac OS X 10.11
- Mac OS X 10.12
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
|Ref||Function||Test Objective||Evaluation Criteria||Test Type||Owners|
|1||Screenshots Firefox button||Verify that the Screenshots button is displayed in the Firefox browser toolbar.||Screenshots scissor button is displayed and active in the toolbar.||Manual||Eng Team|
|2||Screenshots Onboarding tour||Verify that when the Screenshots button is clicked, the Onboarding tour is displayed.||Tour is displayed and describes the Screenshots functionality.||Manual||Eng Team|
|3||Screenshots functionality||Verify all Screenshots functionalities are working.|| 1. Save anything on the web.
2. Share content.
|3||Disable / Re-enable Screenshots system add-on||Verify that the system add-on can be disabled and enabled without any issues.|| 1. Disable Screenshots system add-on from "about:preferences"page. Or from "about:config"page, using extensions.screenshots.system-disabled pref ("True" for disabled and "False" for enabled).
2. Verify that the browser UI returned to default functionality on the affected areas.
3. Re-enable Screenshots system add-on and verify that the browser has changed back.
Firefox Screenshots system add-on can be found in the next builds:
- Nightly - starting with build from 2017-04-18
- Beta - starting with Beta 54.0b2 2017-04-25 (candidate)
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
|Project phase||Start Date||End Date|
|Study documentation/specs received from developers||07/19/2016||07/20/2016|
|QA - Test plan creation||07/21/2016||07/22/2016|
|QA - Test cases/Env preparation||09/01/2016||09/09/2016|
|QA - Beta Testing||04/25/2017|
Detail the tools to be used for testing, for example see the following table:
|Test plan creation||Mozilla wiki|
|Test case creation||TestRail|
|Test case execution||TestRail|
- Nightly - starting with build from [2017-04-18]
- Beta - starting with Beta 54.0b2 [2017-04-25]
|External dependencies||- We should do an end-to-end test of how we will integrate with the browser and the server.|
- We should determine what our exact requirements are for sample size.
- Look into Telemetry as a contingency plan.
|Screenshots system add-on compatibility and usability with content specific websites||- Pictures and Videos from embedded sources.|
- Private websites and private content.
|Scope||- All locales |
- Accessibility requirements == https://github.com/mozilla-services/screenshots/issues/2251
|Tests||- Lack of comprehensive server unit tests (https://github.com/mozilla-services/screenshots/issues/427)|
|Open issues||- Number of open issues not declining (https://github.com/mozilla-services/screenshots/issues), although there aren't currently any beta/release blockers identified.|
List and links for specs:
- Screenshots Wiki page.
- Screenshots Dev website.
- Screenshots Stage website.
- Screenshots Production website.
- Screenshots GitHub repository.
- Screenshots Privacy Statement.
- Screenshots Metrics.
- Screenshots go-live metrics plan.
- Screenshots UX and Measurements Acceptance Criteria.
- Testing the API
- [meta Shipping Firefox Screenshots]
Summary of testing scenarios
|Theme (high contrast)||Yes|
|Enforce PNG check when posting images||Yes||See #2301|
|Verify Screenshots is available in core set of supported locales||Yes||See: #2295, #2294, and #2286|
|Keyboard-only operation||No||See #2251|
|Interaction (scroll, zoom)||Yes|
|Usable with a screen reader||No||e.g. with NVDA|
|Usability and/or discoverability testing||Yes||Is this feature user friendly|
|Feature upgrades/downgrades data as expected||Yes|
|Does sync work across upgrades||No|
|Requires install testing||No|
|Verify user data is migrated from Test Pilot experiment||Yes||See #2260|
|Affects first-run or onboarding||No|
|Does this affect partner builds? Partner build testing||No||We won't have partner builds available for testing in the Nightly or Aurora channels|
|Verify that Screenshots can be "disabled"||Yes||It should be able to be enabled and disabled by a toggle in about:preferences or by pref in about:config|
|Temporary or permanent telemetry monitoring||-||Not covered by the Engineering QA team|
|Telemetry correctness testing||-||Not covered by the Engineering QA team|
| Verify pings are sent when clicking Save
| Verify pings are sent when clicking Cancel
| Verify pings are sent when clicking Download;
| Verify pings are sent when clicking My Shots;
| Verify pings are sent when clicking Save Visible;
| Verify pings are sent when clicking Save Full Page;
|Record if Screenshots is enabled or not via Telemetry||Yes||See #2304|
|Server integration testing||-||Not covered by the Engineering QA team|
|Offline and server failure testing||-||Not covered by the Engineering QA team|
|Load testing||Yes||See screenshots-loadtests page|
|If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.|
|Addon API required?||No|
|Comprehensive API testing||No|
|Testing with existing/popular add-ons||Yes|
|Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.|
|3rd-party security review||-||Not covered by the Engineering QA team|
|Privilege escalation testing||-||Not covered by the Engineering QA team|
|Fuzzing||-||Not covered by the Engineering QA team|
|Testing against target sites||Yes|
|Survey of many sites for compatibility||No|
|Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.|
|Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS||No|
|Interaction of this feature with other browser features||Yes|
- Full Test suite - Screenshots Full Functional (TestRail)
- Smoke Test suite - TBD if available/needed
- Regression Test suite - TBD if available/needed
- Should meet the UX and Measurements Acceptance Criteria
- All test cases should be executed
Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite - Full Functional run from 04/18/2017
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run - N/A
- Full Test suite - N/A
List of OSes covered by testing: Mac OS X 10.11.6, Ubuntu 16.04 x64, Windows 10 x64, Windows 7 x64
|Testing Prerequisites (specs, use cases)||[DONE]|
|Testing Infrastructure setup||No|
|Test Plan Creation||[DONE]|
|Test Cases Creation||[DONE]|
|Full Functional Tests Execution||[DONE]|
|Automation Coverage||-||Not on QA manual end.|
|Performance Testing||-||Not on QA manual end.|
|All Defects Logged||[IN PROGRESS]|
|Critical/Blockers Fixed and Verified||[IN PROGRESS]|
|Daily Status Report (email/etherpad statuses/ gdoc with results)||Status etherpad|
|Load testing||[IN PROGRESS]|
|QA Beta - Full Testing||[DONE]|
|QA Signoff - Beta Release||[DONE]||(YELLOW) Pre-Release Report sent on 07/21/2017|