QA/LZMA Support for Updater
Approvals Required / Received
The following individuals are required to/have approved this Test Plan:
|RyanVM||QA Reviewer||Product Integrity||n/a|
|Matt Howell||Software Engineer||Engineering||6/14/2017|
|David Durst||EPM||Product Management||n/a|
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
|6/7/2017||1.0||Abe Masresha||Created first draft|
|6/13/2017||1.1||Abe Masresha||Added link to automation tests and test cases|
|8/10/2017||1.2||Abe Masresha||added signoffs date and other links|
|8/14/2017||1.3||Abe Masresha||adding more information to the test plan|
- 1 Overview
- 2 Testing summary
- 3 Requirements for testing
- 4 Test Strategy
- 5 Status
- 6 References
- 7 Testcases
- 8 Bug Work
- 9 Sign off
The purpose of this work is to change the update file format using LZMA compression because LZMA compression provides more file compression benefit over the current bzip2 compression.
- Testing will focus on:
- Stability of the browser with this work.
- Whether browser update is applied successfully
- If the browser update is in the desired file format
- How testing will be performed:
- Automation is covered by the developer
QA Eng. Softvision Las Vegas - Nightly testing
QA - Beta testing
Scope of Testing
This feature will be tested manually on Windows, Mac, and Linux operating systems.
Automation is handled by the developer
Out of Scope
Mobile devices are not affected by this work so this testing does not include them
Requirements for testing
Windows 10 64bits Windows 7 32bits Ubuntu 16.04 32 and 64bits MacOSX 10.12
Channel dependent settings (configs) and environment setups
Firefox Nightly 56.0a1, Test environment setting is provided here
This patch was pushed to Mozilla-central on 7-31-2017, so all older Nightly should update through this update watershed.
Beta 56.0b3, Test environment setting is provided here
Feature was backed out Bug 1387231 so the new update watershed of beta will be 56b3.
To test on 56b3, the next beta build (56b4) should be avialable as this is an update related work.
Risk Assessment and Coverage
|ID||Description / Threat Description||Covered by Test Objective||Magnitude||Probability||Priority||Impact Score|
|RAC-1||Update may not be applied to the browser||TO-1||3-High||2-Possible||3-High||18|
|RAC-2||Update may not be applied successfully if the browser has add-on||TO-2||3-High||2-Possible||2-Medium||12|
|RAC-3||The update file may not be in desired MAR file format||TO-3||2-Moderate||1-Unlikely||2-Medium||4|
|RAC-4||May behave differently if update download is interrupted by no internet connection||TO-4||2-Moderate||1-Unlikely||2-Medium||4|
|RAC-5||May behave differently in different operating systems||TO-5||2-Moderate||1-Unlikely||2-Medium||4|
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
|Ref||Function||Test Objective||Evaluation Criteria||Test Type||RAC||Owners|
|1||Update the browser||Browser should update to its latest build||update the browser||Manual||RAC-1,RAC-2,RAC-4,RAC-5||Eng Team|
|2||Check the update file format||The update file should be in desired MAR file format||Check if the file is in MAR format. steps are here||Manual||RAC-3||Eng Team|
This section should contain links for builds with the feature -
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
|Project phase||Start Date||End Date|
|Study documentation/specs received from developers||6/7/2017||8/14/2017|
|QA - Test plan creation||6/7/2017||8/14/2017|
|QA - Test cases/Env preparation||7/31/2017||8/31/2017|
|QA - Nightly Testing||8/1/2017||8/1/2017|
|QA - Beta Testing||8/10/2017||9/8/2017|
Detail the tools to be used for testing, for example see the following table:
|Test plan creation||Mozilla wiki|
|Test case creation||TestRail|
|Test case execution||TestRail|
Landed on Nightly on: 7/31/2017
Merged to Beta on: Beta 56b3 will have the patch so start the testing when 56b4 is ready.
|Private Window||Yes||user should be able to get update from private window as well|
|Multi-Process Enabled||Yes||user should be able to get update from e10s window|
|Multi-process Disabled||Yes||user should be able to get update from Non e10s window as well|
|Theme (high contrast)||Yes||high theme shouldnot affect the update|
|Feature upgrades /Browser update||Yes||Will be covered on browser updates|
|Testing with existing/popular addons||Yes||Feature functionality will be tested with popular addons|
|Survey of popular sites for compatibility||Yes||Firefox will be tested for stability with popular sites|
|Interaction of this feature with other browser features||Yes||Feature will be tested for its interaction with other browser features|
Full Test suite - test rail
|Bug No||Summary||Status||Firefox Version|
|None in pre-beta testing cycle||56|
|None during Beta testing cycle||56|
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools)
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed
List of OSes that will be covered by testing: see 'Environments' section
Preliminary status report on Beta
Preliminary status report analysis shows that updating from Firefox 55.0bX, updates first to 56.0b3 and then to latest beta available on the channel. Also, update.mar files are correct and have the right format. Preliminary recommendation is to "ship it".
Pre-Release Sign-off - GREEN 2017-09-06
- List of OSes covered by testing: Windows 7 64-bit, Windows 10 64-bit, Ubuntu 16.04 64-bit and Mac OS X 10.11
- Link for the tests run - link
- Regression Test suite, No regression bugs were found
|Testing Prerequisites (specs, use cases)||[DONE]|
|Testing Infrastructure setup||[DONE]|
|Test Plan Creation||[DONE]|
|Test Cases Creation||[DONE]|
|Automation Coverage||[DONE]||Covered by developer|
|Performance Testing||[DONE]||Covered by developer|
|All Defects Logged||N/A||No bugs found on pre-beta signoff|
|Critical/Blockers Fixed and Verified||N/A|
|QA mid-Nightly Signoff||[MISSED]||Missed (test build was not ready)|
|QA pre-Beta Signoff||[DONE]||Email sent on 8/1/2017|
|QA Beta - Full Testing||[DONE]|
|QA pre-Release Signoff||[DONE]||Email sent on 9/6/2017|