QA/Fennec/Upgrade to API 26
Jump to navigation
Jump to search
Full Query
Overview
Purpose
The purpose of this wiki is to serve for the QA testing done for Fennec after upgrade to API Version 26
Ownership
Developer contact:
QA:
Overall Status
- [LANDED] Nightly 63
Testing Summary
- Google Play will require all app updates to target Android Oreo (targetSdkVersion 26 or higher) by November 2018. Fennec needs an upgrade to API 26, so full-functional, regression testing and stress testing are required to discover any compatibility issues.
Scope of testing
Testing will be focused on:
- Full functional coverage
- Regression and exploratory testing
- Visible performance issues
Environments
Testing will be performed on the following devices to provide a good coverage:
- Phones - Android versions:
- 6.x
- 7.x
- 8.x
- Tablets - Android versions:
- 4.x
- 5.x
- Both ARM and x86 architecture
Specifications
Fennec Upgrade to API 26 - https://bugzilla.mozilla.org/show_bug.cgi?id=1349309#c4
Acceptance Criteria
No compatibility or performance issues detected
Testing details
Test Cases
Fennec's BFT (basic functional testing) suite
Testing days
Date: 5/21/2018
- Created a test plan
Date: 7/6/2018
- Tested the local test build
- created a doc to keep track of the issues found: https://docs.google.com/spreadsheets/d/1WKsu8-ye9WnaUcgrY6Gg98yw1pHDRMwVRkH6V6SnqfA/edit#gid=0
Date: 7/9/2018
- Tested the local test build
Date: 7/19/2018
- Nightly smoke & exploratory: https://testrail.stage.mozaws.net/index.php?/runs/view/11259&group_by=tests:id&group_order=asc&group_id=-1
- all issues found should be marked as blocking Bug 1349309 - [meta] Android Oreo issues
- logged: Bug 1476966 - Constant notification ringing when downloading a file
- verified: Bug 1476596 - Mozilla Location Services enables itself
Date: 7/23/2018
- 26 API Upgrade for Fennec - BFTs - first run: https://testrail.stage.mozaws.net/index.php?/runs/view/11281&group_by=cases:section_id&group_order=asc&group_id=384
- logged: Bug 1477700 - Android media notification appears swiped down after video starts
Date: 7/25/2018
- Finished up BFTs - first run: https://testrail.stage.mozaws.net/index.php?/runs/view/11281&group_by=cases:section_id&group_order=asc&group_id=384
Date: 8/23/2018
- Finished the 2nd run of BFTs: https://testrail.stage.mozaws.net/index.php?/runs/view/11887&group_by=cases:section_id&group_order=asc&group_id=24347
Date: 8/24/2018
- Sent out [63] [fennec] [feature] Upgrade to API 26 - pre-Beta QA sign off (YELLOW)
Date: 9/28/2018
- 26 API Upgrade for Fennec - BFTs for Beta Sign Off - Samsung S8+ (Android 8.0.0) - https://testrail.stage.mozaws.net/index.php?/runs/view/12697
Date: 10/11/2018
- Upgrade to API 26 - pre-Release QA sign off (GREEN)
Bug Work
Tracking bugs
- Bugs to track the feature
- Meta bug: bug 1349309 - [meta] Android Oreo issues
- bug 1450450 - Target Android API 26
58 Total; 1 Open (1.72%); 33 Resolved (56.9%); 24 Verified (41.38%);
Signoff
Criteria
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed
Results
Nightly testing
- in progress in 63.0a1
- 07/27 mid-Nightly signoff: YELLOW
- 08/24 pre-Beta QA sign off (YELLOW)
Beta testing
- N/A
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | ||
| Testing Infrastructure setup | ||
| Test Plan Creation | Done | |
| Test Cases Creation | Done | |
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | N/A | |
| Basic/Core functionality Nightly testing | Not started | |
| QA mid-Nightly Signoff | Done | Status: YELLOW |
| QA Nightly - Full Testing | Done | |
| QA pre-Beta Signoff | Done | Status: YELLOW |
| QA Beta - Full Testing | Not started | |
| QA pre-Release Signoff | TBD | Status: |