QA/FindToolbarTesting: Difference between revisions
< QA
Jump to navigation
Jump to search
Brindusa.tot (talk | contribs) |
Brindusa.tot (talk | contribs) |
||
| (23 intermediate revisions by 3 users not shown) | |||
| Line 43: | Line 43: | ||
== Scope of Testing == | == Scope of Testing == | ||
=== In Scope === | === In Scope === | ||
Firefox's find toolbar has several usability problems that need to be addressed and redesigned. The most significant updates would be:<br /> | |||
- background dims when at least one result is found, highlights, is being more visible<br /> | |||
- first result found is displayed in a yellow square, highlighted. Next occurrences of the result are displayed with a white background.<br /> | |||
The testing efforts will be invested on the following areas: | |||
* functionality: find toolbar functionality on different web pages(small, large, different web page background, pages with different font size, pdf/txt files testing, etc) | |||
* usability: the find toolbar must be easy to use and straightforward; | |||
* performance: make sure that browser performance is not affected (check-boarding, scrolling) | |||
* accessibility: High Contrast themes, Keyboard operation | |||
=== Out of Scope === | === Out of Scope === | ||
| Line 50: | Line 59: | ||
= Requirements for testing = | = Requirements for testing = | ||
== Environments == | == Environments == | ||
Full Testing will be performed on: | |||
* Windows 7 | |||
* Windows 10 | |||
* Ubuntu 15.04 | |||
* Mac OS X 10.10 | |||
Exploratory/Smoke testing will be performed on | |||
* Windows XP | |||
* Windows 8.1 | |||
Specify other configuration/environmental setup needed: | |||
- Preference '''findbar.modalHighlight''' set on '''true''' will enable new implementation of Find Toolbar. On Nightly channel, this preference is set on true by default. | |||
= Test Strategy = | = Test Strategy = | ||
| Line 63: | Line 81: | ||
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners | ! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners | ||
|- | |- | ||
| 1 || | | 1 || Access Find Toolbar || The objective of these tests suites is to verify that the Find Bar can be opened from different browser locations, using mouse or keyboard || Find Toolbar can be opened from:<br /> | ||
1. Edit > Find menu<br /> | |||
2. Open Menu > Find<br /> | |||
3. Keyboard shortcut: CTRL+F/CMD +F | |||
|| Manual || Eng Team | |||
|- | |||
| 2 || Find Toolbar Functionality || The objective of these test suites is to verify that all requirements related to the Find Toolbar functionality are met and that they are functioning as expected || All the elements of the Find Toolbar are working according with the requirements|| Manual || Eng Team | |||
|- | |||
| 3 || Find Toolbar Usability ||The objective of these test suites is to verify that the application is user friendly|| The Find Toolbar is easy and straight forward to understand. The user has no difficulty in understanding any of the offered functionality.|| Manual||Eng Team | |||
|- | |||
| 4 || Find Toolbar Accessibility || Find Toolbar is visible and usable from accessibility point of view || The accessibility scenarios are successful ||Manual || Eng Team | |||
|- | |- | ||
| | | 5 || Browser Performance while find toolbar is opened || TBD || TBD || TBD || TBD | ||
|- | |- | ||
| | | 6 || Regression testing on the existing logged and fixed Bugzilla bugs || Verify marked as fixed bugs against the Nightly Builds || || || | ||
|} | |} | ||
== Builds == | == Builds == | ||
This section should contain links for builds with the feature | This section should contain links for builds with the feature: | ||
* Links for Nightly builds | * Links for Nightly builds - available since 06.10.2016 [http://archive.mozilla.org/pub/firefox/nightly/2016/06/2016-06-10-07-36-07-mozilla-central/ link] | ||
* Links for Aurora builds | * Links for Aurora builds | ||
* Links for Beta builds | * Links for Beta builds | ||
| Line 84: | Line 111: | ||
|- | |- | ||
| Start project | | Start project | ||
|style="text-align:center;" | | |style="text-align:center;" | 06/22/2016 || | ||
|- | |- | ||
| Study documentation/specs received from developers | | Study documentation/specs received from developers | ||
|style="text-align:center;" | | |style="text-align:center;" | 06/22/2016 | ||
|style="text-align:center;" | 06/24/2016 | |||
|- | |- | ||
| QA - Test plan creation | | QA - Test plan creation | ||
|style="text-align:center;" | | |style="text-align:center;" | 06/23/2016 | ||
|style="text-align:center;" | 06/28/2016 | |||
|- | |- | ||
| QA - Test cases/Env preparation | | QA - Test cases/Env preparation | ||
|style="text-align:center;" | | |style="text-align:center;" | 06/24/2016 | ||
|style="text-align:center;" | 06/29/2016 | |||
|- | |- | ||
| QA - Nightly Testing | | QA - Nightly Testing | ||
|style="text-align:center;" | | |style="text-align:center;" | 06/28/2016 | ||
|style="text-align:center;" | | |||
|- | |- | ||
| QA - Aurora Testing | | QA - Aurora Testing | ||
| Line 125: | Line 156: | ||
= Status = | = Status = | ||
== Overview == | == Overview == | ||
First releases to Nightly 50.0a1: 06.10.2015 [http://archive.mozilla.org/pub/firefox/nightly/2016/06/2016-06-10-07-36-07-mozilla-central/ Nighltly Build] | |||
At merge with Aurora, feature was not sign-off to Aurora due to large number of bugs opened on this feature. Feature remains disabled on Aurora 50 and enabled on Nightly 51 | |||
Continue testing on Nightly 51.0a1 - mostly focus on bug fixing. | |||
At merge with Aurora 51, feature was not sign-off to Aurora due to large number of bugs opened on this feature. Feature remains disabled on Aurora 51 and enabled on Nightly 52 | |||
Continue testing on Nightly 52.0a1 | |||
11/14/2016 - at merge to Aurora 52, feature remains enabled only on Nightly | |||
= References = | = References = | ||
* | * Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=384458 384458] | ||
* New Meta bug - starting with Nightly 51: bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1291278 1291278] | |||
* | * Useful info related to this feature can be found at following [https://public.etherpad-mozilla.org/p/Find_Toolbar_-_QA_Kick-off_meeting etherpad] | ||
= Testcases = | = Testcases = | ||
== | Testcases can be found at [https://docs.google.com/spreadsheets/d/1b6_dcsme2zE44vRB6V14qM_YvZmlAtoeLNCSaG9EDEc/edit#gid=0 link] | ||
Testrail link for full testcases at [https://testrail.stage.mozaws.net/index.php?/suites/view/204&group_by=cases:section_id&group_order=asc link] | |||
== Test Areas == | == Test Areas == | ||
| Line 149: | Line 178: | ||
|- | |- | ||
| Private Window | | Private Window | ||
|style="text-align:center;" | || | |style="text-align:center;" | Yes || | ||
|- | |- | ||
| Multi-Process Enabled | | Multi-Process Enabled | ||
|style="text-align:center;" | | |style="text-align:center;" | Yes || | ||
|- | |- | ||
| Multi-process Disabled | | Multi-process Disabled | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Theme (high contrast) | | Theme (high contrast) | ||
|style="text-align:center;" | | |style="text-align:center;" | Yes || | ||
|- | |- | ||
| '''UI''' | | '''UI''' | ||
| Line 164: | Line 193: | ||
|- | |- | ||
| Mouse-only operation | | Mouse-only operation | ||
|style="text-align:center;" | | |style="text-align:center;" | Yes || | ||
|- | |- | ||
| Keyboard-only operation | | Keyboard-only operation | ||
|style="text-align:center;" | | |style="text-align:center;" | Yes || | ||
|- | |- | ||
| Display (HiDPI) | | Display (HiDPI) | ||
|style="text-align:center;" | || | |style="text-align:center;" | Yes || | ||
|- | |- | ||
| Interraction (scroll, zoom) | | Interraction (scroll, zoom) | ||
|style="text-align:center;" | || | |style="text-align:center;" | Yes || | ||
|- | |- | ||
| Usable with a screen reader | | Usable with a screen reader | ||
|style="text-align:center;" | | |style="text-align:center;" | ?? || e.g. with NVDA | ||
|- | |- | ||
| Usability and/or discoverability testing | | Usability and/or discoverability testing | ||
|style="text-align:center;" | | |style="text-align:center;" | ?? || Is this feature user friendly | ||
|- | |- | ||
| Line 186: | Line 215: | ||
|- | |- | ||
| Help/support interface required | | Help/support interface required | ||
|style="text-align:center;" | | |style="text-align:center;" | No || Make sure link to support/help page exist and is easy reachable. | ||
|- | |- | ||
| Support documents planned(written) | | Support documents planned(written) | ||
|style="text-align:center;" | | |style="text-align:center;" | No || Make sure support documents are written and are correct. | ||
|- | |- | ||
| Line 196: | Line 225: | ||
|- | |- | ||
| Feature upgrades/downgrades data as expected | | Feature upgrades/downgrades data as expected | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Does sync work across upgrades | | Does sync work across upgrades | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Requires install testing | | Requires install testing | ||
|style="text-align:center;" | | |style="text-align:center;" | No || separate feature/application installation needed (not only Firefox) | ||
|- | |- | ||
| Affects first-run or onboarding | | Affects first-run or onboarding | ||
|style="text-align:center;" | | |style="text-align:center;" | No || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned. | ||
|- | |- | ||
| Does this affect partner builds? Partner build testing | | Does this affect partner builds? Partner build testing | ||
|style="text-align:center;" | | |style="text-align:center;" | ?? || yes/no options, add comment with details about who will lead testing | ||
|- | |- | ||
| Line 215: | Line 244: | ||
|- | |- | ||
| Enterprise administration | | Enterprise administration | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Network proxies/autoconfig | | Network proxies/autoconfig | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| ESR behavior changes | | ESR behavior changes | ||
|style="text-align:center;" | | |style="text-align:center;" | ?? || | ||
|- | |- | ||
| Locked preferences | | Locked preferences | ||
|style="text-align:center;" | | |style="text-align:center;" | ?? || | ||
|- | |- | ||
| Line 231: | Line 260: | ||
|- | |- | ||
| Temporary or permanent telemetry monitoring | | Temporary or permanent telemetry monitoring | ||
|style="text-align:center;" | | |style="text-align:center;" | No || List of error conditions to monitor | ||
|- | |- | ||
| Telemetry correctness testing | | Telemetry correctness testing | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Server integration testing | | Server integration testing | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Offline and server failure testing | | Offline and server failure testing | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Load testing | | Load testing | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Line 250: | Line 279: | ||
|- | |- | ||
| Addon API required? | | Addon API required? | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Comprehensive API testing | | Comprehensive API testing | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Permissions | | Permissions | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Testing with existing/popular addons | | Testing with existing/popular addons | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Line 266: | Line 295: | ||
|- | |- | ||
| 3rd-party security review | | 3rd-party security review | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Privilege escalation testing | | Privilege escalation testing | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Fuzzing | | Fuzzing | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Line 279: | Line 308: | ||
|- | |- | ||
| Testing against target sites | | Testing against target sites | ||
|style="text-align:center;" | | |style="text-align:center;" | Yes || | ||
|- | |- | ||
| Survey of many sites for compatibility | | Survey of many sites for compatibility | ||
|style="text-align:center;" | | |style="text-align:center;" | Yes || | ||
|- | |- | ||
| Line 289: | Line 318: | ||
|- | |- | ||
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | | Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | | Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|- | |- | ||
| Interaction of this feature with other browser features | | Interaction of this feature with other browser features | ||
|style="text-align:center;" | | |style="text-align:center;" | No || | ||
|} | |} | ||
| Line 304: | Line 333: | ||
= Bug Work = | = Bug Work = | ||
Meta: [https://bugzilla.mozilla.org/show_bug.cgi?id=1291278 Bug 1291278 - [meta<nowiki>]</nowiki> Get dimmed highlighting mode ready to ship] | |||
< | |||
== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1291278 1291278] ) == | |||
< | <bugzilla> | ||
{ | |||
"blocks":[1291278], | |||
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone" | |||
</ | } | ||
</bugzilla> | |||
== Bug Verification (blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1291278 1291278]) == | |||
<bugzilla> | |||
{ | |||
"blocks":[1291278], | |||
"resolution":"FIXED", | |||
"include_fields": "id, priority, component, assigned_to, summary, status, target_milestone" | |||
} | |||
</bugzilla> | |||
= Sign off = | = Sign off = | ||
| Line 334: | Line 356: | ||
Check list | Check list | ||
* All test cases should be executed | * All test cases should be executed | ||
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA) | * All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA) | ||
| Line 342: | Line 363: | ||
List of OSes that will be covered by testing<br /> | List of OSes that will be covered by testing<br /> | ||
*Link for the tests run | *Link for the tests run | ||
** | ** Full Test suite, run on Nighlty 50.0a1 [https://docs.google.com/spreadsheets/d/1b6_dcsme2zE44vRB6V14qM_YvZmlAtoeLNCSaG9EDEc/edit#gid=0 link] | ||
** Full Test suite, | ** Full Test suite, run on Nighlty 52.0a1 [https://testrail.stage.mozaws.net/index.php?/plans/view/886 link] | ||
'''Merge to Aurora Sign-off''' | '''Merge to Aurora Sign-off''' | ||
| Line 357: | Line 378: | ||
|- | |- | ||
| Testing Prerequisites (specs, use cases) | | Testing Prerequisites (specs, use cases) | ||
| style="text-align:center;" | | | style="text-align:center;" | {{mdone|}} | ||
| style="text-align:center;" | | | style="text-align:center;" | | ||
|- | |- | ||
| Testing Infrastructure setup | | Testing Infrastructure setup | ||
|style="text-align:center;" | | |style="text-align:center;" | {{mdone|}} || | ||
|- | |- | ||
| Test Plan Creation | | Test Plan Creation | ||
| style="text-align:center;" | | | style="text-align:center;" | {{mdone|}} || | ||
|- | |- | ||
| Test Cases Creation | | Test Cases Creation | ||
|style="text-align:center;" | | |style="text-align:center;" | {{mdone|}} || | ||
|- | |- | ||
| Full Functional Tests Execution | | Full Functional Tests Execution | ||
|style="text-align:center;" | | |style="text-align:center;" | {{mdone|}} | ||
|style="text-align:center;" | | |style="text-align:center;" | | ||
|- | |- | ||
| Automation Coverage | | Automation Coverage | ||
|style="text-align:center;" | | |style="text-align:center;" | TBD || | ||
|- | |- | ||
| Performance Testing | | Performance Testing | ||
|style="text-align:center;" | | |style="text-align:center;" | TBD || | ||
|- | |- | ||
| All Defects Logged || || | | All Defects Logged | ||
|style="text-align:center;" | {{mdone|}} | |||
|style="text-align:center;" | | |||
|- | |- | ||
| Critical/Blockers Fixed and Verified | | Critical/Blockers Fixed and Verified | ||
|- | |style="text-align:center;" | {{mdone|}} | ||
| | |style="text-align:center;" | | ||
|- | |- | ||
| Metrics/Telemetry|| | | Metrics/Telemetry|| | ||
Latest revision as of 15:06, 14 November 2016
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 06/23/2016 | 1.0 | Brindusa Tot | Created first draft |
Overview
Purpose
Detail the purpose of this document. For example:
- The test scope, focus areas and objectives
- The test responsibilities
- The test strategy for the levels and types of test for this release
- The entry and exit criteria
- The basis of the test estimates
- Any risks, issues, assumptions and test dependencies
- The test schedule and major milestones
- The test deliverables
Scope
This wiki details the testing that will be performed by the project team for the Find Toolbar project. Its purpose is to document:
- What will be tested
- How testing will be performed
Ownership
Engineering contact: Mike de Boer
QA:
Rares Bologa - PM for QA team
Brindusa Tot - leading QA efforts
Liviu Cirdei, QA
Testing summary
Scope of Testing
In Scope
Firefox's find toolbar has several usability problems that need to be addressed and redesigned. The most significant updates would be:
- background dims when at least one result is found, highlights, is being more visible
- first result found is displayed in a yellow square, highlighted. Next occurrences of the result are displayed with a white background.
The testing efforts will be invested on the following areas:
- functionality: find toolbar functionality on different web pages(small, large, different web page background, pages with different font size, pdf/txt files testing, etc)
- usability: the find toolbar must be easy to use and straightforward;
- performance: make sure that browser performance is not affected (check-boarding, scrolling)
- accessibility: High Contrast themes, Keyboard operation
Out of Scope
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.
Requirements for testing
Environments
Full Testing will be performed on:
- Windows 7
- Windows 10
- Ubuntu 15.04
- Mac OS X 10.10
Exploratory/Smoke testing will be performed on
- Windows XP
- Windows 8.1
Specify other configuration/environmental setup needed:
- Preference findbar.modalHighlight set on true will enable new implementation of Find Toolbar. On Nightly channel, this preference is set on true by default.
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
| Ref | Function | Test Objective | Evaluation Criteria | Test Type | Owners |
|---|---|---|---|---|---|
| 1 | Access Find Toolbar | The objective of these tests suites is to verify that the Find Bar can be opened from different browser locations, using mouse or keyboard | Find Toolbar can be opened from: 1. Edit > Find menu |
Manual | Eng Team |
| 2 | Find Toolbar Functionality | The objective of these test suites is to verify that all requirements related to the Find Toolbar functionality are met and that they are functioning as expected | All the elements of the Find Toolbar are working according with the requirements | Manual | Eng Team |
| 3 | Find Toolbar Usability | The objective of these test suites is to verify that the application is user friendly | The Find Toolbar is easy and straight forward to understand. The user has no difficulty in understanding any of the offered functionality. | Manual | Eng Team |
| 4 | Find Toolbar Accessibility | Find Toolbar is visible and usable from accessibility point of view | The accessibility scenarios are successful | Manual | Eng Team |
| 5 | Browser Performance while find toolbar is opened | TBD | TBD | TBD | TBD |
| 6 | Regression testing on the existing logged and fixed Bugzilla bugs | Verify marked as fixed bugs against the Nightly Builds |
Builds
This section should contain links for builds with the feature:
- Links for Nightly builds - available since 06.10.2016 link
- Links for Aurora builds
- Links for Beta builds
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 06/22/2016 | |
| Study documentation/specs received from developers | 06/22/2016 | 06/24/2016 |
| QA - Test plan creation | 06/23/2016 | 06/28/2016 |
| QA - Test cases/Env preparation | 06/24/2016 | 06/29/2016 |
| QA - Nightly Testing | 06/28/2016 | |
| QA - Aurora Testing | ||
| QA - Beta Testing | ||
| Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | Google docs |
| Test case execution | Google docs |
| Bugs management | Bugzilla |
Status
Overview
First releases to Nightly 50.0a1: 06.10.2015 Nighltly Build At merge with Aurora, feature was not sign-off to Aurora due to large number of bugs opened on this feature. Feature remains disabled on Aurora 50 and enabled on Nightly 51 Continue testing on Nightly 51.0a1 - mostly focus on bug fixing. At merge with Aurora 51, feature was not sign-off to Aurora due to large number of bugs opened on this feature. Feature remains disabled on Aurora 51 and enabled on Nightly 52 Continue testing on Nightly 52.0a1 11/14/2016 - at merge to Aurora 52, feature remains enabled only on Nightly
References
- Meta bug 384458
- New Meta bug - starting with Nightly 51: bug 1291278
- Useful info related to this feature can be found at following etherpad
Testcases
Testcases can be found at link Testrail link for full testcases at link
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | Yes | |
| Multi-Process Enabled | Yes | |
| Multi-process Disabled | No | |
| Theme (high contrast) | Yes | |
| UI | ||
| Mouse-only operation | Yes | |
| Keyboard-only operation | Yes | |
| Display (HiDPI) | Yes | |
| Interraction (scroll, zoom) | Yes | |
| Usable with a screen reader | ?? | e.g. with NVDA |
| Usability and/or discoverability testing | ?? | Is this feature user friendly |
| Help/Support | ||
| Help/support interface required | No | Make sure link to support/help page exist and is easy reachable. |
| Support documents planned(written) | No | Make sure support documents are written and are correct. |
| Install/Upgrade | ||
| Feature upgrades/downgrades data as expected | No | |
| Does sync work across upgrades | No | |
| Requires install testing | No | separate feature/application installation needed (not only Firefox) |
| Affects first-run or onboarding | No | Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned. |
| Does this affect partner builds? Partner build testing | ?? | yes/no options, add comment with details about who will lead testing |
| Enterprise | Raise up the topic to developers to see if they are expecting to work different on ESR builds | |
| Enterprise administration | No | |
| Network proxies/autoconfig | No | |
| ESR behavior changes | ?? | |
| Locked preferences | ?? | |
| Data Monitoring | ||
| Temporary or permanent telemetry monitoring | No | List of error conditions to monitor |
| Telemetry correctness testing | No | |
| Server integration testing | No | |
| Offline and server failure testing | No | |
| Load testing | No | |
| Add-ons | If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | |
| Addon API required? | No | |
| Comprehensive API testing | No | |
| Permissions | No | |
| Testing with existing/popular addons | No | |
| Security | Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | |
| 3rd-party security review | No | |
| Privilege escalation testing | No | |
| Fuzzing | No | |
| Web Compatibility | depends on the feature | |
| Testing against target sites | Yes | |
| Survey of many sites for compatibility | Yes | |
| Interoperability | depends on the feature | |
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | No | |
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | No | |
| Interaction of this feature with other browser features | No |
Test suite
Full Test suite - Link with the gdoc, follow the format from link Smoke Test suite - Link with the gdoc, follow the format from link Regression Test suite - Link with the gdoc - if available/needed.
Bug Work
Meta: Bug 1291278 - [meta] Get dimmed highlighting mode ready to ship
Logged bugs ( blocking 1291278 )
134 Total; 29 Open (21.64%); 60 Resolved (44.78%); 45 Verified (33.58%);
Bug Verification (blocking 1291278)
60 Total; 0 Open (0%); 15 Resolved (25%); 45 Verified (75%);
Sign off
Criteria
Check list
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
Merge to Aurora Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | [DONE] | |
| Testing Infrastructure setup | [DONE] | |
| Test Plan Creation | [DONE] | |
| Test Cases Creation | [DONE] | |
| Full Functional Tests Execution | [DONE] | |
| Automation Coverage | TBD | |
| Performance Testing | TBD | |
| All Defects Logged | [DONE] | |
| Critical/Blockers Fixed and Verified | [DONE] | |
| Metrics/Telemetry | ||
| QA Signoff - Nightly Release | Email to be sent | |
| QA Aurora - Full Testing | ||
| QA Signoff - Aurora Release | Email to be sent | |
| QA Beta - Full Testing | ||
| QA Signoff - Beta Release | Email to be sent |