MozillaWiki

QA/Turning On Search Suggestions: Difference between revisions

Page Discussion

QA/Turning On Search Suggestions (view source)

Revision as of 14:54, 7 June 2017

2,087 bytes added ,  7 June 2017
→‎Test Objectives
 
(21 intermediate revisions by the same user not shown)
Line 44: Line 44:
Turning On Search Suggestions is a Mozilla feature that aims to enable the search suggestions by default and give the users the possibility to disable them (before Firefox 55, the search suggestions were disabled by default and the users were asked to enable them).  
Turning On Search Suggestions is a Mozilla feature that aims to enable the search suggestions by default and give the users the possibility to disable them (before Firefox 55, the search suggestions were disabled by default and the users were asked to enable them).  


With this feature, a new hint is displayed in the Awesome's bar dropdown informing the users (that had the search suggestions enabled or that ignored the enabling/disabling of them) about the enabling of the search suggestions. The users that had the search suggestions blocked previously, won't see the hint and the search suggestions won't be enabled in their case.
With this feature, a new hint is displayed in the Awesome's bar dropdown informing the users (that ignored the enabling/disabling of the search suggestions or on clean installs) about the enabling of the search suggestions. The users that previously allowed the search suggestions or blocked them, won't see the hint.


=== Out of Scope ===
=== Out of Scope ===
Line 69: Line 69:
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
|-
|-
| RAC-1 || Risk description 1 || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
| RAC-1 || Awesomebar and One offs functionality/performance might be affected by the enabling of the search suggestions || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
|-
| RAC-2 || Risk description 2 || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
| RAC-2 || Enabling the search suggestions by default confuses users because the search suggestions hint not noticeable enough || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
|-
|-
| RAC-3 || Risk description 3 || TO-2 || 2-Moderate || 2-Possible || 3-High || 12
| RAC-3 || The search suggestions hint is displayed also to the non targeted users  || TO-2 || 2-Moderate || 2-Possible || 3-High || 12
|-
| RAC-4 || The search suggestions are enabled also for the non targeted users  || TO-2 || 2-Moderate || 2-Possible || 3-High || 12
|}
|}


Line 97: Line 99:
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
|-
|-
| 1 || Search suggestions hint || Search suggestions hint is displayed for the users that do not have the search suggestions blocked || The hint informing users that search suggestions are enabled is displayed in the Awesom bar drop down just under the URL || Manual || RAC-1, RAC-2, RAC-3 || Eng Team
| 1 || Search suggestions hint || Search suggestions hint is displayed for the users that do not have the search suggestions blocked || The hint informing users that search suggestions are enabled is displayed in the Awesome bar drop down just under the URL || Manual || RAC-1, RAC-2, RAC-3 || Eng Team
|-
| 2 || Search suggestions disabling || Search suggestions in the Awesome Bar can be disabled || Clicking on Change Settings in the search suggestions hint takes you to the about:preferences#general from where search suggestions can be disabled || Manual  || RAC-1, RAC-2, RAC-3 || Eng Team
|-
|-
| 2 || Search suggestions disabling || Search suggestions in the Awesome Bar can be disabled || Clicking on Change Settings in the hint takes you to the about:preferences#general from where search suggestions can be disabled || Manual || || Eng Team
| 3 || Accessibility || Search suggestions and the hint announcing them being enabled are visible and usable from accessibility point of view|| Testing for accessibility includes: <br />
- testing using high Contract themes <br />
- Screen Reader validation<br />
- Keyboard-only 
|| Manual || RAC-1, RAC-2, RAC-3 || Eng Team
|-
|-
| 3 || || || || || ||
| 4 || Upgrade-Downgrade || The search suggestions hint behavior at FF Upgrade/Downgrade || One-Off searches Awesomebar buttons are not lost during FF upgrade/downgrade || Manual || RAC-1, RAC-2, RAC-3, RAC-4 || Eng Team
|}
|}


== Builds ==
== Builds ==
This section should contain links for builds with the feature -
This section should contain links for builds with the feature  
* Links for Nightly builds
* Links for Nightly builds:
** First Landed on Nightly 55.0a1 - on 2017-05-24 - [http://archive.mozilla.org/pub/firefox/nightly/2017/05/2017-05-24-03-02-04-mozilla-central/ link for Windows and Mac OS X], [http://archive.mozilla.org/pub/firefox/nightly/2017/05/2017-05-24-10-02-15-mozilla-central/ link for Ubuntu]
 
* Links for Beta builds
* Links for Beta builds


Line 116: Line 126:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.04.04 || TBD
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.04.04 || 2017.04.05
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.05.11 || 2017.05.11
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.04.04 || 2017.04.06
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.05.25 || 2017.06.05
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
Line 159: Line 169:


= References =
= References =
* Meta bug:
** [https://bugzilla.mozilla.org/show_bug.cgi?id=1344919 1344919 - META - search suggestions and one-off buttons in Awesome Bar] <br />
* List and links for specs
* List and links for specs
  List and links for available specs - documents, user stories, specifications
** Useful info about Awesomebar search at [https://support.mozilla.org/en-US/kb/search-suggestions-firefox link]
* Meta bug
 
= Testcases =  
= Testcases =  
Full test suites can be found on TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/813&group_by=cases:section_id&group_order=asc link] or in the  PDF doc - to addthe pdf
== Test Areas ==
== Test Areas ==
{| class="wikitable" style="width:80%"
{| class="wikitable" style="width:80%"
Line 170: Line 184:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 185: Line 199:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Interaction (scroll, zoom)  
| Interaction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | Yes || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | Yes || Is this feature user friendly
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes ||
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 209: Line 223:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | No || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | No || Make sure support documents are written and are correct.


|-
|-
Line 219: Line 233:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | No || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | NA || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | NA || yes/no options, add comment with details about who will lead testing


|-
|-
Line 238: Line 252:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 254: Line 268:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | No || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 273: Line 287:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 289: Line 303:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 302: Line 316:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 312: Line 326:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]
  Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/813&group_by=cases:section_id&group_order=asc TestRail Link] - [https://wiki.mozilla.org/images/c/c6/Turning_on_Search_Suggestions_-EngQA-_-_TestRail.pdf PDF doc]
Smoke Test suite - Link with the tests - if available/needed.
Regression Test suite - Link with the tests - if available/needed.


= Bug Work =
= Bug Work =
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=1344919 1344919- META - search suggestions and one-off buttons in Awesome Bar]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1344919 1344919] )======


<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[12345],
         "blocks":[1344919],
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
     }
     }
Line 348: Line 360:
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[12345],
         "blocks":[1344919],
         "resolution":"FIXED",
         "resolution":"FIXED",
         "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
Line 357: Line 369:


= Sign off =
= Sign off =
== Criteria ==
== Criteria for Mid Nightly ==
Checklist
Check list
* All the smoke tests (designed to ensure the main functionality is working as expected) are executed.
 
== Criteria for Pre-Beta Release and Pre-Release ==
Check list
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


== Results ==
== Results ==
'''Nightly testing'''<br />
'''Mid Nightly testing'''<br />


List of OSes that will be covered by testing<br />
- was missed due to the fact the feature landed after the Mid Nightly sign off.
*Link for the tests run
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]
** Daily Smoke, if needed/available
** Regression Test suite, if needed/available
<br />


'''Merge to Beta Sign-off'''<br />
'''Pre Beta testing'''<br />
List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
*Link for the tests run - [https://testrail.stage.mozaws.net/index.php?/plans/view/4371 TestRail link] [https://wiki.mozilla.org/images/5/56/Manual_Testing_Summary_Turning_on_Search_Suggestions_%28Fx55%29_-_Pre-Beta_feature_sign_off_%282017-06-06%29_-_TestRail.pdf PDF doc]
** Full Test suite


== Checklist ==
== Checklist ==
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:80%"
|-
|-
! Exit Criteria !! Status !! Notes/Details
! Exit Criteria !! Status !! Notes/Details
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |   
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Automation Coverage ||
|  Automation Coverage
|style="text-align:center;" |  
|style="text-align:center;" | N/A
|style="text-align:center;" |
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | {{mprog|}}
|style="text-align:center;" | 
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | {{mprog|}}
|style="text-align:center;" | 
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry
|style="text-align:center;" | N/A
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  Basic/Core functionality Nightly testing
|  QA mid-Nightly Signoff
|style="text-align:center;" | 
|style="text-align:center;" | {{missed|}}
|style="text-align:center;" | 
|style="text-align:center;" |
|-
|  QA mid-Nightly Signoff||
|style="text-align:center;" | Email to be sent
|-
| QA Nightly - Full Testing
|style="text-align:center;" |  ||  
|-
|-
|  QA pre-Beta Signoff||  
|  QA pre-Beta Signoff
|style="text-align:center;"| Email to be sent  
|style="text-align:center;" | {{done|}}
|style="text-align:center;" | [[https://wiki.mozilla.org/images/6/61/-desktop-_-feature-_Turning_on_Search_Suggestions_-_pre-Beta_sign_off_%28YELLOW%29.pdf  Email sent in 6/06/2017]]
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
Simona Badau
Confirmed users
1,333

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1170949...1172894"