QA/Search refactoring: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
 
(35 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= Overview =
= Overview =
== Summary ==
== Purpose ==
The purpose of this wiki is to serve as a general test plan for verifying Desktop Search engine works as expected after the refactoring.
The purpose of this wiki is to serve as a general test plan for verifying Desktop Search engine works as expected after the refactoring.


The testing will be focus on:
== Scope ==
The testing will be focus on
* verify that search works correct
* verify that search works correct
* verify that customized user data/profile is not affected after update
* verify that customized user data/profile is not affected after update


Testing will be performed manually, by QA engineering team.
   
   
== Ownership ==
== Ownership ==
 
Developer contacts: [mailto:florian@queze.net Florian Queze] <br />
Developer contacts: Florian Queze<br />
QA:<br />
QA:<br />


Line 19: Line 20:
[mailto:ovidiu.boca@softvision.ro Ovidiu Boca],  QA
[mailto:ovidiu.boca@softvision.ro Ovidiu Boca],  QA


= Status =
= Testing Summary =  
== Overview ==
== Scope of testing ==
<small>'''<font color="green">[LANDED]</font>'''</small> Nightly 45: first landed on - '''10.29.2015'''
=== In Scope ===
 
The main testing scope is to assure that the search works correct. For this fix, testing will include steps where users updates profile and we should verify that the user profile is not affected.
== Risk analysis==
- No Risks found at the moment
 
= References =
Meta: [https://bugzilla.mozilla.org/show_bug.cgi?id=1203167 Bug 1203167]


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Testing will be performed on following OSes:
Testing will be performed on following OSes (both x86 & x64 infrastructures):
* Mac OS X 10
* Mac OS X 10
* Windows 7
* Windows 7
* Windows 8
* Windows 8
* Windows 8.1
* Windows 10
* Windows 10
* Windows XP
* Windows XP
Line 40: Line 37:
* Ubuntu 14.04
* Ubuntu 14.04


== Test Types ==
Testing on specific devices is not needed.


= Test Strategy =
== Test Objectives ==
Following table contains main features to be tested:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Type of Testing !! Required (Y/N) !! Owners !! Details/Comments
! Ref !! Function !! Test Objective !! Evaluation Criteria
|-
| 1 || Search - basic scenarios || Search using different settings and from different broswer locations || Search is performed correctly.
|-
| 2 || Search - user profile verification after update || Validate that after update to latest firefox user profile is not affected || User profile is not updated
|-
| 3 || Search - hijack search with external apps || Try to hijack firefox search engine || Validate that search engine are not hijacked
|}
 
== Test Types ==
{|class="wikitable" style="width:60%"
|-
! Type of Testing !! Required (Yes/No) !! Owners !! Details/Comments
|-
|-
| Manual|| Y || Moz QA eng team ||
| Manual
|style="text-align:center;" | Yes
|style="text-align:center;" | Moz QA eng team ||
|-
|-
| Automation || Y || ||
| Automation  
|style="text-align:center;" | Yes
|style="text-align:center;" | Florian Queze ||
|-
|-
| Performance || N ||  ||
| Performance  
|style="text-align:center;" | No ||  ||
|-
|-
| Security || N ||  ||
| Security  
|style="text-align:center;" | No ||  ||
|-
|-
| Metrics/Telemetry|| TBD||  ||
| Usability
|style="text-align:center;" | No ||  ||
|-
| Metrics/Telemetry
|style="text-align:center;" | TBD||  ||
|}
|}


= Test cases =
== Builds ==
== Overview ==
This section should contain links for builds with the feature -
* Perform search from the Search Bar – using different terms
* Perform search from all the available search fields
* Add/Update search engine and validate search results is given by the new engine
* Change search settings (select/deselect engine, remove/restore engine, reorder engines) and validate search results is given by the new engine
* Verify if custom search engine settings (new engines added, reordered engines) are preserved after update
* Verify that after update, search engines settings can be updates
* Install external software's that automatically change the search engine for installed browsers and verify that the default search engine or users custom search engine is not altered


== Test suites ==
Nightly builds with the search refactor fix can be reached at [http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ link]
Notes: Testing will be performed with multi-process Nightly Enabled and Disabled
Manual/Automation: Test will be conducted manually <br />
Browser Window: Test will be conducted in standard browser window and Private browser windows


 
== Test Execution Schedule ==
{| class="wikitable"
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
|-
! Project phase !! Start Date !! End Date
|-
| Start project
|style="text-align:center;" | 10.20.2015 ||
|-
| Study documentation/specs received from developers
|style="text-align:center;" | 10.20.2015 ||  -
|-
| QA - Test plan creation
|style="text-align:center;" | 10.23.2015 ||
|-
|-
! Title !! Description
| QA - Test cases/Env preparation
|style="text-align:center;" | 10.23.2015  || 10.30.2015
|-
|-
| Perform a search from the Search Bar without using a search term|| Launch Firefox.
| QA - Nightly Testing
Click the Search Bar from the toolbar.
|style="text-align:center;" | 10.30.2015 || 11.25.2015
Don't write anything and press [enter].
|-
|-
| Perform a search from the Search Bar using a specific search term || Enter a search term in the Search Bar.
| QA - Aurora Testing
Click the go icon (->)
|style="text-align:center;" | 11.30.2015 || 01.25.2016
|-
|-
| Change the search provider using the Search Bar
| QA - Beta Testing
 
|style="text-align:center;" | 01.26.2016 ||
|| Select "Change search settings" when the search drop-down panel is opened.
Set a different provider as default search engine.
Perform a search for a random keyword.
|-
|-
| Perform searches from all the available search fields
| Release Date
|style="text-align:center;" | ||
|}


 
== Testing Tools ==
|| Perform a search from the Location Bar.<br />
Detail the tools to be used for testing, for example see the following table:
Perform a search from the Search Bar.<br />
{| class="wikitable" style="width:60%"
Perform a search from the about:newtab page. <br />
Perform a search from the about:home page.
|-
|-
| Search toolbar
! Process !! Tool
|| Launch Firefox.
Click the Search bar text box.<br />
Don't write anything and press Enter<br />
Click the down arrow next to the magnifying glass icon<br />
Select "Change Search Settings" and choose another search engine as default from the Default Search Engine drop-down. Perform searches from search toolbar, URL toolbar, about:newtab and about:home<br />
Restart Firefox and check the default search engine
|-
|-
| Change Search Settings - Add new search engine Youtube || Launch Firefox and open a new tab (Ctrl/Cmd+T).
| Test plan creation || Mozilla wiki
Open the drop down from the Search box field by clicking on the magnifying glass icon and select "Change Search Settings"<br />
Select "Add more search engines..." link<br />
Install an add-on from the provided list (for example Youtube) and then check it to be the default search engine, when asked<br />
Search using the search field from the new tab page, search bar, URL bar, and about:home page.<br />
Restart Firefox and check the search engines installed
|-
|-
| Change Search Settings - Add new search engine - other|| Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
| Test case creation || Google docs
Navigate to pages that could contain different search engines who can be added to Firefox. Select the magnifying glass from the search toolbar and add them to the search engines list<br />
Add several search engines <br />
Set one of the added engine as default.<br />
Search using the search field from the new tab page, search bar, URL bar, and about:home page.<br />
Restart Firefox and check the search engines installed
|-
|-
| Change Search Settings - Update default search engine || Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
| Test case execution || Google docs
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
Update the default engine, for e.g. set eBay the default engine.<br />
Search using the search field from the new tab page, search bar, URL bar, and about:home page.<br />
Restart Firefox and check the search engines installed<br />
Search
|-
|-
| Change Search Settings - deselect the default search engines, but do not remove it ||  
| Bugs management || Bugzilla
Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
|}


Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
= Status =
== Overview ==
<small>'''<font color="green">[LANDED]</font>'''</small> Nightly 45: first landed on - '''10.29.2015'''


Deselect the default search engines from the "One-click search engines" list. <br />
== Risk analysis ==
- No Risks found at the moment


Write something in the toolbar search field<br />
= References =
Meta: [https://bugzilla.mozilla.org/show_bug.cgi?id=1203167 Bug 1203167]


Search<br />
= Test cases =
== Overview ==
* Perform search from the Search Bar – using different terms
* Perform search from all the available search fields
* Add/Update search engine and validate search results is given by the new engine
* Change search settings (select/deselect engine, remove/restore engine, reorder engines) and validate search results is given by the new engine
* Verify if custom search engine settings (new engines added, reordered engines) are preserved after update
* Verify that after update, search engines settings can be updates
* Install external software's that automatically change the search engine for installed browsers and verify that the default search engine or users custom search engine is not altered


Restart firefox.<br />


Search again<br />
== Test Areas ==


{| class="wikitable" style="width:60%"
|-
|-
| Change Search Settings - remove and restore options || Launch Firefox and open a new tab (Ctrl/Cmd+T).
! Test Areas !! Covered !! Details
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
Select one search engine from the "One-click search engines" list, and click the "Remove" button<br />
Open the drop down menu from the Search Toolbar and open/refresh about:preferences#search<br />
Restart firefox.<br />
Open a new tab and a private window and open/refresh about:preferences#search<br />
Select  "Change Search Settings" and click on 'Restore default Search engines'
Restart firefox.
Open a new tab/private window and open the drop down from the search toolbar.
|-
|-
| Change Search Settings - remove and restore default engine || Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
| Private Window
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
|style="text-align:center;" | {{mdone|}} ||
Select the default search engine from the "One-click search engines" list, and click the "Remove" button<br />
Open the drop down menu from the Search Toolbar and refresh the about:preferences#search page<br />
Search.<br />
Restart firefox.<br />
Search again.<br />
Select again "Change Search Settings" and click on 'Restore default Search engines'<br />
Search.<br />
Set as default the engine previously restored.<br />
Restart firefox.<br />
Search.
|-
|-
| Change Search Settings - reorder search engines || Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
| Multi-Process Enabled
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
|style="text-align:center;" | {{mdone|}} ||
Select a search engine from the "One-click search engines" list, and  move it's position up in the list by dragging it<br />
Open the drop down menu from the Search fields in about:home, about:newtab and from the search box (toolbar)<br />
Search<br />
Restart firefox<br />
Go back to Search Preferences and reoder all search engines<br />
Restart firefox<br />
Search
|-
|-
| Change Search Settings - add aliases to search engine|| Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
| Multi-process Disabled
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
|style="text-align:center;" | {{mdone|}} ||
Select a search engine from the "One-click search engines" list, and for one engine add alias(duble click under the Keyword column and set an alias for one search engine(not the default one))<br />
Open a new tab and in the location bar write: alias name followed by the a search term.<br />
Search, without using the alias<br />
Restart firefox<br />
Go back to Search Preferences<br />
Open a new tab and in the location bar write: alias name followed by the a search term.<br />
Search, without using the alias<br />
|-
|-
| Change Search Settings - Add new search engine Youtube before search refactor || Install ff without fix (44.0a1 nightly version from 25 OCT). Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
| Display (HiDPI)  
Open the drop down from the Search box field by clicking on the magnifying glass icon and select "Change Search Settings"<br />
|style="text-align:center;" | N/A || 
Select "Add more search engines..." link<br />
Install an add-on from the provided list (for example Youtube) and then check it to be the default search engine, when asked<br />
Search <br />
Update ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
Search <br />
Add new search engine. Set it the default search engine.<br />
Search.
 
|-
|-
| Change Search Settings - Update default search engine || Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
| Interraction (scroll, zoom)  
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
|style="text-align:center;" | N/A ||
Change the default engine, for e.g. set eBay the default engine.<br />
Search using the search field from the new tab page, search bar, URL bar, and about:home page.<br />
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
Search
|-
|-
| Change Search Settings - deselect the default search engines, but do not remove it || Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
| Theme (high contrast)  
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
|style="text-align:center;" | N/A ||  
Deselect the default search engine from the "One-click search engines" list. <br />
"Write something in the toolbar search field and perform a Search"<br />
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
Search again<br />
 
|-
| Change Search Settings - remove and restore options || Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
Select one search engine from the "One-click search engines" list, and click the "Remove" button<br />
Open the drop down menu from the Default Search field (in about:preferences#search) and from the search box (toolbar)<br />
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
Open a new tab and a private window and repeat step 4.<br />
Select again "Change Search Settings" and click on 'Restore default Search engines'<br />
Restart firefox.<br />
Open a new tab and a private window and repeat step 4.
|-
| Change Search Settings - remove and restore default engine || Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
Select the default search engine from the "One-click search engines" list, and click the "Remove" button
Search.<br />
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
Search again.<br />
Select again "Change Search Settings" and click on 'Restore default Search engines'<br />
Search.
|-
| Change Search Settings - reorder search engines || Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
Select a search engine from the "One-click search engines" list, and move it's position up in the list by dragging it<br />
Open the drop down menu from the Search fields in about:home, about:newtab and from the search box (toolbar)
Search<br />
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).
Search<br />
Go back to Search Preferences and reoder all search engines<br />
Restart firefox<br />
Search
|-
| Change Search Settings - add aliases to search engine || Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".<br />
Select a search engine from the "One-click search engines" list, and from one engine add alias(duble click under the Keyword column and set an alias for one search engine(not the default one))
Open a new tab and on the location bar write: alias name followed by  a search term.<br />
Search, without using the alias<br />
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T). <br />
Go back to Search Preferences<br />
Open a new tab and on the location bar write: alias name followed by the a search term.<br />
Search, without using the alias
|-
| Downgrade to older version || Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).<br />
Open the drop down from the Search box field by clicking on the magnifying glass icon and select "Change Search Settings"<br />
Select "Add more search engines..." link<br />
Install an add-on from the provided list (for example Youtube) and then check it to be the default search engine, when asked<br />
Search <br />
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).
Search <br />
Downgrade to firefox version without the fix.
Search.<br />
|-
| Try to hijack mozilla search || Install different apps and try to hijack search engines.
|}
|}


Full Test Suites: [https://docs.google.com/spreadsheets/d/1jO0kwC2gQVvV66PY4MMIaYdAoTt4CKiDZLkJsuVDNEM/edit?usp=sharing]
== Test suites ==
Full Test Suite - can be reached at [https://docs.google.com/spreadsheets/d/1kRJjH1Itoi-3q7E5HZCCNx3jTYKDPq-31brFsA3jAA4/edit?usp=sharing link ]


= Bugs Work =
= Bugs Work =
Line 278: Line 190:
! Bug No !! Summary !! Status !! Firefox Verion
! Bug No !! Summary !! Status !! Firefox Verion
|-
|-
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1203161 1203161] ||remove the browser.search.cache.enabled pref ||Resolved - Fixed ||  
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1203167 1203167] ||Store user-installed search plugins in a JSON file and stop loading [profile]/searchplugins/*.xml ||Resolved - Fixed || Firefox 45
|-
|-
| || || ||  
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1224150 1224150] || Downgrading to FF 44 and upgrading back to FF 45 loses the search settings saved after the downgrade || RESOLVED WONTFIX ||  
|-
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1226608 1226608] || When restore a removed search engine, Google is set the default search engine || RESOLVED INVALID||
|-
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1227045 1227045] ||  The default search engine is not kept after selecting the "restore default search engines" || Verified FIXED|| Firefox 45
|}
|}
</div></div>
</div></div>
Line 289: Line 205:
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
[https://bugzilla.mozilla.org/show_bug.cgi?id=1224150 Bug 1224150] - Downgrading to FF 44 and upgrading back to FF 45 breaks the search engine<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1224150 Bug 1224150] - Downgrading to FF 44 and upgrading back to FF 45 breaks the search engine<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1218324 Bug 1218324] - Search engine "ask.com" couldn't be added to the existing list of search engines.<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1218368 Bug 1218368] - Search engines like Aol.com and Wow.com can't be added on the list of search engines from the Search Toolbar.<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1226608 Bug 1226608] - When restore a removed search engine, Google is set the default search engine <br />
[https://bugzilla.mozilla.org/show_bug.cgi?id=1227045 Bug 1227045] - The default search engine is not kept after selecting the "restore default search engines"<br />
[https://bugzilla.mozilla.org/show_bug.cgi?id=1228258 Bug 1228258] - One click search aliases should be left trimmed<br />
[https://bugzilla.mozilla.org/show_bug.cgi?id=1228263 Bug 1228263] - One click search alias save issue<br />[https://bugzilla.mozilla.org/show_bug.cgi?id=1229678 Bug 1229678] - Change search settings inaccessible under search engine list load<br />
[https://bugzilla.mozilla.org/show_bug.cgi?id=1228318 Bug 1228318] - Checklist for one click engine aren't restored when "Restore Default Engines" is used<br />
[https://bugzilla.mozilla.org/show_bug.cgi?id=1229685 Bug 1229685] - Aliasses function only in address bar<br />
[https://bugzilla.mozilla.org/show_bug.cgi?id=1228594 Bug 1228594] - Firefox Search toolbar is not restored when Yandex Extension is removed<br />
</div>
</div>
</div>
</div>
Line 300: Line 227:
== Results ==
== Results ==
'''Nightly testing'''
'''Nightly testing'''
* [In progress] Full Test  suite results [https://docs.google.com/spreadsheets/d/1xd31SYxmMfr5s-7C0AVS0v4oGDrxvripqgyIAK6Qae0/edit?usp=sharing ]<br />
* {{mdone}} Full Test  suite results [https://docs.google.com/spreadsheets/d/1xd31SYxmMfr5s-7C0AVS0v4oGDrxvripqgyIAK6Qae0/edit?usp=sharing ]<br />
** There were no blockers found while testing this feature.
** There were no blockers found while testing this feature.
** Manual testing covered Windows 7 x64, Windows 8 x64, Windows 10 x64, Windows XP x32, Windows Vista x32, Ubuntu 14.04 x32 and Mac OS X 10.
** Manual testing covered Windows 7 x64, Windows 8 x64, Windows 8.1 x32  Windows 10 x64, Windows XP x64, Windows Vista x32, Ubuntu 14.04 x32, Ubuntu 15.04 x 64, Mac OS x 10.10, Mac OS x 10.11.
<br />
<br />


'''Merge to Aurora Sign-off'''
{{mdone|}} '''Merge to Aurora Sign-off''' (2015-12-14)
 
'''Mid-Aurora Sign-off''' (2016-01-11)
* Full Test suite results [https://docs.google.com/spreadsheets/d/1sCNv3h5gVJkT9QKvs8lFVJ0Wpgta-SGsTH3ofyVE0Yw/edit#gid=1652877211 ]<br />
** Manual testing covered Windows 7 x64, Ubuntu 14.04 x32 and Mac OS X 10.9.5.
** There were no new issues found during sign-off.
 
'''Pre-Beta Sign-off''' (2016-01-22)
** Smoke testing covered Windows 7 x64, Ubuntu 14.04 x64 and Mac OS X 10.9.5.
 
'''Pre-Release Sign-off''' (2016-02-29)
** Smoke testing covered Windows 7 x64, Ubuntu 14.04 x64 and Mac OS X 10.9.5.
 
 
== Checklist ==
 
{| class="wikitable" style="width:60%"
|-
! Exit Criteria !! Status !! Notes/Details
|-
|  Testing Prerequisites (specs, use cases)
| style="text-align:center;" | {{mdone|}} 
| style="text-align:center;" | [https://bugzilla.mozilla.org/show_bug.cgi?id=1203167 Bug 1203167]
|-
|  Testing Infrastructure setup
|style="text-align:center;" |  {{mdone|}} ||
|-
|  Test Plan Creation
| style="text-align:center;" |  {{mdone|}} ||
|-
|  Test Cases Creation
|style="text-align:center;" |  {{mdone|}} ||
|-
|  Full Functional Tests Execution
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" | 
|-
|  Performance Testing
|style="text-align:center;" | N/A ||
|-
|  All Defects Logged
|style="text-align:center;"| {{mdone|}} ||
|-
|  Critical/Blockers Fixed and Verified
|style="text-align:center;"| {{mdone|}} ||
|-
|  Metrics/Telemetry
|style="text-align:center;" |  N/A  ||
|-
|  UAT Test Cases (signoff By Users)
|style="text-align:center;" |  N/A  ||
|-
|  QA Signoff  - Nightly Release
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | Email sent: [https://wiki.mozilla.org/images/4/43/Nighty_45_build_-_Search_Refactor_SignOFF_1.pdf Search SignOff email]
 
|-
|  QA Postmortem
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | [https://public.etherpad-mozilla.org/p/r.cb6c5a88838a1f3369153af4476ba62f Meeting Notes]
|}

Latest revision as of 13:47, 8 June 2016

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying Desktop Search engine works as expected after the refactoring.

Scope

The testing will be focus on

  • verify that search works correct
  • verify that customized user data/profile is not affected after update

Testing will be performed manually, by QA engineering team.

Ownership

Developer contacts: Florian Queze
QA:

Brindusa Tot - leading QA efforts

Simona Badau, QA

Ovidiu Boca, QA

Testing Summary

Scope of testing

In Scope

The main testing scope is to assure that the search works correct. For this fix, testing will include steps where users updates profile and we should verify that the user profile is not affected.

Requirements for testing

Environments

Testing will be performed on following OSes (both x86 & x64 infrastructures):

  • Mac OS X 10
  • Windows 7
  • Windows 8
  • Windows 8.1
  • Windows 10
  • Windows XP
  • Windows Vista
  • Ubuntu 14.04

Testing on specific devices is not needed.

Test Strategy

Test Objectives

Following table contains main features to be tested:

Ref Function Test Objective Evaluation Criteria
1 Search - basic scenarios Search using different settings and from different broswer locations Search is performed correctly.
2 Search - user profile verification after update Validate that after update to latest firefox user profile is not affected User profile is not updated
3 Search - hijack search with external apps Try to hijack firefox search engine Validate that search engine are not hijacked

Test Types

Type of Testing Required (Yes/No) Owners Details/Comments
Manual Yes Moz QA eng team
Automation Yes Florian Queze
Performance No
Security No
Usability No
Metrics/Telemetry TBD

Builds

This section should contain links for builds with the feature -

Nightly builds with the search refactor fix can be reached at link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 10.20.2015
Study documentation/specs received from developers 10.20.2015 -
QA - Test plan creation 10.23.2015
QA - Test cases/Env preparation 10.23.2015 10.30.2015
QA - Nightly Testing 10.30.2015 11.25.2015
QA - Aurora Testing 11.30.2015 01.25.2016
QA - Beta Testing 01.26.2016
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Google docs
Test case execution Google docs
Bugs management Bugzilla

Status

Overview

[LANDED] Nightly 45: first landed on - 10.29.2015

Risk analysis

- No Risks found at the moment

References

Meta: Bug 1203167

Test cases

Overview

  • Perform search from the Search Bar – using different terms
  • Perform search from all the available search fields
  • Add/Update search engine and validate search results is given by the new engine
  • Change search settings (select/deselect engine, remove/restore engine, reorder engines) and validate search results is given by the new engine
  • Verify if custom search engine settings (new engines added, reordered engines) are preserved after update
  • Verify that after update, search engines settings can be updates
  • Install external software's that automatically change the search engine for installed browsers and verify that the default search engine or users custom search engine is not altered


Test Areas

Test Areas Covered Details
Private Window [DONE]
Multi-Process Enabled [DONE]
Multi-process Disabled [DONE]
Display (HiDPI) N/A
Interraction (scroll, zoom) N/A
Theme (high contrast) N/A

Test suites

Full Test Suite - can be reached at link

Bugs Work

Tracking bug: Meta bug 1203167

Bug fix verification
Bug No Summary Status Firefox Verion
1203167 Store user-installed search plugins in a JSON file and stop loading [profile]/searchplugins/*.xml Resolved - Fixed Firefox 45
1224150 Downgrading to FF 44 and upgrading back to FF 45 loses the search settings saved after the downgrade RESOLVED WONTFIX
1226608 When restore a removed search engine, Google is set the default search engine RESOLVED INVALID
1227045 The default search engine is not kept after selecting the "restore default search engines" Verified FIXED Firefox 45
Logged bugs

Bug 1224150 - Downgrading to FF 44 and upgrading back to FF 45 breaks the search engine
Bug 1218324 - Search engine "ask.com" couldn't be added to the existing list of search engines.
Bug 1218368 - Search engines like Aol.com and Wow.com can't be added on the list of search engines from the Search Toolbar.
Bug 1226608 - When restore a removed search engine, Google is set the default search engine
Bug 1227045 - The default search engine is not kept after selecting the "restore default search engines"
Bug 1228258 - One click search aliases should be left trimmed
Bug 1228263 - One click search alias save issue
Bug 1229678 - Change search settings inaccessible under search engine list load

Bug 1228318 - Checklist for one click engine aren't restored when "Restore Default Engines" is used
Bug 1229685 - Aliasses function only in address bar
Bug 1228594 - Firefox Search toolbar is not restored when Yandex Extension is removed

Sign off

Criteria

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

  • [DONE] Full Test suite results [1]
    • There were no blockers found while testing this feature.
    • Manual testing covered Windows 7 x64, Windows 8 x64, Windows 8.1 x32 Windows 10 x64, Windows XP x64, Windows Vista x32, Ubuntu 14.04 x32, Ubuntu 15.04 x 64, Mac OS x 10.10, Mac OS x 10.11.


[DONE] Merge to Aurora Sign-off (2015-12-14)

Mid-Aurora Sign-off (2016-01-11)

  • Full Test suite results [2]
    • Manual testing covered Windows 7 x64, Ubuntu 14.04 x32 and Mac OS X 10.9.5.
    • There were no new issues found during sign-off.

Pre-Beta Sign-off (2016-01-22)

    • Smoke testing covered Windows 7 x64, Ubuntu 14.04 x64 and Mac OS X 10.9.5.

Pre-Release Sign-off (2016-02-29)

    • Smoke testing covered Windows 7 x64, Ubuntu 14.04 x64 and Mac OS X 10.9.5.


Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE] Bug 1203167
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Performance Testing N/A
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry N/A
UAT Test Cases (signoff By Users) N/A
QA Signoff - Nightly Release [DONE] Email sent: Search SignOff email
QA Postmortem [DONE] Meeting Notes
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Search_refactoring&oldid=1136419"