MozillaWiki

QA/Search refactoring

< QA
Revision as of 15:02, 16 November 2015 by Brindusa.tot (talk | contribs) (→‎Sign off)

Overview

Summary

The purpose of this wiki is to serve as a general test plan for verifying Desktop Search engine works as expected after the refactoring.

The testing will be focus on:

  • verify that search works correct
  • verify that customized user data/profile is not affected after update


Ownership

Developer contacts: Florian Queze
QA:

Brindusa Tot - leading QA efforts

Simona Badau, QA

Ovidiu Boca, QA

Status

Overview

[LANDED] Nightly 45: first landed on - 10.29.2015

Risk analysis

- No Risks found at the moment

References

Meta: Bug 1203167

Requirements for testing

Environments

Testing will be performed on following OSes (both x86 & x64 infrastructures):

  • Mac OS X 10
  • Windows 7
  • Windows 8
  • Windows 10
  • Windows XP
  • Windows Vista
  • Ubuntu 14.04

Test Types

Type of Testing Required (Y/N) Owners Details/Comments
Manual Y Moz QA eng team
Automation Y
Performance N
Security N
Metrics/Telemetry TBD

Test cases

Overview

  • Perform search from the Search Bar – using different terms
  • Perform search from all the available search fields
  • Add/Update search engine and validate search results is given by the new engine
  • Change search settings (select/deselect engine, remove/restore engine, reorder engines) and validate search results is given by the new engine
  • Verify if custom search engine settings (new engines added, reordered engines) are preserved after update
  • Verify that after update, search engines settings can be updates
  • Install external software's that automatically change the search engine for installed browsers and verify that the default search engine or users custom search engine is not altered

Test suites

Notes: Testing will be performed with multi-process Nightly Enabled and Disabled

Browser Window: Test will be conducted in standard browser window and Private browser windows


Title Description
Perform a search from the Search Bar without using a search term Launch Firefox.

Click the Search Bar from the toolbar. Don't write anything and press [enter].

Perform a search from the Search Bar using a specific search term Enter a search term in the Search Bar.

Click the go icon (->)

Change the search provider using the Search Bar Select "Change search settings" when the search drop-down panel is opened.

Set a different provider as default search engine. Perform a search for a random keyword.

Perform searches from all the available search fields


Perform a search from the Location Bar.

Perform a search from the Search Bar.
Perform a search from the about:newtab page.
Perform a search from the about:home page.

Search toolbar Launch Firefox.

Click the Search bar text box.
Don't write anything and press Enter
Click the down arrow next to the magnifying glass icon
Select "Change Search Settings" and choose another search engine as default from the Default Search Engine drop-down. Perform searches from search toolbar, URL toolbar, about:newtab and about:home
Restart Firefox and check the default search engine

Change Search Settings - Add new search engine Youtube Launch Firefox and open a new tab (Ctrl/Cmd+T).

Open the drop down from the Search box field by clicking on the magnifying glass icon and select "Change Search Settings"
Select "Add more search engines..." link
Install an add-on from the provided list (for example Youtube) and then check it to be the default search engine, when asked
Search using the search field from the new tab page, search bar, URL bar, and about:home page.
Restart Firefox and check the search engines installed

Change Search Settings - Add new search engine - other Launch Firefox and open a new tab (Ctrl/Cmd+T).

Navigate to pages that could contain different search engines who can be added to Firefox. Select the magnifying glass from the search toolbar and add them to the search engines list
Add several search engines
Set one of the added engine as default.
Search using the search field from the new tab page, search bar, URL bar, and about:home page.
Restart Firefox and check the search engines installed

Change Search Settings - Update default search engine Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Update the default engine, for e.g. set eBay the default engine.
Search using the search field from the new tab page, search bar, URL bar, and about:home page.
Restart Firefox and check the search engines installed
Search

Change Search Settings - deselect the default search engines, but do not remove it

Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".

Deselect the default search engines from the "One-click search engines" list.

Write something in the toolbar search field

Search

Restart firefox.

Search again

Change Search Settings - remove and restore options Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Select one search engine from the "One-click search engines" list, and click the "Remove" button
Open the drop down menu from the Search Toolbar and open/refresh about:preferences#search
Restart firefox.
Open a new tab and a private window and open/refresh about:preferences#search
Select "Change Search Settings" and click on 'Restore default Search engines' Restart firefox. Open a new tab/private window and open the drop down from the search toolbar.

Change Search Settings - remove and restore default engine Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Select the default search engine from the "One-click search engines" list, and click the "Remove" button
Open the drop down menu from the Search Toolbar and refresh the about:preferences#search page
Search.
Restart firefox.
Search again.
Select again "Change Search Settings" and click on 'Restore default Search engines'
Search.
Set as default the engine previously restored.
Restart firefox.
Search.

Change Search Settings - reorder search engines Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Select a search engine from the "One-click search engines" list, and move it's position up in the list by dragging it
Open the drop down menu from the Search fields in about:home, about:newtab and from the search box (toolbar)
Search
Restart firefox
Go back to Search Preferences and reoder all search engines
Restart firefox
Search

Change Search Settings - add aliases to search engine Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Select a search engine from the "One-click search engines" list, and for one engine add alias(duble click under the Keyword column and set an alias for one search engine(not the default one))
Open a new tab and in the location bar write: alias name followed by the a search term.
Search, without using the alias
Restart firefox
Go back to Search Preferences
Open a new tab and in the location bar write: alias name followed by the a search term.
Search, without using the alias

Change Search Settings - Add new search engine Youtube before search refactor Install ff without fix (44.0a1 nightly version from 25 OCT). Launch Firefox and open a new tab (Ctrl/Cmd+T).

Open the drop down from the Search box field by clicking on the magnifying glass icon and select "Change Search Settings"
Select "Add more search engines..." link
Install an add-on from the provided list (for example Youtube) and then check it to be the default search engine, when asked
Search
Update ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).
Search
Add new search engine. Set it the default search engine.
Search.

Change Search Settings - Update default search engine Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Change the default engine, for e.g. set eBay the default engine.
Search using the search field from the new tab page, search bar, URL bar, and about:home page.
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).
Search

Change Search Settings - deselect the default search engines, but do not remove it Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Deselect the default search engine from the "One-click search engines" list.
"Write something in the toolbar search field and perform a Search"
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).
Search again

Change Search Settings - remove and restore options Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Select one search engine from the "One-click search engines" list, and click the "Remove" button
Open the drop down menu from the Default Search field (in about:preferences#search) and from the search box (toolbar)
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).
Open a new tab and a private window and repeat step 4.
Select again "Change Search Settings" and click on 'Restore default Search engines'
Restart firefox.
Open a new tab and a private window and repeat step 4.

Change Search Settings - remove and restore default engine Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Select the default search engine from the "One-click search engines" list, and click the "Remove" button Search.
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).
Search again.
Select again "Change Search Settings" and click on 'Restore default Search engines'
Search.

Change Search Settings - reorder search engines Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Select a search engine from the "One-click search engines" list, and move it's position up in the list by dragging it
Open the drop down menu from the Search fields in about:home, about:newtab and from the search box (toolbar) Search
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T). Search
Go back to Search Preferences and reoder all search engines
Restart firefox
Search

Change Search Settings - add aliases to search engine Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).

Click on the magnifying glass from the Search toolbox and choose "Change Search Settings".
Select a search engine from the "One-click search engines" list, and from one engine add alias(duble click under the Keyword column and set an alias for one search engine(not the default one)) Open a new tab and on the location bar write: alias name followed by a search term.
Search, without using the alias
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).
Go back to Search Preferences
Open a new tab and on the location bar write: alias name followed by the a search term.
Search, without using the alias

Downgrade to older version Install ff without fix. Launch Firefox and open a new tab (Ctrl/Cmd+T).

Open the drop down from the Search box field by clicking on the magnifying glass icon and select "Change Search Settings"
Select "Add more search engines..." link
Install an add-on from the provided list (for example Youtube) and then check it to be the default search engine, when asked
Search
Update to ff with fix. Launch Firefox and open a new tab (Ctrl/Cmd+T). Search
Downgrade to firefox version without the fix. Search.

Try to hijack mozilla search Install different apps and try to hijack search engines.

Bugs Work

Tracking bug: Meta bug 1203167

Bug fix verification
Bug No Summary Status Firefox Verion
1203161 remove the browser.search.cache.enabled pref Resolved - Fixed
Logged bugs

Bug 1224150 - Downgrading to FF 44 and upgrading back to FF 45 breaks the search engine
Bug 1218324 - Search engine "ask.com" couldn't be added to the existing list of search engines.
Bug 1218368 - Search engines like Aol.com and Wow.com can't be added on the list of search engines from the Search Toolbar.

Sign off

Criteria

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

  • [In progress] Full Test suite results [1]
    • There were no blockers found while testing this feature.
    • Manual testing covered Windows 7 x64, Windows 8 x64, Windows 10 x64, Windows XP x32, Windows Vista x32, Ubuntu 14.04 x32 and Mac OS X 10.


Merge to Aurora Sign-off

Checklist

Exit Criteria Status Notes/Details
Testing prerequisites (specs, use cases) Yes Bug 1203167
Testing Infrastructure setup Yes
Test Plan creation Yes
Test cases creation Yes
Full Functional Tests execution In progress
Automation coverage TBD
Performance Testing N/A
All defects logged
Critical/Blockers fixed and verified
Daily status report (email/etherpad statuses/ gdoc with results)
Metrics/Telemetry TBD
UAT Test Cases (signoff By Users) N/A
QA signoff on Nightly builds - email sent
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Search_refactoring&oldid=1105757"