QA/Search refactoring

From MozillaWiki
Jump to navigation Jump to search

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying Desktop Search engine works as expected after the refactoring.

Scope

The testing will be focus on

  • verify that search works correct
  • verify that customized user data/profile is not affected after update

Testing will be performed manually, by QA engineering team.

Ownership

Developer contacts: Florian Queze
QA:

Brindusa Tot - leading QA efforts

Simona Badau, QA

Ovidiu Boca, QA

Testing Summary

Scope of testing

In Scope

The main testing scope is to assure that the search works correct. For this fix, testing will include steps where users updates profile and we should verify that the user profile is not affected.

Requirements for testing

Environments

Testing will be performed on following OSes (both x86 & x64 infrastructures):

  • Mac OS X 10
  • Windows 7
  • Windows 8
  • Windows 8.1
  • Windows 10
  • Windows XP
  • Windows Vista
  • Ubuntu 14.04

Testing on specific devices is not needed.

Test Strategy

Test Objectives

Following table contains main features to be tested:

Ref Function Test Objective Evaluation Criteria
1 Search - basic scenarios Search using different settings and from different broswer locations Search is performed correctly.
2 Search - user profile verification after update Validate that after update to latest firefox user profile is not affected User profile is not updated
3 Search - hijack search with external apps Try to hijack firefox search engine Validate that search engine are not hijacked

Test Types

Type of Testing Required (Yes/No) Owners Details/Comments
Manual Yes Moz QA eng team
Automation Yes Florian Queze
Performance No
Security No
Usability No
Metrics/Telemetry TBD

Builds

This section should contain links for builds with the feature -

Nightly builds with the search refactor fix can be reached at link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 10.20.2015
Study documentation/specs received from developers 10.20.2015 -
QA - Test plan creation 10.23.2015
QA - Test cases/Env preparation 10.23.2015 10.30.2015
QA - Nightly Testing 10.30.2015 11.25.2015
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Google docs
Test case execution Google docs
Bugs management Bugzilla

Status

Overview

[LANDED] Nightly 45: first landed on - 10.29.2015

Risk analysis

- No Risks found at the moment

References

Meta: Bug 1203167

Test cases

Overview

  • Perform search from the Search Bar – using different terms
  • Perform search from all the available search fields
  • Add/Update search engine and validate search results is given by the new engine
  • Change search settings (select/deselect engine, remove/restore engine, reorder engines) and validate search results is given by the new engine
  • Verify if custom search engine settings (new engines added, reordered engines) are preserved after update
  • Verify that after update, search engines settings can be updates
  • Install external software's that automatically change the search engine for installed browsers and verify that the default search engine or users custom search engine is not altered


Test Areas

Test Areas Covered Details
Private Window [DONE]
Multi-Process Enabled [DONE]
Multi-process Disabled [DONE]
Display (HiDPI) N/A
Interraction (scroll, zoom) N/A
Theme (high contrast) N/A

Test suites

Full Test Suite - can be reached at link

Bugs Work

Tracking bug: Meta bug 1203167

Bug fix verification
Bug No Summary Status Firefox Verion
1203167 Store user-installed search plugins in a JSON file and stop loading [profile]/searchplugins/*.xml Resolved - Fixed Firefox 45
1224150 Downgrading to FF 44 and upgrading back to FF 45 loses the search settings saved after the downgrade RESOLVED WONTFIX
1226608 When restore a removed search engine, Google is set the default search engine RESOLVED INVALID
1227045 The default search engine is not kept after selecting the "restore default search engines" Verified FIXED Firefox 45
Logged bugs

Bug 1224150 - Downgrading to FF 44 and upgrading back to FF 45 breaks the search engine
Bug 1218324 - Search engine "ask.com" couldn't be added to the existing list of search engines.
Bug 1218368 - Search engines like Aol.com and Wow.com can't be added on the list of search engines from the Search Toolbar.
Bug 1226608 - When restore a removed search engine, Google is set the default search engine
Bug 1227045 - The default search engine is not kept after selecting the "restore default search engines"

Sign off

Criteria

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

  • [In progress] Full Test suite results [1]
    • There were no blockers found while testing this feature.
    • Manual testing covered Windows 7 x64, Windows 8 x64, Windows 8.1 x32 Windows 10 x64, Windows XP x64, Windows Vista x32, Ubuntu 14.04 x32, Ubuntu 15.04 x 64, Mac OS x 10.10, Mac OS x 10.11.


Merge to Aurora Sign-off

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE] Bug 1203167
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [IN PROGRESS]
Automation Coverage TBD
Performance Testing N/A
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified
Daily Status Report (email/etherpad statuses/ gdoc with results) [DONE]
Metrics/Telemetry TBD
UAT Test Cases (signoff By Users) N/A
QA Signoff - Nightly Release Email to be sent
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Search_refactoring&oldid=1108983"