MozillaWiki

QA/Tab Group Migration

< QA

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
12/04/2015 1.0 Brindusa Tot Created first draft

Overview

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed for the Tab Group Migration project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Developer contacts: Gijs Kruitbosch

QA:
Rares Bologa - PM for QA team

Brindusa Tot - leading QA efforts

Adrian Florinescu, QA

Simona Badau, QA

Cosmin Muntean, QA

Testing summary

Scope of Testing

In Scope

Panorama (tab groups) will be discontinued, and all the tab groups will be saved into the Bookmarks folder. The main scope of testing is to assure that all tab groups (Panorama) are migrated correctly to Bookmarks into the latest Firefox build and Bookmarks works correctly.

Out of Scope

-

Requirements for testing

Environments

Testing will be performed on following OSes (both x86 & x64 infrastructures):

  • Bulleted list item
  • Mac OS X 10,11
  • Windows 7
  • Windows 8
  • Windows 8.1
  • Windows 10
  • Windows XP
  • Windows Vista
  • Ubuntu 14.04

Testing on specific devices is not needed.

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 Migrate Panorama with one tab Create one group with one tab on FF 44 and update to FF 45 Validate that tab from Panorama is saved into Bookmarks Manual Eng Team
2 Migrate Panorama with multiple tabs Create Panorama with multiple tabs on FF 44 and update to FF 45 Validate that tabs from Panorama are saved into Bookmarks Manual Eng Team
3 Migrate Panorama - multiple groups of one tab Create Panorama having multiple groups each with one tab on FF 44 and update to FF 45 Validate that tab from groups are saved into Bookmarks Manual Eng Team
4 Migrate Panorama - multiple groups with multiple tabs Create Panorama having multiple groups with multiple tabs on FF 44 and update to FF 45 Validate that tabs from Panorama groups are saved into Bookmarks Manual Eng Team
5 Smoke testing on bookmarks Bookmarks can be created and are accessible on FF 45 Validate bookmarks are created and accessible on FF 45 Manual/automation Eng Team/Dev Team
6 User notification about Panorama Disruption In a previous version of Firefox, users are informed that panorama will be removed Manual Eng Team
7 At update user is notified that Panorama is removed After update to new ff, users are informed that Panorama is removed and the tabs from panorama are saved into Bookmarks Manual Eng Team

Builds

This section should contain links for builds with the feature -

  • Nightly builds with the fix are available to the the link
  • Links for Aurora builds
  • Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 12/02/2015
Study documentation/specs received from developers 12/02/2015
QA - Test plan creation 12/03/2015
QA - Test cases/Env preparation 12/03/2015
QA - Nightly Testing 12/07/2015
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Google docs
Test case execution Google docs
Bugs management Bugzilla

Status

Overview

[LANDED] Nightly 45: first landed on - 12.01.2015
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

- Inform message, bar, that the Panorama will be removed Firefox is not present in the Nightly release line so cannot be tested, when update from Nightly 44 to Nightly 45. For this part of the fix there are two bugs: bug 1224635 and bug 1221500, first one is new and second Resolved fix. Not sure if this is fixed or not, yet.
- Lack of the documentation - for this kind of feature we should receive a Release Notes document, summing up at least a basic description of what is to be expected from the feature in that release.

References

  • List and links for specs - no documents with specs was received.

Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) No
UI
Mouse-only operation No
Keyboard-only operation No
Display (HiDPI) No
Interraction (scroll, zoom) No
Usable with a screen reader Maybe Should be tested with NVDA the notification/bar that inform users about the tab groups discontinuing
Usability and/or discoverability testing No Is this feature user friendly
Help/Support
Help/support interface required Yes Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Yes Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected Yes
Does sync work across upgrades
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing No yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration
Network proxies/autoconfig
ESR behavior changes
Locked preferences
Data Monitoring
Temporary or permanent telemetry monitoring No List of error conditions to monitor
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing NO
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites No
Survey of many sites for compatibility No
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test suite

Full Test suite - link

Bug Work

Tracking bug – meta bug

Bug fix verification
Bug No Summary Status Firefox Verion
Logged bugs

Bug 1231112 Tab Groups are not migrated from Safe mode
Bug 1231114 Tab Groups do not have correct names on the informing page: "Tab Groups are no more"
Bug 1231126 If the active group had a name, the bookmark tab group migration names it Group (null) instead of its name
Bug 1231135 Tab Groups migration page is not focused after the migration
Bug 1231405 Panorama data is not accessible when 44->45->44
Bug 1231629 Tab Groups migration is not done on a second update (44->45->44->45)
Bug 1232288 Windows Nightly updater (44->45) malfunction

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite - link
    • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage
Performance Testing
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Daily Status Report (email/etherpad statuses/ gdoc with results) [DONE]
Metrics/Telemetry N/A
QA Signoff - Nightly Release [DONE] SignOff email sent on 12.14.2015
QA Aurora - Full Testing
QA Signoff - Aurora Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Tab_Group_Migration&oldid=1135656"