QA/Tab Group Migration: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
 
(22 intermediate revisions by the same user not shown)
Line 41: Line 41:
[mailto:simona.marcu@softvision.ro Simona Badau], QA
[mailto:simona.marcu@softvision.ro Simona Badau], QA


[mailto:cosmin.muntean@softvision.ro Cosmin Muntean], QA
 
= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
Panorama (tab groups) will be discontinued, and all the tab groups will be copied into the Bookmarks folder. The main scope of testing is to assure that all tab groups(Panorama) are migrated correctly to Bookmarks into the latest Firefox build and Bookmarks works correctly.
Panorama (tab groups) will be discontinued, and all the tab groups will be saved into the Bookmarks folder. The main scope of testing is to assure that all tab groups (Panorama) are migrated correctly to Bookmarks into the latest Firefox build and Bookmarks works correctly.


=== Out of Scope ===
=== Out of Scope ===
  - will be detail later
  -


= Requirements for testing =
= Requirements for testing =
Line 75: Line 76:
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners  
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! Owners  
|-
|-
| 1 || Name of the feature or sub-function being tested || The objective the test is trying to demonstrate || The criteria that will be evaluated to demonstrate the test is successful || Manual/ Automation/ Regression/ Performance/ Usability/ Security/ Telemetry || Eng Team
| 1 || Migrate Panorama with one tab || Create one group with one tab on FF 44 and update to FF 45 || Validate that tab from Panorama is saved into Bookmarks || Manual || Eng Team
|-
| 2 || Migrate Panorama with multiple tabs || Create Panorama with multiple tabs on FF 44 and update to FF 45 || Validate that tabs from Panorama are saved into Bookmarks || Manual || Eng Team
|-
| 3 || Migrate Panorama - multiple groups of one tab || Create Panorama having multiple groups each with one tab on FF 44 and update to FF 45 || Validate that tab from groups are saved into Bookmarks || Manual || Eng Team
|-
| 4 || Migrate Panorama - multiple groups with multiple tabs || Create Panorama having multiple groups with multiple tabs on FF 44 and update to FF 45 || Validate that tabs from Panorama groups are saved into Bookmarks || Manual || Eng Team
|-
| 5 || Smoke testing on bookmarks || Bookmarks can be created and are accessible on FF 45 || Validate bookmarks are created and accessible on FF 45|| Manual/automation || Eng Team/Dev Team
|-
|-
| 2 || Repeat for each feature/sub-function || || || ||
| 6 || User notification about Panorama Disruption || In a previous version of Firefox, users are informed that panorama will be removed ||   || Manual|| Eng Team
|-
|-
| 3 || || || || ||
| 7 || At update user is notified that Panorama is removed || After update to new ff, users are informed that Panorama is removed and the tabs from panorama are saved into Bookmarks ||   || Manual|| Eng Team
|}
|}


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly builds
* Nightly builds with the fix are available to the the [http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ link]
* Links for Aurora builds
* Links for Aurora builds
* Links for Beta builds
* Links for Beta builds
Line 137: Line 145:
= Status =  
= Status =  
== Overview ==
== Overview ==
  Track the dates and build number where feature was released to Nightly
  <small>'''<font color="green">[LANDED]</font>'''</small> Nightly 45: first landed on - '''12.01.2015'''
  Track the dates and build number where feature was merged to Aurora
  Track the dates and build number where feature was merged to Aurora
  Track the dates and build number where feature was merged to Release/Beta
  Track the dates and build number where feature was merged to Release/Beta


== Risk analysis ==
== Risk analysis ==
  Identify the high-risk assumptions
  - Inform message, bar, that the Panorama will be removed Firefox is not present in the Nightly release line so cannot be tested, when update from Nightly 44 to Nightly 45. For this part of the fix there are two bugs: bug 1224635 and bug 1221500, first one is new and second Resolved fix. Not sure if this is fixed or not, yet.
  Identify existing bugs on the feature with high risk
  - Lack of the documentation - for this kind of feature we should receive a Release Notes document, summing up at least a basic description of what is to be expected from the feature in that release.
Identify if other areas are affected by the fix
 
= References =
= References =
* List and links for specs
* List and links for specs - no documents with specs was received.
  List and links for available specs - documents, user stories, specifications
 
* Meta bug
* Meta bug : bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1221050 1221050]
 
= Testcases =  
= Testcases =  
== Overview ==
== Overview ==
Line 161: Line 168:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 176: Line 183:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | Maybe || Should be tested with NVDA the notification/bar that inform users about the tab groups discontinuing
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | No || Is this feature user friendly


|-
|-
Line 198: Line 205:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | Yes || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | Yes || Make sure support documents are written and are correct.


|-
|-
Line 208: Line 215:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
Line 214: Line 221:
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | No || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
Line 220: Line 227:
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | No || yes/no options, add comment with details about who will lead testing


|-
|-
Line 243: Line 250:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | No || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ||
|style="text-align:center;" | NO ||


|-
|-
Line 262: Line 269:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 278: Line 285:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 291: Line 298:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" |  ||  
|style="text-align:center;" | No ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 301: Line 308:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/192BDmxxPirNZMxaH8a9bN4ak1m0sp3U1PXT-G-THpWk/edit?usp=sharing link]
  Full Test suite - [https://docs.google.com/spreadsheets/d/1LvFqRZufK8VtrgGcaEyMG9mTEsxZj3zqor2lanVi5I8/edit?usp=sharing link]
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/192BDmxxPirNZMxaH8a9bN4ak1m0sp3U1PXT-G-THpWk/edit?usp=sharing link]
Regression Test suite - Link with the gdoc - if available/needed.


= Bug Work =
= Bug Work =
Line 325: Line 330:
|-
|-
! Bug No !! Summary !! Status !! Firefox Verion
! Bug No !! Summary !! Status !! Firefox Verion
|-
|  123 || bug summary || NEW || Nighly 45
|-
|-
| || || ||  
| || || ||  
Line 336: Line 339:
====== Logged bugs ======
====== Logged bugs ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
Bug 111111<br/>
Bug 1231112 Tab Groups are not migrated from Safe mode<br />
Bug 211111<br/>
Bug 1231114 Tab Groups do not have correct names on the informing page: "Tab Groups are no more"<br />
Bug 1231126 If the active group had a name, the bookmark tab group migration names it Group (null) instead of its name<br />
Bug 1231135 Tab Groups migration page is not focused after the migration<br />
Bug 1231405 Panorama data is not accessible when 44->45->44<br />
Bug 1231629 Tab Groups migration is not done on a second update (44->45->44->45)<br />
Bug 1232288 Windows Nightly updater (44->45) malfunction<br />
</div>
</div>
</div>
</div>


= Sign off =
= Sign off =
Line 354: Line 361:
List of OSes that will be covered by testing<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite - [https://docs.google.com/spreadsheets/d/1LvFqRZufK8VtrgGcaEyMG9mTEsxZj3zqor2lanVi5I8/edit?usp=sharing link]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Regression Test suite, if needed/available
** Regression Test suite, if needed/available


Line 369: Line 375:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |   
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |  ||  
|style="text-align:center;" | {{mdone|}}   ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
Line 391: Line 397:
|style="text-align:center;" |  ||  
|style="text-align:center;" |  ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | {{mdone|}}||  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Daily Status Report (email/etherpad statuses/ gdoc with results) || ||  
|  Daily Status Report (email/etherpad statuses/ gdoc with results)  
|style="text-align:center;" | {{mdone|}}||  
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry
|style="text-align:center;" | N/A
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | [https://wiki.mozilla.org/images/1/17/Nighty_45_build_-_Tab_Groups_Migration_SignOFF.zip SignOff email] sent on 12.14.2015
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  

Latest revision as of 09:57, 3 June 2016

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
12/04/2015 1.0 Brindusa Tot Created first draft

Overview

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed for the Tab Group Migration project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Developer contacts: Gijs Kruitbosch

QA:
Rares Bologa - PM for QA team

Brindusa Tot - leading QA efforts

Adrian Florinescu, QA

Simona Badau, QA

Cosmin Muntean, QA

Testing summary

Scope of Testing

In Scope

Panorama (tab groups) will be discontinued, and all the tab groups will be saved into the Bookmarks folder. The main scope of testing is to assure that all tab groups (Panorama) are migrated correctly to Bookmarks into the latest Firefox build and Bookmarks works correctly.

Out of Scope

-

Requirements for testing

Environments

Testing will be performed on following OSes (both x86 & x64 infrastructures):

  • Bulleted list item
  • Mac OS X 10,11
  • Windows 7
  • Windows 8
  • Windows 8.1
  • Windows 10
  • Windows XP
  • Windows Vista
  • Ubuntu 14.04

Testing on specific devices is not needed.

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 Migrate Panorama with one tab Create one group with one tab on FF 44 and update to FF 45 Validate that tab from Panorama is saved into Bookmarks Manual Eng Team
2 Migrate Panorama with multiple tabs Create Panorama with multiple tabs on FF 44 and update to FF 45 Validate that tabs from Panorama are saved into Bookmarks Manual Eng Team
3 Migrate Panorama - multiple groups of one tab Create Panorama having multiple groups each with one tab on FF 44 and update to FF 45 Validate that tab from groups are saved into Bookmarks Manual Eng Team
4 Migrate Panorama - multiple groups with multiple tabs Create Panorama having multiple groups with multiple tabs on FF 44 and update to FF 45 Validate that tabs from Panorama groups are saved into Bookmarks Manual Eng Team
5 Smoke testing on bookmarks Bookmarks can be created and are accessible on FF 45 Validate bookmarks are created and accessible on FF 45 Manual/automation Eng Team/Dev Team
6 User notification about Panorama Disruption In a previous version of Firefox, users are informed that panorama will be removed Manual Eng Team
7 At update user is notified that Panorama is removed After update to new ff, users are informed that Panorama is removed and the tabs from panorama are saved into Bookmarks Manual Eng Team

Builds

This section should contain links for builds with the feature -

  • Nightly builds with the fix are available to the the link
  • Links for Aurora builds
  • Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 12/02/2015
Study documentation/specs received from developers 12/02/2015
QA - Test plan creation 12/03/2015
QA - Test cases/Env preparation 12/03/2015
QA - Nightly Testing 12/07/2015
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Google docs
Test case execution Google docs
Bugs management Bugzilla

Status

Overview

[LANDED] Nightly 45: first landed on - 12.01.2015
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

- Inform message, bar, that the Panorama will be removed Firefox is not present in the Nightly release line so cannot be tested, when update from Nightly 44 to Nightly 45. For this part of the fix there are two bugs: bug 1224635 and bug 1221500, first one is new and second Resolved fix. Not sure if this is fixed or not, yet.
- Lack of the documentation - for this kind of feature we should receive a Release Notes document, summing up at least a basic description of what is to be expected from the feature in that release.

References

  • List and links for specs - no documents with specs was received.

Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) No
UI
Mouse-only operation No
Keyboard-only operation No
Display (HiDPI) No
Interraction (scroll, zoom) No
Usable with a screen reader Maybe Should be tested with NVDA the notification/bar that inform users about the tab groups discontinuing
Usability and/or discoverability testing No Is this feature user friendly
Help/Support
Help/support interface required Yes Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Yes Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected Yes
Does sync work across upgrades
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing No yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration
Network proxies/autoconfig
ESR behavior changes
Locked preferences
Data Monitoring
Temporary or permanent telemetry monitoring No List of error conditions to monitor
Telemetry correctness testing No
Server integration testing No
Offline and server failure testing No
Load testing NO
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites No
Survey of many sites for compatibility No
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test suite

Full Test suite - link

Bug Work

Tracking bug – meta bug

Bug fix verification
Bug No Summary Status Firefox Verion
Logged bugs

Bug 1231112 Tab Groups are not migrated from Safe mode
Bug 1231114 Tab Groups do not have correct names on the informing page: "Tab Groups are no more"
Bug 1231126 If the active group had a name, the bookmark tab group migration names it Group (null) instead of its name
Bug 1231135 Tab Groups migration page is not focused after the migration
Bug 1231405 Panorama data is not accessible when 44->45->44
Bug 1231629 Tab Groups migration is not done on a second update (44->45->44->45)
Bug 1232288 Windows Nightly updater (44->45) malfunction

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite - link
    • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage
Performance Testing
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Daily Status Report (email/etherpad statuses/ gdoc with results) [DONE]
Metrics/Telemetry N/A
QA Signoff - Nightly Release [DONE] SignOff email sent on 12.14.2015
QA Aurora - Full Testing
QA Signoff - Aurora Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Tab_Group_Migration&oldid=1135656"