Add-ons/QA/Testplan/Webextensions Language Packs: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(18 intermediate revisions by 2 users not shown)
Line 6: Line 6:
|-
|-
| 09/14/2017 || 1.0 || Valentina Virlics || Created first draft
| 09/14/2017 || 1.0 || Valentina Virlics || Created first draft
|-
| 10/02/2017 || 1.1 || Marius Santa || Updateds: Overview, Purpose, Scope, Bug Work, Builds, Checklist, Channel
|}
|}


= Overview =
= Overview =
* A language pack is an extension (add-on) that changes the language of the user interface in a Mozilla application (Firefox, Thunderbird, SeaMonkey, etc.).  
* A language pack is an add-on that changes the language of the user interface in a Mozilla application (Firefox, Thunderbird, SeaMonkey, etc.).  
*
* Switching to the new language pack formats will allow us to remove language packs from requiring the old add-ons ecosystem, they will be safer(no direct access to chrome registry) and will enable L10nRegistry ([https://bugzilla.mozilla.org/show_bug.cgi?id=1333980 bug 1333980]) which is required for transitioning to the new l10n API ([https://bugzilla.mozilla.org/show_bug.cgi?id=1365426 bug 1365426]).


== Purpose ==
== Purpose ==
* Introducing Language Packs that use WebExtensions packaging.
* This document's purpose is to detail the test approach to the Language Packs Webextensions packaging, including Entry/Exit criteria, Scope for testing, links to testcases etc
*


=== Entry Criteria ===
=== Entry Criteria ===
* QA has access to PRD and some mocks (found in bugs)
* QA has access to PRDs and mocks (from bugs)
* The feature has landed in [https://addons-dev.allizom.org/en-US/firefox/ -dev]
* The feature has landed in [https://addons-dev.allizom.org/en-US/firefox/ -dev]


Line 30: Line 31:


== Scope ==
== Scope ==
This section describes what parts of the feature will be tested and what parts won't be.
===what's in scope?===
===what's in scope?===
* Lang Packs validation - recognize the special kind of extension based on some entry in manifest.json
* Lang Packs submission
* Lang Packs submission
* Lang Packs auto-approval
* Lang Packs auto-approval
* Lang Packs structure and validation
* Lang Packs structure and validation
* Lang Packs installation
* Lang Packs installation
*


===what's out of scope?===
===what's out of scope?===
Line 45: Line 48:
QA Lead: [mailto:vcarciu@softvision.ro Victor Carciu]; irc nick :victorc<br />
QA Lead: [mailto:vcarciu@softvision.ro Victor Carciu]; irc nick :victorc<br />
Add-ons QA: [mailto:valentina.peleskei@softvision.ro Valentina Virlics]; irc nick :ValentinaV<br />
Add-ons QA: [mailto:valentina.peleskei@softvision.ro Valentina Virlics]; irc nick :ValentinaV<br />
Webextensions QA: [mailto:cosmin.badescu@softvision.ro Cosmin Badescu]; irc nick :CosminB<br />
Webextensions QA: [mailto:marius.santa@softvision.ro Marius Santa]; irc nick :Santa<br />


= Requirements for testing =
= Requirements for testing =
Line 50: Line 55:
* Windows
* Windows
* Mac OS
* Mac OS
* Linux


== Servers ==
== Servers ==
Line 56: Line 62:
* Production: https://addons.mozilla.org/en-US/  
* Production: https://addons.mozilla.org/en-US/  


== Channels ==
== Channel dependent settings (configs) and environment setups ==
* Nightly
* Beta
* Release
* Release


= Test Strategy =  
= Test Strategy =  
== Builds ==
This section should contain links for builds with the feature -
* [http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Nightly builds]
* [https://archive.mozilla.org/pub/firefox/candidates/ Beta builds]
* Beta Unbranded builds:
** [https://archive.mozilla.org/pub/devedition/candidates/ All Builds]
* [https://archive.mozilla.org/pub/firefox/releases/ Release builds]
* Release Unbranded builds:
** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-win64-add-on-devel/ Windows Link]
** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-linux64-add-on-devel/ Linux Link]
** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-macosx64-add-on-devel/ Mac Link]
== Test Execution Schedule ==
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
The following table identifies the anticipated testing period available for test execution.
Line 92: Line 112:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || TestRail / Google docs  
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1703&group_by=cases:section_id&group_order=asc TestRail] / [https://docs.google.com/spreadsheets/d/1r64jOLIjDXi9Tuv30j7MTWS6ujr0MhQ8TCQjTRgAWgc/edit#gid=1 Google docs]
|-
|-
| Test case execution || TestRail
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/runs/view/7005&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Bugs management || Github
| Bugs management || Github
Line 107: Line 127:
* Submission/Approvals/Installation of Webextensions Lang Packs
* Submission/Approvals/Installation of Webextensions Lang Packs


== Test suite ==
== Test Areas ==
  Full Test suite
{| class="wikitable" style="width:80%"
|-
! Test Areas !! Covered !! Details
|-
| '''Installation from AMO'''
|style="text-align:center;" | ||
|-
| '''Installation from Add-ons Manager'''
|style="text-align:center;" | ||
|-
| '''Installation from Third-party'''
|style="text-align:center;" |  ||
|-
| '''Language Packs updates'''
||  ||
|-
| '''Sideloading''' 
|style="text-align:center;" |  ||
|-
| '''Other''' 
|style="text-align:center;" |  ||
|-
|}


= Bug Work =
= Bug Work =
Line 116: Line 158:
  - [https://github.com/mozilla/addons-server/issues/6350 6350]   
  - [https://github.com/mozilla/addons-server/issues/6350 6350]   
  - [https://github.com/mozilla/addons-server/issues/6360 6360]   
  - [https://github.com/mozilla/addons-server/issues/6360 6360]   
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bug fix verification ======
<div class="mw-collapsible-content">
* 
</div></div>


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
* {{Bug|1395363}} - Switch to webextension-langpacks
<bugzilla>
    {
        "blocks":[1395363],
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>  


====== Logged bugs ======
* {{Bug|1365031}} - Turn langpacks into webextension-like packages
<div class="mw-collapsible-content">
<bugzilla>
* 
    {
        "blocks":[1365031],
</div></div>
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>  


= Sign off =
= Sign off =
Line 139: Line 180:
* All test cases should be executed
* All test cases should be executed
* All blockers must be fixed and verified or have an agreed-upon timeline for being fixed
* All blockers must be fixed and verified or have an agreed-upon timeline for being fixed
== Checklist ==
{| class="wikitable" style="width:60%"
|-
! Exit Criteria !! Status !! Notes/Details
|-
|  Testing Prerequisites (specs, use cases)
| style="text-align:center;" | 
| style="text-align:center;" |
|-
|  Testing Infrastructure setup
|style="text-align:center;" |  ||
|-
|  Test Plan Creation
| style="text-align:center;" |  ||
|-
|  Test Cases Creation
|style="text-align:center;" |  ||
|-
|  Full Functional Tests Execution
|style="text-align:center;" | 
|style="text-align:center;" | 
|-
|  Automation Coverage ||
|style="text-align:center;" |
|-
|  Performance Testing
|style="text-align:center;" |  ||
|-
|  All Defects Logged || ||
|-
|  Critical/Blockers Fixed and Verified || ||
|-
|  Metrics/Telemetry||
|style="text-align:center;" |
|-
|  QA Signoff  - Nightly Release||
|style="text-align:center;" |
|-
|  QA Beta - Full Testing
|style="text-align:center;" |  ||
|-
|  QA Signoff  - Beta Release ||
|style="text-align:center;" |
|}

Latest revision as of 16:20, 6 November 2017

Revision History

Date Version Author Description
09/14/2017 1.0 Valentina Virlics Created first draft
10/02/2017 1.1 Marius Santa Updateds: Overview, Purpose, Scope, Bug Work, Builds, Checklist, Channel

Overview

  • A language pack is an add-on that changes the language of the user interface in a Mozilla application (Firefox, Thunderbird, SeaMonkey, etc.).
  • Switching to the new language pack formats will allow us to remove language packs from requiring the old add-ons ecosystem, they will be safer(no direct access to chrome registry) and will enable L10nRegistry (bug 1333980) which is required for transitioning to the new l10n API (bug 1365426).

Purpose

  • This document's purpose is to detail the test approach to the Language Packs Webextensions packaging, including Entry/Exit criteria, Scope for testing, links to testcases etc

Entry Criteria

  • QA has access to PRDs and mocks (from bugs)
  • The feature has landed in -dev

Current Status

  • The feature is in -dev

Exit Criteria

  • All related bugs triaged
  • All blockers fixed
  • All resolved bugs verified by QA
  • Found-fixed bugs rate going down in time

Scope

This section describes what parts of the feature will be tested and what parts won't be.

what's in scope?

  • Lang Packs validation - recognize the special kind of extension based on some entry in manifest.json
  • Lang Packs submission
  • Lang Packs auto-approval
  • Lang Packs structure and validation
  • Lang Packs installation

what's out of scope?

  • Extensions functionality

Ownership

Product Manager: Jorge Villalobos; irc nick :jorgev
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Add-ons QA: Valentina Virlics; irc nick :ValentinaV
Webextensions QA: Cosmin Badescu; irc nick :CosminB
Webextensions QA: Marius Santa; irc nick :Santa

Requirements for testing

Environments

  • Windows
  • Mac OS
  • Linux

Servers

Channel dependent settings (configs) and environment setups

  • Nightly
  • Beta
  • Release

Test Strategy

Builds

This section should contain links for builds with the feature -

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 17.05.2017
Study PRD/mocks received
QA - Test plan creation 14.09.2017
QA - Test cases preparation
QA - Test cases execution
Release Date

Testing Tools

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail / Google docs
Test case execution TestRail
Bugs management Github

References

* Webextenstions Language Packs accepted structure  
* Tracker: 6297

Testcases

Test Areas

  • Submission/Approvals/Installation of Webextensions Lang Packs

Test Areas

Test Areas Covered Details
Installation from AMO
Installation from Add-ons Manager
Installation from Third-party
Language Packs updates
Sideloading
Other

Bug Work

  • Feature bugs
- 6297 (traker)  
- 1526  
- 6350  
- 6360   

* bug 1395363 - Switch to webextension-langpacks
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1365709 -- Add-ons Manager Zibi Braniecki [:zbraniecki][:gandalf] Consume new webextension-based langpacks RESOLVED mozilla57
1395456 P3 Internationalization Zibi Braniecki [:zbraniecki][:gandalf] Ensure we generate correct min/max versions for the new langpacks RESOLVED mozilla57
1395457 P3 Internationalization Zibi Braniecki [:zbraniecki][:gandalf] De-hardcode product name when producing webext-langpack RESOLVED mozilla57
1395459 P3 Internationalization Zibi Braniecki [:zbraniecki][:gandalf] When building webext-langpack pick up language version from the VCS if possible RESOLVED mozilla58
1396467 P3 Add-ons Manager Zibi Braniecki [:zbraniecki][:gandalf] Add tests for the new langpacks consumption code RESOLVED mozilla58
1405264 P2 General Andrew Swan [:aswan] webext-langpacks register chrome entries later than old-style langpacks VERIFIED mozilla58
1407395 -- Internationalization Zibi Braniecki [:zbraniecki][:gandalf] Guard against empty defines entry when generating langpack manifest RESOLVED mozilla58
1408380 -- Internationalization Axel Hecht [:Pike] localized nightlies broken after switch to webextensions if locale is missing toolkit/defines.inc RESOLVED mozilla58
1411951 P1 Internationalization Zibi Braniecki [:zbraniecki][:gandalf] Windows langpacks are generated with invalid paths VERIFIED mozilla58
1415117 P5 Internationalization SessionRestore does not update tab title if language changes on restart NEW ---

10 Total; 1 Open (10%); 7 Resolved (70%); 2 Verified (20%); 


* bug 1365031 - Turn langpacks into webextension-like packages
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1333980 -- Internationalization Zibi Braniecki [:zbraniecki][:gandalf] Land L10nRegistry RESOLVED mozilla57
1365440 -- General Zibi Braniecki [:zbraniecki][:gandalf] Generate webextension language packs VERIFIED mozilla57
1365709 -- Add-ons Manager Zibi Braniecki [:zbraniecki][:gandalf] Consume new webextension-based langpacks RESOLVED mozilla57
1377911 -- General Zibi Braniecki [:zbraniecki][:gandalf] Remove the override chrome entries from language manifests RESOLVED mozilla57
1385544 P3 Internationalization Zibi Braniecki [:zbraniecki][:gandalf] Add mozl10n: protocol RESOLVED ---
1389397 -- Android Studio and Gradle Integration Zibi Braniecki [:zbraniecki][:gandalf] Remove the override chrome entries from language manifests (Fennec) RESOLVED mozilla57
1393848 -- General Zibi Braniecki [:zbraniecki][:gandalf] Move search-plugin resource entry to browser's content jar.mn RESOLVED mozilla57
1411951 P1 Internationalization Zibi Braniecki [:zbraniecki][:gandalf] Windows langpacks are generated with invalid paths VERIFIED mozilla58
1413322 P3 Add-ons Manager Andrew Swan [:aswan] Stop loading old-style langpacks VERIFIED mozilla58
1425689 -- Android Studio and Gradle Integration Zibi Braniecki [:zbraniecki][:gandalf] Move search-plugin resource entry to fennec's chrome jar.mn RESOLVED mozilla59

10 Total; 0 Open (0%); 7 Resolved (70%); 3 Verified (30%);


Sign off

Criteria

Check list

  • All test cases should be executed
  • All blockers must be fixed and verified or have an agreed-upon timeline for being fixed

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release
QA Beta - Full Testing
QA Signoff - Beta Release
Retrieved from "https://wiki.mozilla.org/index.php?title=Add-ons/QA/Testplan/Webextensions_Language_Packs&oldid=1183524"