Features/Add-ons/Add-ons Default to Compatible/TestPlan

Silent Update Add-ons Default To Compatible

Feature	Lead engineer	Status	QA Lead	Status
Add-ons Default To Compatible	Blair McBride	Complete	Virgil Dicu	Signed Off

Summary

• As majority of add-ons work from one version to the next without the need for developer maintenance, Firefox will mark addons compatible with new versions of Firefox by default, except for the following:
  • All binary add-ons
  • All strict incompatible add-ons
  • All blacklisted add-ons

• The Feature is planned for Firefox 10.

The feature is enabled by default in Firefox 10. The feature can be disabled by setting the "extensions.strictCompatibility" preference to true in about:config.

Reference

• Feature wiki page - Link

• Bug Tracking:
  • Add-on Manager: Bug 692664
  • AMO: Bug 691834

Non-Goals

• Will not be tested:
  • Add-ons individual functionality after being enabled to default.

Fixed bugs verification

• Verified: bug 693897, bug 693899, bug 702868, bug 700201, bug 693901, bug 698653, bug 695977, bug 527141

Status per weeks

Status-week 11/7/2011-11/14/2011

Implementation work has started on Add-on manager side starting with the following bugs:

• Allow add-ons to opt-in to strict compatibility checking
• Add preference to globally (re)enable strict compatibility checks
• Support detecting binary components, and enable strict compatibility checking when found

TO DO list:

• create 75% basic functionality test cases
• run feature basic functionality test cases on all platforms (Windows XP, Windows 7, Mac OS 10.6, Ubuntu 11.10)
• run feature basic functionality test cases on Firefox 7, 8 and 9, while upgrading from each one to Nightly.
• create test cases for edge areas and different add-on states.

Status-week 11/14/2011-11/21/2011

Latest feature implementation:

• bug 695977
• bug 700201

DONE list:

• created 75% basic functionality test cases
• ran feature basic functionality test cases on all platforms (Windows XP, Windows 7, Mac OS 10.6, Ubuntu 11.10)
• ran feature basic functionality test cases on Firefox 7, 8 and 9, while upgrading from each one to Nightly.
• created test cases for edge areas and different add-on states.

TO DO list:

• create and run test cases for recently implemented bugs
• provide help for 18 November test day exploratory testing around Add-ons Default to compatible:Testday
• add feature test cases in Litmus

Status-week 11/21/2011-11/28/2011

DONE list:

• started running a weekly exploratory testing on Windows XP
• started adding feature test cases in Litmus
• create and ran test cases for recently implemented bugs
• started verifying fixed bugs-two bugs remaining to be verified: bug 695977 and bug 698653

TO DO list for next week:

• run the weekly exploratory on Windows XP
• finish off adding test cases in litmus
• upgrade with heavy add-on profile (more than 30)
• upgrade from lower versions than Firefox 7.0.1

Status-week 11/28/2011-12/5/2011

DONE list:

• ran the weekly exploratory on Windows XP
• ran basic test cases on all platforms: Windows XP, Windows 7 (x86, x86_64), Ubuntu 11.10 (x86, x86_64), Mac OS 10.6, 10.7, Windows Vista, Windows 2000
• ran other test scenarios (upgrading from lower versions of Firefox, heavy add-on profiles)

TO DO list for next week:

• verify recently fixed bugs
• run Litmus test cases on 4 platforms: Ubuntu 11.10, Mac OS 10.6, Windows XP, Windows 7 x86

Status-week 12/5/2011-12/12/2011

DONE:

• modified some test cases in Litmus-added indication to report Firefox issues due to add-ons marked as incompatible in Bugzilla (Detection and Mitigation Document)
• ran basic test cases on Windows XP and Ubuntu 11.10

TO DO:

• verify remaining fixed bugs
• run test cases on Mac OS 10.6 and Windows 7
• insist on dark matter add-ons through testing on Windows 7

Status-week 12/12/2011-12/19/2011

DONE:

• verified bug 527141
• ran exploratory testing across 4 platforms (Windows XP, 7, Mac OS 10.6, Ubuntu 11.10)Document

TO DO:

• to verify bugs 693906 and 705530
• run a weekly exploratory on Windows XP

Status-week 12/19/2011-12/26/2011

DONE:

• ran exploratory testing across 3 platforms (Windows XP, Mac OS 10.6, Ubuntu 11.10)Document

TO DO:

• run a weekly exploratory on Windows XP

Status-week 01/04/2012-01/11/2012

DONE:

• ran exploratory testing across 2 platforms (Windows XP, Ubuntu 11.10) Document
• test day preparation: created test scenarios based on Marcia's information-Link
• run the Litmus test cases (BFTs) for Firefox 10 Beta 4
• created test plan for community involvement:http://mzl.la/xrsNr6

TO DO:

• run the weekly exploratory on at least 3 platforms
• run the Litmus test cases(BFTs) for Firefox 10 Beta 4
• Friday testday involvement: https://etherpad.mozilla.org/testday-20120113

Status-week 01/11/2012-01/18/2012

DONE:

• Friday add-ons compatibile by default test day involvement: https://etherpad.mozilla.org/testday-20120113
  • verified add-ons provided by Jorge from the list: http://bit.ly/vZvg1b
  • this has been tested mostly on Linux during the testday with community help. Going to verify Windows platform more thoroughly in the next days (with Beta 5 too)
• Fixed bugs: logged bug https://bugzilla.mozilla.org/show_bug.cgi?id=718630
• Ran the weekly exploratory on 3 platforms (Windows XP, Windows 7, Linux)
• Kept track of https://bugzilla.mozilla.org/show_bug.cgi?id=711275
  • The update window now lists all add-ons that would stop working with Strict Compatibility enabled for the Firefox version to which the update is made (10 and 11 as well given the workaround fix for this)

TO DO:

• Finish off testing as many add-ons from the list http://bit.ly/vZvg1b on Windows and Mac OS 10.6 on Beta5. *Started this together with colleagues in Romania in the following etherpad: https://mozqa.etherpad.mozilla.org/ADT
• Run the weekly tests for this feature on Beta5
• Run an all platforms check before the Beta Release candidate next week.

Status-week 01/18/2012-01/25/2012

DONE:

• executed tests on popular add-ons from the Spreadsheet
• all platforms have been checked using the add-ons listed there.
• verified bugs: bug 693906 and bug 692664 (bug 705530 remaining-no examples yet)

Use Cases to Test

• An add-on can have different states in a Firefox version: compatible-incompatible, enabled/disabled, pending restart, updated/in need of update.
• User installs several types of add-ons on a Firefox version with one or more of the above mentioned states
• When upgrading to a new version, Firefox will automatically set his add-ons to compatible with the new Firefox version, unless the installed add-ons are binary, blacklisted or strictly set to incompatible with the Firefox version.

Edge cases

• Upgrade and verify that installed third party and restartless add-ons are default to compatible or incompatible if binary/blacklisted/strictCompatibility listed
• Upgrade and check behavior for add-ons installed on older Firefox versions (<Firefox 7)
• Upgrade from Firefox4 successively to Firefox 5,6,7,8,9 and 10 and verify that all add-ons are listed as expected.

Test Cases

Test cases based on this feature are listed in the following spreadsheet: Link

Signed-off for Aurora

Signed off for Aurora:

• No major or critical bugs found
• Feature's basic functionality test cases work as expected
• Add-on are enabled and working after update

Signed-off for Beta

Signed off for Beta:

• The feature has landed and is now set as resolved fix in tracking bug: Bug 692664
• Feature's basic functionality test cases work as expected
• Feature has been tested on 4 platforms: Ubuntu 11.10, Mac OS 10.6, Windows XP and Windows 7.

Sign-off Criteria for Release

• No unresolved major or critical bugs
• Feature's basic functionality test cases work as expected
• Ensure users can upgrade their add-ons successively from one Firefox version to another without any unnecessary change of states (e. g. from disabled to enabled)
• No other issues are triggered by this feature implementation