Releases/Firefox 10/Test Plan

From MozillaWiki
Jump to navigation Jump to search

Firefox 10 Test Plan

Summary

Milestone Firefox 10
Status Currently in Aurora -- merging to Beta on December 20th, 2011
Lead Juan Becerra (irc: juanb)
2nd Anthony Hughes (irc: ashughes)

Latest Build: [ ftp], [ http], [ web]

The following is the test plan for Firefox 10 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 10. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

Daily Tasks
  • [ON TRACK] [softvision] Automation: Check dashboards for new failures
  • [ON TRACK] [softvision] Bug Triage: Check Bugzilla for regressions
  • [ON TRACK] [softvision] Features: Test features and triage feature bugs targeted for this releases
  • [ON TRACK] [softvision] Input: Triage Input.MO for prevalent issues and feature requests
One-time-only Tasks

Automation

BFTs

Triage the Mozmill dashboard for test failures and debug for potential Firefox regressions.

  • Lead: Vlad Maniac (Softvision)
  • Peer: Alex Lakatos (Softvision)
  • Failure Status: NEW (12) | ASSIGNED (7) | FIXED (3) | REGRESSIONS (0)
  • Tasks:
    • Check the Mozmill dashboard for failing tests for the day
    • Cross reference any failure to the list of NEW and ASSIGNED bugs
      • If the bug is NEW, assign it to yourself and try to fix it
      • If the bug is ASSIGNED and there has been no activity on the bug in over a week, offer your assistance to fix the bug
      • If there is NO bug on file, file it using this template
    • If you need assistance, ask ashughes in the #mozmill channel


Endurance

Performed during NIGHTLY phase only

Triage and report the performance numbers from the daily Endurance tests.

  • Lead: Mihaela Velimiroviciu, irc:mihaelav (Softvision)
  • Endurance reports
  • Spreadsheet for tracking results
  • NOTE: Report any spikes in performance degradation to the QA release driver
Issues Found

Bug Triage

Check UNCONFIRMED, NEW, FIXED, REOPEN, and tracking/status bugs

Notes

Features

Track features which have landed in Firefox 10 and are ready to be tested (spreadsheet)

Feature

Description of feature

  • Leads: Person (role)
  • QA Owner: Owner
  • Status: Status
  • Links: Feature page, test plan, tracking bug, etc
  • Notes:

Silent Update: Background updates

We want to move the bulk of the update process to the current session so that when the user restarts, there's not much left to do to complete the update and we can make it appear as if not interruption actually happened. This will only apply to Mac and Linux. For Windows, the service will take care of this.

  • Leads: Chris Lee (Product), Lawrence Mandel (Directly Responsible Individual), Ehsan Akhgari (Lead Engineer)
  • QA Owner: Vlad Ghetiu
  • Status: Development in progress
  • Links: Feature Page, Tracking Bug
  • Notes:
  • Sign-off status:
    • Aurora: not implemented yet in Firefox 10 - 11.11.2011
    • Beta:
    • Release:

CSS Rules View

The Style Inspector provides a new view of CSS that is a hybrid between a computed styles view and the traditional rules-oriented view. While this view is useful and will become more useful over time, there are use cases for which the traditional rules-based view is better.

  • Leads: Kevin Dangoor (Product), Dave Camp (Lead Enginner)
  • QA Owner:
  • Status: Landed. Complete
  • Links: Feature Page, Tracking Bug
  • Notes:
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Test Suite Fixes for WebGL

Need to fix issues found by the WebGL test suite.

  • Leads: Chris Blizzard (Product), JPR (Directly Responsible Individual)
  • QA Owner:
  • Status: In progress - feature in development stage
  • Links: Feature Page
  • Notes:
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

DOM Bindings: Node List and Array Bindings

This is the next phase of the DOM bindings work for performance. Changing how we handle node list changes and array bindings.

  • Leads: Chris Blizzard (Product), Johnny Stenback (Directly Responsible Individual)
  • QA Owner:
  • Status: Landed.Complete
  • Links: Feature Page
  • Notes:
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Anti-Aliasing for WebGL

Adds support for anti-aliasing to our WebGL implementation.

  • Leads: Chris Blizzard (Product), Johnny Stenback (Directly Responsible Individual)
  • QA Owner:
  • Status: Development
  • Links: Feature Page, |Tracking Bug
  • Notes:
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Firefox Home Tab

Firefox Home Tab is a small pinned tab placed at the far left of the tab strip that will serve as the user's home on the Web. The plan is for this interface to be locally hosted, customizable, and personalized to the needs of each individual user.

  • Leads: Chris Lee (Product), Erin Lancaster (Directly Responsible individual), Margaret Leibovic (Lead Engineer), Alex Faaborg (UX Lead)
  • QA Owner: Vlad Ghetiu (vladg)
  • Status: In progress. Health: OK - feature in development stage
  • Links: Feature Page, Tracking Bug
  • Notes: Feature did not land in Firefox 10. Feature was moved to another target release: Firefox 11
  • Sign-off status:
    • Aurora: Not implemented for Firefox 10 Aurora-11.11.2011
    • Beta:
    • Release:

Add-on hotfix

There are many issues that affect our audience after release that shouldn't require a full Firefox update. Firefox will have a hardcoded add-on id. When checking for add-on updates (every 24 hours), it will always check for a specific add-on id and install it if an update is found.

  • Leads: Christian Legnitto (Directly Responsible Individual), Dave Townsend (Lead Engineer)
  • QA Owner: Vlad Ghetiu (vladg)
  • Status: In progress. Health: OK - feature in development stage
  • Links: Feature Page, Tracking Bug
  • Notes:
  • Sign-off status:
    • Aurora: Not implemented yet for Firefox 10 Aurora-11.11.2011
    • Beta:
    • Release:

New Tab Page

Whenever Firefox users open a new tab, their goal is to use it to navigate somewhere. Firefox currently displays a blank page when Firefox users open a new page. This is guaranteed to not help them perform their next task.
We'd like to implement a page which displays on new tabs and helps users complete their next task. This will likely include offering options that are most likely to include the user's next task so that valuable time and steps are saved.

  • Leads: Asa Dotzler (Product), Jennifer Boriss (Feature, UX Lead), Margaret Leibovic (Lead Engineer)
  • QA Owner: Virgil Dicu, irc: VirgilDicu (SoftVision)
  • Status: Working in progress. Health: OK - Feature in development stage.
  • Links: Feature Page
  • Notes: Release target-Firefox 11
  • Sign-off status:
    • Aurora: Not implemented yet for Firefox 10 Aurora-11.11.2011
    • Beta:
    • Release:

Chrome Migration

Firefox offers new users migration of settings and data from Internet Explorer, Safari, and Opera. It does not offer users migration from Chrome. It should.
The goal is to make Firefox a seamless conversion from Chrome.
This feature falls primarily in the Experience category (from the "Discover, Experience, and Connect" vision statement.)

Code Editor

New developer tools features will provide opportunities for users to work directly with their code from within the browser. We would like for our tools to provide an editing interface that is more suited to editing code than a plain textarea or an HTML WYSIWYG interface.

  • Leads: Kevin Dangoor (Product), Kevin Dangoor (Feature), Mihai Sucan (Lead Engineer)
  • QA Owner: Alex Lakatos
  • Status: Landed, Complete.
  • Links: Feature Page
  • Notes:
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

CSSEditor (Style Editor)

The Style Editor allows users to make changes to their CSS and see the changes immediately. This is likely one of the most common things that web developers use these tools for: interactively iterating on designs.

  • Leads: Kevin Dangoor (Product, Directly Responsible individual), Cedric Vivier (Lead Engineer)
  • QA Owner: Alex Lakatos, irc:AlexLakatos (SoftVision)
  • Status: - feature in development stage
  • Links: Feature Page
  • Notes: Main patch missed Firefox 10. Feature is targeted for Firefox 11.
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

CSS3 3D Transforms

CSS 3D Transforms extends CSS Transforms to allow elements rendered by CSS to be transformed in three-dimensional space.

  • Leads: Martin Best (Directly Responsible individual), Matt Woodrow (Lead Engineer)
  • QA Owner: Ioana Budnar, irc:Ioana (SoftVision)
  • Status: Complete. Health: OK - feature landed
  • Links: Feature Page, Tracking Bug, Test Plan
  • Notes: most bugs listed in the test plan still need to be fixed.
  • Sign-off status:
    • Aurora: SIGNED OFF
    • Beta:
    • Release:

Console Object Completion

This feature tracks work that was originally blended into the WebConsole7 feature, but not completed for Firefox 7.
When the Web Console shipped with Firefox 4, it provided the first console object to be always available to content in Firefox. It provided enough of the console object to be useful (console.log being by far the most popular method on the object). However, it was still lacking in features that developers commonly use.

  • Leads: Kevin Dangoor (Product, Directly Responsible individual), Panos Astithas (Lead Engineer)
  • QA Owner: Vlad Maniac, irc:vladmaniac (SoftVision)
  • Status: In progress. Health: OK - feature in development stage
  • Links: Feature Page, Tracking bug for time/timeEnd, Tracking bug for group/groupEnd
  • Notes:
  • Sign-off status:
    • Aurora: Signed-off
    • Beta: Signed-off
    • Release: Signed-off

Console Queued Message Display

If a user's JavaScript logs messages using console.log and similar methods before the Web Console has been opened, those messages are lost. In order to see that log output, the user needs to reload the page.
With this feature in place, the user will see those messages as soon as they open the Web Console.

  • Leads: Kevin Dangoor (Product, Directly Responsible individual), David Dahl (Lead Engineer), Additional members: Mihai Sucan
  • QA Owner:
  • Status: Blocked - feature in development stage
  • Links: Feature Page
  • Notes: Feature did not land in Firefox 10
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Panel-Based Download Manager

The current Download Manager user interface is not optimized for several common use cases, and is not yet integrated with the latest Firefox user experience design. Integrating this interface with the latest design is the first step to improve the downloads user experience.
In the new design, an easily accessible panel shows you current state of downloads, your recent downloads, and gives you a good indicator of how far along your download is. Advanced operations like searching, clearing entries, sorting, and everything else related to download history is integrated with the interface for browsing history.

  • Leads: Asa Dotzler (Product), Marco Bonardo (Feature), Paolo Amadini, Jared Wein (Lead Engineer), Alexander Limi (UX Lead), Additional members: Sinchan Banerjee, Stephen Horlander, Shawn Wilsher, Mehdi Mulani
  • QA Owner: irc: Simonab
  • Status: In Development. Health: OK
  • Links: Feature Page, Tracking Bug,Test Plan
  • Notes: Feature was moved to another release: Firefox 11
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Style Inspector

The Style Inspector works in conjunction with the Highlighter to allow the user to view the CSS rules being applied to an element. It will be an updated version of the style inspector that shipped disabled in Firefox 4.

  • Leads: Kevin Dangoor (Product, Feature), Mike Ratcliffe (Lead Engineer)
  • QA Owner: Florin Strugariu, irc:Bebe (SoftVision)
  • Status: Landed, Feature Complete
  • Links: Feature Page, Test Plan
  • Notes:
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Highlighter

The Highlighter is a visual entry point into tools that help web developers work om their page designs. It gets its name from the distinctive way in which it highlights an element on the page. The Highlighter presents some information and controls right near the element itself, making the most common operations quick and easy.
The Highlighter is a gateway to "element-centric" tools. The most important of these tools is the Style Inspector, which provides detailed information about the CSS applied to the selected node. These other tools are selected via a toolbar that also provides quick access to other elements around the highlighted node.
It is also planned that the HTML Tree view will be available as another way to move the highlight from one node to another.

  • Leads: Kevin Dangoor (Product, Feature), Rob Campbell (Lead Engineer), Other members: Paul Rouget
  • QA Owner: Teodosia Pop, irc:teodosia (SoftVision)
  • Status: Complete. Landed
  • Links: Feature Page, Tracking Bug, Feature TestPlan
  • Notes:
  • Sign-off status:
    • Aurora: Signed-off
    • Beta:
    • Release:

Media Control Refresh

Our HTML5 media controls have changed very little since they were introduced in Firefox 3.5 -- mostly just bug fixed and keeping them working as the spec evolves. HTML5 media is an area we've made serious investment in, and it's time to make sure our default controls show it.
This project is primarily a collection of bug-fixes, updated design work, and small-scope featurettes; all building on top of the existing implementation.
This feature falls primarily in the Experience category (from the "Discover, Experience, and Connect" vision statement.)

  • Leads: Asa Dotzler (Product), Justin Dolske (Feature), Jared Wein, Paul Adenot (Lead Engineer), Steven Horlander (UX Lead)
  • QA Owner: irc:simonab
  • Status: In Progress. Health: OK Note: Need UI for bug 681548
  • Links: Feature Page, Tracking Bug, Test Plan
  • Notes: Feature was moved to another release: Firefox 11
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Firefox Share

This is the bare minimum pieces needed to use the F1 share feature built into Firefox with a single working account.

  • Leads: Bryan Clark (Product, Feature), Shane Caraveo, Philipp von Weitershausen, Tarek Ziadé (Lead Engineer), Curtis Koenig, Yvan Boily, Daniel Veditz, Sid Stamm, David Chan, Lucas Adamski, Pete Fritchman (Security Lead), Sid Stamm (Privacy Lead), Aaron Train, Paul Silaghi(QA Lead), Stephen Horlander, Alex Faaborg (UX Lead)
  • QA Owner: Aaron Train, Paul Silaghi
  • Status: In Progress. Feature in planning. No ETA for now
  • Links: Feature Page, Tracking Bug, Test Plan
  • Notes: Feature was removed from Firefox 10 tracking. Currently it has no ETA
  • Sign-off status:
    • Aurora: Not implemented in Firefox 10 Aurora-11.11.2011
    • Beta:
    • Release:

Full Screen APIs

This feature allows you to build a web application that runs full screen. This includes any HTML element so you can build full screen games, full interactive video experiences, presentation software or anything else that should dominate the experience.

  • Leads: Chris Blizzard (Product), Chris Pearce (Feature Manager, Lead Engineer), Curtis Koenig (Security Lead), Tantek (Additional Members)
  • QA Owner: Paul Silaghi, irc: PaulS (SoftVision)
  • Status: Complete. Feature Landed
  • Links: Feature Page, Tracking Bug
  • Notes:
  • Sign-off status:
    • Aurora: Signed-off
    • Beta:
    • Release:

Improve display of location bar results

The current locationBar results feel heavy, busy, hard to scan and have not kept pace with system look and feel.
By streamlining and enhancing the results we aim to increase readability and the improve the appearance.
This feature falls primarily in the Discover category (from the "Discover, Experience, and Connect" vision statement.

  • Leads: Asa Dotzler (Product), Stephen Horlander (Feature), Steven Horlander (UX Lead)
  • QA Owner: Simona Badau, irc:simonab (SoftVision)
  • Status: In Progress. Health: OK Stage: Design; Note: Mockups complete
  • Links: Feature Page,Traking Bug
  • Notes: Feature was moved to another release: Firefox 11
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Silent Update: Add-ons Default to Compatible

The vast majority of add-ons work from one version of Firefox to the next without the need for developer maintenance, but under the current system, compatibility information must be updated in order for Firefox to enable the add-on for use. For add-ons hosted on AMO, this is done automatically. However, 75% of add-ons in use are not hosted on AMO, and are therefore a major compatibility obstacle for our users. All of the compatibility effort put into each release is simply because Firefox still assumes add-ons will be incompatible between versions, when they usually aren't.
We should change Firefox's assumption to be that add-ons are compatible, with a few exceptions. Binary add-ons are never compatible between releases and are also the highest risk of negative side effects. Firefox should automatically enable low-risk (non-binary) add-ons in new versions of Firefox, and check AMO for additional compatibility information.
When users upgrade to a new version of Firefox, only the add-ons that are actually incompatible should be disabled, and the rest are assumed to be compatible. Because Nightly, Aurora, and Beta users will test out the add-ons for weeks before stable users, we should be able to identify and blacklist incompatible add-ons before stable users would be affected by a truly incompatible add-on.

  • Leads: Justin Scott (Product, Feature), Dave Townsend, Wil Clouser (Lead Engineer)
  • QA Owner: Virgil Dicu, VirgilDicu (SoftVision)
  • Status: In Progress. Health: OK - Feature in development stage
  • Links: Feature Page, Tracking Bug, Test Plan
  • Notes:
  • Sign-off status:
    • Aurora: SIGNED OFF
    • Beta:
    • Release:

Silent Update: Removal of OS security dialog for Windows

We have broken out the OS dialogs into two feature pages given the requirement differences between Windows and Mac.
Below are the requirements for Windows where the current upgrade process runs into modal dialogs that interrupt the user's work flow.
For Windows: The UAC (user account control) dialog pop-ups during the update process
The requirement here is to override this dialog so it does not impact the user's workflow in any way

  • Leads: Chris Lee (Product), Lawrence Mandel (Directly Responsible individual), Brian Bondy (Lead Engineer), Alex Faaborg (UX Lead)
  • QA Owner: Simona Badau, irc:simonab (SoftVision)
  • Status: In progress. Health: OK - feature in development stage
  • Links: Feature Page, Tracking bug, Test Plan
  • Notes: Feature is not yet landed on the Aurora branch, available builds for testing can be found on the elm branch - Link
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Add plugincheck functionality to Add-on Manager

Integrate Plugin Check with installed plugins in the Add-ons Manager.

  • Leads: Kev Needham (Product), Dave Townsend, Wil Clouser (Lead Engineer), Jennifer Boriss (UX Lead)
  • QA Owner: Henrik Skupin. Ownership might be transferred to SoftVision team
  • Status: Planning. Health: Blocked. Feature has no ETA now
  • Links: Feature PageTracking Bug
  • Notes:Feature was not implemented in Firefox 10.
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Detailed Element Layout View

The browser goes through quite a few computations in order to decide how big an element should be on the page. The purpose of the Layout tool is to help the user see detailed information about the final dimensions of a page element.
Some layout information will be displayed in the Highlighter directly (like after this feature lands, however). Due to space constraints, all of the useful numbers cannot be displayed there. The Layout tool will provide the details that can't be easily represented over the page element itself.

  • Leads: Kevin Dangoor (Product, Feature), Paul Rouget (Lead Engineer),
  • QA Owner:
  • Status: In Progress. Health: OK - Feature in development stage.
  • Links: Feature PageTracking Bug
  • Notes:
  • Sign-off status:
    • Aurora:
    • Beta:
    • Release:

Input

The following are the details of daily triage conducted on http://input.mozilla.org.

Popular Feedback

Full Functional Tests

The following tracks testing done by iBeta for full functional test coverage. This task is performed once during Aurora only.

  • Lead: Anthony Hughes
  • [ link to test suite]
  • [ link to results of testing]
  • [ link to bugs filed by iBeta]

Website Compatibility

The following tracks testing done by uTest for website compatibility. This task is performed after every merge.

  • Lead: Anthony Hughes
  • Link to test suite
  • Link to report for Nightly Web Compatibility
  • Link to report for Aurora Web Compatibility
  • Link to report for Beta Web Compatibility
  • Link to bugs filed

Nightly Merge

Verify that the builds generated from the merge to Nightly have updated properly.

Nightly
  • [ON TRACK] [ashughes] Windows XP:
  • [ON TRACK] [ashughes] Windows 7:
  • [ON TRACK] [ashughes] Mac OS X 10.6:
  • [ON TRACK] [ashughes] Ubuntu 11.04:
Issues Found

Aurora Merge

Verify the first builds generated from the bits merged to Aurora.

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Assignee Result
Windows 2000 ashughes PASS - 2011-11-10
Windows XP ashughes PASS - 2011-11-10
Windows Vista ashughes PASS - 2011-11-10
Windows 7 32-bit ashughes PASS - 2011-11-10
Windows 7 64-bit ashughes PASS - 2011-11-10
Mac OS X 10.6 ashughes PASS - 2011-11-10
Mac OS X 10.7 ashughes PASS - 2011-11-10
Ubuntu 11.10 32-bit ashughes PASS - 2011-11-10
Ubuntu 11.10 64-bit ashughes PASS - 2011-11-10
Reports
Mozmill

Update Tests

The following are details of update verifications

Platform Assignee Result: Auroratest Result: Aurora
Windows 2000 ashughes PASS - 2011-11-10
Windows XP ashughes PASS - 2011-11-10
Windows Vista ashughes PASS - 2011-11-10
Windows 7 32-bit ashughes PASS - 2011-11-10
Windos 7 64-bit ashughes PASS - 2011-11-10
Mac OS X10.6 ashughes PASS - 2011-11-10
Mac OS 10.7 ashughes PASS - 2011-11-10
Ubuntu1.10 32-bit ashughes PASS - 2011-11-10
Ubuntu1.10 64-bit ashughes PASS - 2011-11-10
Reports
Auroratest, [ Aurora]

Manual Spot Checks and Feature Sign-Off

Are all features implemented at this point working?

  • CSS3 3D Transforms: signed-off
  • Full Screen APIs: signed-off
  • Silent Update: Add-ons Default To Compatible: signed-off

Manual spot Checks spreadsheet

  • Windows XP RESULTS
  • Windows 7 RESULTS
  • Ubuntu 11.10 RESULTS
  • Mac 10.6 RESULTS
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_10/Test_Plan&oldid=368574"