Releases/Firefox 8/Test Plan

From MozillaWiki
Jump to: navigation, search

« Firefox 7 | Firefox 8 | Firefox 9 »

Firefox 8 Test Plan

Summary

Milestone Firefox 8
Status Currently in Beta -- Releasing on November 8th, 2011
Lead Juan Becerra (irc: juanb)
2nd Anthony Hughes (irc: ashughes)

Latest Build: ftp, http, web

The following is the test plan for Firefox 8 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 8. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

Daily Tasks
  • [ON TRACK] [softvision] Automation: Check dashboards for new failures
  • [ON TRACK] [softvision] Bug Triage: Check Bugzilla for regressions
  • [ON TRACK] [softvision] Features: Test features and triage feature bugs targeted for this releases
  • [ON TRACK] [softvision] Input: Triage Input.MO for prevalent issues and feature requests
One-time-only Tasks
  • [DONE] [juanb] Nightly Merge: Sign-off initial merge to Nightly
  • [ON TRACK] [ashughes] Aurora Merge: Sign-off merge from Nightly to Aurora
  • [ON TRACK] [iBeta] Full Functional Tests: Run the full suite of full functional tests on Aurora
  • [ON TRACK] [uTest] Website Compatibility: Run suite of Aurora website compatibility tests
  • [ON TRACK] [ashughes] Beta Merge: Sign-off merge from Aurora to Beta
  • [ON TRACK] [uTest] Website Compatibility: Run suite of Beta website compatibility tests
  • [ON TRACK] [ashughes] Release: Sign-off final release

Automation

BFTs

Triage the Mozmill dashboard for test failures and debug for potential Firefox regressions.

  • Lead: Vlad Maniac (Softvision)
  • Peer: Alex Lakatos (Softvision)
  • Failure Status: NEW (12) | ASSIGNED (7) | FIXED (3) | REGRESSIONS (0)
  • Tasks:
    • Check the Mozmill dashboard for failing tests for the day
    • Cross reference any failure to the list of NEW and ASSIGNED bugs
      • If the bug is NEW, assign it to yourself and try to fix it
      • If the bug is ASSIGNED and there has been no activity on the bug in over a week, offer your assistance to fix the bug
      • If there is NO bug on file, file it using this template
    • If you need assistance, ask ashughes in the #mozmill channel

Endurance

Performed during NIGHTLY phase only

Triage and report the performance numbers from the daily Endurance tests.

  • Lead: Vlad Maniac (Softvision)
  • Peer: Alex Lakatos (Softvision)
  • Endurance reports
  • Spreadsheet for tracking results
  • NOTE: Report any spikes in performance degradation to the QA release driver
Analysis (20110722 - 20110815)
  • Windows Memory Usage: +15.58% (77MB -> 89MB)
  • Mac Memory Usage: +3.27% (153MB -> 158MB)
  • Linux Memory Usage: +0.00% (84MB -> 84MB)
Issues Found

Bug Triage

Check UNCONFIRMED, NEW, FIXED, REOPEN, and tracking/status bugs

  • Lead: Vlad Maniac (Softvision)
  • Peer: Alex Lakatos (Softvision)
  • Bugs: QA+, FIXED
  • Results: VERIFIED, REOPENED
  • Tips:
    • If you don't know how to verify a bug, just skip it, move onto the next bug
    • Ignore code-level bugs, bugs in tests, build issues
Notes

Features

Track features which have landed in Firefox 8 and are ready to be tested (spreadsheet)

Feature

Description of feature

  • Leads: Person (role)
  • QA Owner: Owner
  • Status: Status
  • Links: Feature page, test plan, tracking bug, etc
  • Notes:

Tab move detach animations

When dragging tabs on the tab bar, we should show the tab and animate when you rearrange them, instead of showing a placeholder indicator like we currently do. We should also show a thumbnail of the page on detach operations, and make sure we have a good threshold, so this isn't triggered accidentally.

There are regressions and the team backed this change out for Firefox 8. The changes have been verified. The feature was backed out without other regressions.

Third Party Add-on Warnings

We currently do not provide adequate warning to users that new third-party provided add-ons have been installed. This project will ensure that users opt-in to all add-ons that aren't installed through the Firefox UI. This feature falls primarily in the Experience category (from the "Discover, Experience, and Connect" vision statement.)

  • Leads: Asa Dotzler (Product), Dave Townsend (Feature, Lead), Curtis Koenig (Security lead), Jennifer Boriss (UX Lead)
  • QA Owner: Vlad Maniac (:vladmaniac)
  • Status: Complete. Health: OK - Feature Landed.
  • Links: Feature Page, Tracking Bug Test Plan
  • Notes:

People Search

Users can currently search for general web content, ecommerce items, and reference material, but they cannot search for people. We should add some form of People search to our search list. Twitter search is probably a good first step.

Confirm add-ons on upgrade

Then next time existing users upgrade Firefox they should be presented with a list of add-ons they have installed where they can choose to disable them. By default add-ons from third-parties which may have been installed without the users' knowledge will be disabled. This replaces the compatibility wizard for that one time upgrade, subsequent upgrades use the compatibility wizard still.
This feature falls primarily in the Experience category (from the "Discover, Experience, and Connect" vision statement.)

  • Leads: Justin Scott (Product), Dave Townsend (Feature, Lead), Curtis Koenig (Security lead), Alex Faaborg (UX Lead)
  • QA Owner: Vlad Maniac (:vladmaniac)
  • Status: Complete. Health: OK - Feature Landed.
  • Links: Feature Page, Tracking Bug, Test Plan
  • Notes:

Expose prefs for tabs-on-demand

Firefox 4 has the hidden ability to load tabs on-demand by setting the maximum number of concurrent tabs being restored to "0" — now that the feature has been in the wild for a while, we'd like to expose it as the default behavior for session restore. It makes startup a lot faster, and uses a lot less memory.
This feature falls primarily in the Experience category (from the "Discover, Experience, and Connect" vision statement.)

  • Leads: Asa Dotzler (Product), Alex Limi (Feature), Paul O'Shannessy (Lead Engineer)
  • QA Owner: Mihaela Velimiroviciu
  • Status: Complete. Health: OK - Feature Landed.
  • Links: Feature Page, Tracking Bug, Test Plan
  • Notes:

Silent Update: Reduce Displayed UI

To reduce update fatigue, don't display application update download and notifications user interface unless necessary.
Twelve hours after an update is downloaded the user is told to restart Firefox by the display of the application update user interface. Instead of always notifying after twelve hours and possibly interrupting the user the notification should only be displayed when it is important for the user to apply the update (e.g. when the update contains an important security fix the user should be notified). From analyzing crashes (questionable source), approximately 15% of users run Firefox longer than 12 hours and would be affected (see bug 659425 for details).
This feature falls primarily in the Experience category (from the "Discover, Experience, and Connect" vision statement.)

  • Leads: Chris Lee (Product), Lawrence Mandel (Directly Responsible individual), Tim Taubert (Lead Engineer), Lucas Adamki (Security Lead), Alex Limi (UX Lead)
  • QA Owner: Paul Silaghi
  • Status: In progress. Health: OK
  • Links: Feature Page, Test Plan, Tracking Bug
  • Notes:

Input

The following are the details of daily triage conducted on http://input.mozilla.org.

  • Lead: Simona Badau, irc:simonab (Softvision)
  • Tasks:
    • Go to http://input.mozilla.org
    • Filter the results by Firefox version
    • Put your results here: spreadsheet
    • Any feedback which is spiking should listed below and escalated to the QA release driver
Popular Feedback

Full Functional Tests

The following tracks testing done by iBeta for full functional test coverage. This task is performed once during Aurora only.

  • Lead: Juan Becerra
  • [ link to test suite]
  • [ link to results of testing]
  • [ link to bugs filed by iBeta]

Website Compatibility

The following tracks testing done by uTest for website compatibility. This task is performed after every merge.

  • Lead: Juan Becerra
  • Link to test suite
  • Link to report for Aurora Web Compatibility
  • Link to report for Beta Web Compatibility
  • Link to bugs filed

Aurora Merge

Verify that the builds generated from the merge to Aurora have updated properly.

Auroratest
  • [DONE] [ashughes] Windows XP: PASS
    • Source: Mozilla/5.0 (Windows NT 5.1; rv:7.0a2) Gecko/20110813 Firefox/7.0a2
    • Target: Mozilla/5.0 (Windows NT 5.1; rv:8.0a2) Gecko/20110817 Firefox/8.0a2
  • [DONE] [ashughes] Windows 7: PASS
    • Source: Mozilla/5.0 (Windows NT 6.1; rv:7.0a2) Gecko/20110814 Firefox/7.0a2
    • Target: Mozilla/5.0 (Windows NT 6.1; rv:8.0a2) Gecko/20110817 Firefox/8.0a2
  • [DONE] [ashughes] Mac OS X 10.6: PASS
    • Source: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0a2) Gecko/20110812 Firefox/7.0a2
    • Target: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0a2) Gecko/20110817 Firefox/8.0a2
  • [DONE] [ashughes] Ubuntu 11.04: PASS
    • Source: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:7.0a2) Gecko/20110814 Firefox/7.0a2
    • Target: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:8.0a2) Gecko/20110817 Firefox/8.0a2
Aurora
  • [DONE] [ashughes] Windows XP: PASS
    • Source: Mozilla/5.0 (Windows NT 5.1; rv:7.0a2) Gecko/20110813 Firefox/7.0a2
    • Target: Mozilla/5.0 (Windows NT 5.1; rv:8.0a2) Gecko/20110819 Firefox/8.0a2
  • [DONE] [ashughes] Windows 7: PASS
    • Source: Mozilla/5.0 (Windows NT 6.1; rv:7.0a2) Gecko/20110814 Firefox/7.0a2
    • Target: Mozilla/5.0 (Windows NT 6.1; rv:8.0a2) Gecko/20110819 Firefox/8.0a2
  • [DONE] [ashughes] Mac OS X 10.6: PASS
    • Source: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0a2) Gecko/20110812 Firefox/7.0a2
    • Target: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0a2) Gecko/20110819 Firefox/8.0a2
  • [DONE] [ashughes] Ubuntu 11.04: PASS
    • Source: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:7.0a2) Gecko/20110814 Firefox/7.0a2
    • Target: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:8.0a2) Gecko/20110819 Firefox/8.0a2
Issues Found

Beta 1

The following tasks need to be completed to sign-off the Beta merge

  • [ON TRACK] [ashughes] Automated Tests: details
  • [ON TRACK] [softvision] Manual Tests: details
  • [ON TRACK] [ashughes] Updates: details

Builds: Firefox 8.0b1(1)
Litmus:testrun

Automated Tests

The following are the detailed results of the automated BFT test runs.

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
  • Reports: Litmus, Mozmill
Issues Found
  • Two tests are constantly failing and which need to be fixed as soon as possible.

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [Ioana] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [Andrei] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Ioana] Ubuntu 11.04: PASS
  • [DONE] [Virgil] Ubuntu 11.04 64-bit: PASS
  • Litmus testrun, report
  • Firefox 8.0b1 build1
Exploratory Tests

The following are the detailed results of the exploratory testing - Web App compatibility

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [Ioana] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [Andrei] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Ioana] Ubuntu 11.04: PASS
  • [DONE] [Virgil] Ubuntu 11.04 64-bit PASS
  • Litmus report
Issues Found
  • Html5
    • bug 690356 - Lower half of html5 youtube videos flickers in full screen
  • Tab animation
    • bug 689554 - The use of closing browser window shortcut key (ctrl+shift+w) while dnd a tab disturbs tab animation [NEW]
    • bug 689566 - Keyboard shortcut cmd+shift+w disturbs tab animation - dnd a tab from a window to another window [mac] [NEW]
    • bug 689575 - Disable keyboard shortcuts while tab animation is active [NEW]
    • bug 690341 - Using the close tab option from the menu opens a new tab when dragging a tab [NEW]

Updates

The following are details of update verifications for Firefox 8 beta 1

Releasetest: report

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS

Beta: report

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
Issues Found

Beta 2 (build1)

The following tasks need to be completed to sign-off the Beta merge

  • [ON TRACK] [ashughes] Automated Tests: details
  • [ON TRACK] [softvision] Manual Tests: details
  • [ON TRACK] [softvision] Bug fix verifications details
  • [ON TRACK] [ashughes] Updates: details

Builds: Firefox 8.0b2(1)
Litmus:testrun

Automated Tests

The following are the detailed results of the automated BFT test runs.

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
  • Reports: Litmus, Mozmill
Issues Found
  • /restartTests/testAddons_installMultipleExtensions/test1.js is failing on Win XP

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [Ioana] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [Mihaela] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Ioana] Ubuntu 11.04: PASS
  • [DONE] [Virgil] Ubuntu 11.04 64-bit: PASS
  • Litmus testrun, report
  • Firefox 8.0b2 build1
Exploratory Tests

The following are the detailed results of the exploratory testing - Web App compatibility

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [Ioana] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [Mihaela] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Ioana] Ubuntu 11.04: PASS
  • [DONE] [Virgil] Ubuntu 11.04 64-bit: PASS
  • Litmus report
Issues Found
  • bug 692418 -Print error is triggered the first time when attempting to print a yahoo mail message (New)
  • bug 692446 -Firefox prompts to install windows media player plugin even though it is already installed (New)

Bug Fix Verification

  • Release Driver: Triage these FIXED bugs to build the QA+ query
  • Release Testers: Verify these QA+ bugs 100%
  • Results: spreadsheet

Updates

The following are details of update verifications for Firefox 8 beta 2

Betatest:

  • run some manual exploratory tests around updating with add-ons, plugins, session/profile data, etc
  • How to set update channel to betatest:
    • Find the defaults/prefs/channel-prefs.js file in your Firefox application folder
    • Open it with a text editor and change the "beta" channel to "betatest", save and exit
    • Start Firefox and check app.update.channel in about:config to verify it changed
    • NOTE: On Windows Vista and 7 will not let you save the changes unless you run your text editor as an Administrator -- do so by right clicking the launcher for the text editor and select "Run as Administrator"
  • What to test
    • Build a realistic amount of profile and session data on one of the previous betas (4.0b8 through 8.0b1):
    • try installing several add-ons (some out of date)
    • try installing several plugins (some out of date)
    • go to various top sites like Youtube, Banking sites, Social networks, news sites, games, webmail etc
    • put some of the tabs in App Tabs and Panorama groupings
    • Ultimately just play around with your browser for an hour or so like you would on a day to day basis
    • After a while, check for and install the Firefox update to 8.0b2 and restart
  • Result -- check after restart:
    • no loss of session data
    • no loss of profile data
    • no loss of history, bookmarks
    • no loss of add-ons
    • no obvious breakage with Firefox and the sites you use most often
  • What to report
    • File bugs using the [8b2] whiteboard tag


Releasetest: report

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS

Beta: [ report]

  • [ON TRACK] [tester] Windows 2000: result
  • [ON TRACK] [tester] Windows XP: result
  • [ON TRACK] [tester] Windows Vista: result
  • [ON TRACK] [tester] Windows 7: result
  • [ON TRACK] [tester] Windows 7 64-bit: result
  • [ON TRACK] [tester] Mac OS X 10.6: result
  • [ON TRACK] [tester] Mac OS X 10.7: result
  • [ON TRACK] [tester] Ubuntu 11.04: result
  • [ON TRACK] [tester] Ubuntu 11.04 64-bit: result
Issues Found

Beta 2 (build 2)

The following tasks need to be completed to sign-off on this beta milestone.

  • [ON TRACK] [ashughes] Automated Tests: details
  • [ON TRACK] [softvision] Manual Tests: details
  • [ON TRACK] [softvision] Bug fix verifications details
  • [ON TRACK] [ashughes] Updates: details

Builds: Firefox 8.0b2(build 2)
Litmus:testrun

Automated Tests

The following are the detailed results of the automated BFT test runs.

  • [DONE] [ashughes] Windows 2000: RESULT
  • [DONE] [ashughes] Windows XP: RESULT
  • [DONE] [ashughes] Windows Vista: RESULT
  • [DONE] [ashughes] Windows 7: RESULT
  • [DONE] [ashughes] Windows 7 64-bit: RESULT
  • [DONE] [ashughes] Mac OS X 10.6: RESULT
  • [DONE] [ashughes] Mac OS X 10.7: RESULT
  • [DONE] [ashughes] Ubuntu 11.04: RESULT
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: RESULT
  • Reports: Litmus, Mozmill
Issues Found
  • /testSecurity/testSubmitUnencryptedInfoWarning.js failed on Mac 10.7 and Windows 2000
  • /restartTests/testAddons_installMultipleExtensions failed on Mac 10.7
  • /testTabbedBrowsing/testBackgroundTabScrolling.js failed on Windows Vista

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

  • [DONE] [Simona] Windows 2000: RESULT
  • [DONE] [Ioana] Windows XP: RESULT
  • [DONE] [Simona] Windows Vista: RESULT
  • [DONE] [Mihaela] Windows 7: RESULT
  • [DONE] [Mihaela] Windows 7 64-bit: RESULT
  • [DONE] [VladG] Mac OS X 10.6: RESULT
  • [DONE] [VladG] Mac OS X 10.7: RESULT
  • [DONE] [Ioana] Ubuntu 11.04: RESULT
  • [DONE] [Virgil] Ubuntu 11.04 64-bit: RESULT
  • Litmus testrun, report
  • Firefox 8.0b2 build2
Exploratory Tests

The following are the detailed results of the exploratory testing - Web App compatibility

  • [DONE] [Simona] Windows 2000: RESULT
  • [DONE] [Ioana] Windows XP: RESULT
  • [DONE] [Simona] Windows Vista: RESULT
  • [DONE] [Mihaela] Windows 7: RESULT
  • [DONE] [Mihaela] Windows 7 64-bit: RESULT
  • [DONE] [VladG] Mac OS X 10.6: RESULT
  • [DONE] [VladG] Mac OS X 10.7: RESULT
  • [DONE] [Ioana] Ubuntu 11.04: RESULT
  • [DONE] [Virgil] Ubuntu 11.04 64-bit: RESULT
  • Litmus report
Issues Found
  • bug 692418 -Print error is triggered the first time when attempting to print a yahoo mail message (New)
  • bug 692446 -Firefox prompts to install windows media player plugin even though it is already installed (New)

Bug Fix Verification

Verify 100% QA+ bugs, as many as possible FIXED bugs

Updates

The following are details of update verifications for Firefox 8 beta 2

Betatest:

  • run some manual exploratory tests around updating with add-ons, plugins, session/profile data, etc
  • How to set update channel to betatest:
    • Find the defaults/prefs/channel-prefs.js file in your Firefox application folder
    • Open it with a text editor and change the "beta" channel to "betatest", save and exit
    • Start Firefox and check app.update.channel in about:config to verify it changed
    • NOTE: On Windows Vista and 7 will not let you save the changes unless you run your text editor as an Administrator -- do so by right clicking the launcher for the text editor and select "Run as Administrator"
  • What to test
    • Build a realistic amount of profile and session data on one of the previous betas (4.0b8 through 8.0b1):
    • try installing several add-ons (some out of date)
    • try installing several plugins (some out of date)
    • go to various top sites like Youtube, Banking sites, Social networks, news sites, games, webmail etc
    • put some of the tabs in App Tabs and Panorama groupings
    • Ultimately just play around with your browser for an hour or so like you would on a day to day basis
    • After a while, check for and install the Firefox update to 8.0b2 and restart
  • Result -- check after restart:
    • no loss of session data
    • no loss of profile data
    • no loss of history, bookmarks
    • no loss of add-ons
    • no obvious breakage with Firefox and the sites you use most often
  • What to report
    • File bugs using the [8b2] whiteboard tag

Exploratory testing

  • [DONE] [Simona] Windows XP:PASS
  • [DONE] [Virgil] Ubuntu 11.04 x64:PASS
  • [DONE] [Ioana] Windows 7:PASS
  • [DONE] [VladG] MacOS X 10.6:PASS
    • Exploratory work spreadsheet Link

Releasetest: report

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS

Beta: report

  • [ON TRACK] [tester] Windows 2000: PASS
  • [ON TRACK] [tester] Windows XP: PASS
  • [ON TRACK] [tester] Windows Vista: PASS
  • [ON TRACK] [tester] Windows 7: PASS
  • [ON TRACK] [tester] Windows 7 64-bit: PASS
  • [ON TRACK] [tester] Mac OS X 10.6: PASS
  • [ON TRACK] [tester] Mac OS X 10.7: PASS
  • [ON TRACK] [tester] Ubuntu 11.04: PASS
  • [ON TRACK] [tester] Ubuntu 11.04 64-bit: PASS
Issues Found
  • There were three failures in the automation, which I wasn't able to reproduce manually (juanb).

Beta 3

The following tasks need to be completed to sign-off the Beta merge

  • [ON TRACK] [ashughes] Automated Tests: details
  • [ON TRACK] [softvision] Manual Tests: details
  • [ON TRACK] [softvision] Bug fix verifications details
  • [ON TRACK] [ashughes] Updates: details

Builds: Firefox 8.0b3(build1)
Litmus:testrun

Automated Tests

The following are the detailed results of the automated BFT test runs.

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
  • Reports: Litmus, Mozmill
Issues Found

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [AndreiD] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [AndreiD] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Ioana] Ubuntu 11.04: PASS
  • [DONE] [Mihaela] Ubuntu 11.04 64-bit: PASS
  • Litmus testrun, report
  • Firefox 8.0b3 build1
Exploratory Tests

The following are the detailed results of the exploratory testing - Web App compatibility

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [AndreiD] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [AndreiD] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Ioana] Ubuntu 11.04: PASS
  • [DONE] [Mihaela] Ubuntu 11.04 64-bit: PASS
  • Litmus report
Issues Found
  • bug 694288 - Vimeo.com - embedded video player is not displayed on Windows 2000 (New)
Dark Matter Addons

(re: bug 693698) -- Using this spreadsheet -- test on 32-bit Windows only -- use this etherpad to record your results

  • Pick an add-on and record it's name and URL from the spreadsheet to the etherpad
  • Go to the URL and install the software package
    • The notes will give you a hint of what to look for
    • There's some non-free software there, so we won't be able to get to all of them
  • Once the software is installed, start Firefox
    • Firefox may need to be closed to install some of the software packages
  • Open about:support and look for the name and ID of the new add-ons
    • They should be recorded in the GUID section
  • Look for the registry keys specified in this article
    • If there are registry keys for the add-on, enter Registry in the Installed in column
    • If not, look in Program Files/Mozilla Firefox/extensions -- if the add-on is there, enter App Dir in the Installed in column
    • If not, look in %Profile-Folder%/extensions -- if the add-on is there, enter Profile in the Installed in column

Add notes if you have any problems getting to the extension or getting it installed in Firefox, or if additional add-ons are being installed by a single package. Anything that gives us an idea of why an add-on couldn't be tested, or other add-ons we should be concerned about.

If you have any questions, please contact Andrew Williamson (eviljeff) on IRC in the #amo or #amo-editors channel.

Bug Fix Verification

Verify 100% QA+ bugs, as many as possible FIXED bugs

Updates

The following are details of update verifications for Firefox 8 beta 3

Betatest:

  • run some manual exploratory tests around updating with add-ons, plugins, session/profile data, etc
  • How to set update channel to betatest:
    • Find the defaults/prefs/channel-prefs.js file in your Firefox application folder
    • Open it with a text editor and change the "beta" channel to "betatest", save and exit
    • Start Firefox and check app.update.channel in about:config to verify it changed
    • NOTE: On Windows Vista and 7 will not let you save the changes unless you run your text editor as an Administrator -- do so by right clicking the launcher for the text editor and select "Run as Administrator"
  • What to test
    • Build a realistic amount of profile and session data on one of the previous betas (4.0b8 through 8.0b1):
    • try installing several add-ons (some out of date)
    • try installing several plugins (some out of date)
    • go to various top sites like Youtube, Banking sites, Social networks, news sites, games, webmail etc
    • put some of the tabs in App Tabs and Panorama groupings
    • Ultimately just play around with your browser for an hour or so like you would on a day to day basis
    • After a while, check for and install the Firefox update to 8.0b2 and restart
  • Result -- check after restart:
    • no loss of session data
    • no loss of profile data
    • no loss of history, bookmarks
    • no loss of add-ons
    • no obvious breakage with Firefox and the sites you use most often
  • What to report
    • File bugs using the [8b3] whiteboard tag

Exploratory testing

  • [DONE] [Simona] Windows XP:PASS
  • [DONE] [Ioana] Ubuntu 11.04 x86:PASS - with known issues: bug 658913
  • [DONE] [Mihaela] Windows 7:PASS
  • [DONE] [VladG] MacOS X 10.6:PASS
    • Exploratory work spreadsheet Link


Releasetest: report

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS

Beta: report

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
Issues Found

Beta 4

The following tasks need to be completed to sign-off the Beta merge

  • [ON TRACK] [ashughes] Automated Tests: details
  • [ON TRACK] [softvision] Manual Tests: details
  • [ON TRACK] [softvision] Bug fix verifications details
  • [ON TRACK] [ashughes] Updates: details

Builds: Firefox 8.0b4(build1)
Litmus:testrun

Automated Tests

The following are the detailed results of the automated BFT test runs.

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
  • Reports: Litmus, Mozmill
Issues Found

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [AndreiD] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [Paul] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Virgil] Ubuntu 11.04: PASS
  • [DONE] [Virgil] Ubuntu 11.10 64-bit: PASS
  • Litmus testrun, report
  • Firefox 8.0b4 build1
Exploratory Tests

The following are the detailed results of the exploratory testing - Web App compatibility

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [AndreiD] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [Paul] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Virgil] Ubuntu 11.04: PASS
  • [DONE] [Virgil] Ubuntu 11.10 64-bit: PASS
  • Litmus report
Issues Found

bug 696096 Items are not shown correctly in shopping cart [NEW]

Bug Fix Verification

Verify 100% QA+ bugs, as many as possible FIXED bugs

Updates

The following are details of update verifications for Firefox 8 beta 4

Betatest:

  • run some manual exploratory tests around updating with add-ons, plugins, session/profile data, etc
  • How to set update channel to betatest:
    • Find the defaults/prefs/channel-prefs.js file in your Firefox application folder
    • Open it with a text editor and change the "beta" channel to "betatest", save and exit
    • Start Firefox and check app.update.channel in about:config to verify it changed
    • NOTE: On Windows Vista and 7 will not let you save the changes unless you run your text editor as an Administrator -- do so by right clicking the launcher for the text editor and select "Run as Administrator"
  • What to test
    • Build a realistic amount of profile and session data on one of the previous betas (4.0b8 through 8.0b4):
    • try installing several add-ons (some out of date)
    • try installing several plugins (some out of date)
    • go to various top sites like Youtube, Banking sites, Social networks, news sites, games, webmail etc
    • put some of the tabs in App Tabs and Panorama groupings
    • Ultimately just play around with your browser for an hour or so like you would on a day to day basis
    • After a while, check for and install the Firefox update to 8.0b4 and restart
  • Result -- check after restart:
    • no loss of session data
    • no loss of profile data
    • no loss of history, bookmarks
    • no loss of add-ons
    • no obvious breakage with Firefox and the sites you use most often
  • What to report
    • File bugs using the [8b4] whiteboard tag

Exploratory testing

  • [DONE] [Andrei] Windows XP:PASS
  • [DONE] [Virgil] Ubuntu 11.04 x86:PASS
  • [DONE] [Mihaela] Windows 7:PASS
  • [DONE] [VladG] MacOS X 10.6:PASS
    • Exploratory work spreadsheet Link


Releasetest: report

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS

Beta: report

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
Issues Found

Beta 5

The following tasks need to be completed to sign-off the Beta merge

  • [ON TRACK] [ashughes] Automated Tests: details
  • [ON TRACK] [softvision] Manual Tests: details
  • [ON TRACK] [softvision] Bug fix verifications details
  • [ON TRACK] [ashughes] Updates: details

Builds: Firefox 8.0b5(build1)
Litmus:testrun

Automated Tests

The following are the detailed results of the automated BFT test runs.

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
  • Reports: Litmus, Mozmill
Issues Found

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [Ioana] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [AndreiD] Windows 7: PASS
  • [DONE] [AndreiD] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Mihaela] Ubuntu 11.04: PASS
  • [DONE] [Virgil] Ubuntu 11.10 64-bit: PASS
  • Litmus testrun, report
  • Firefox 8.0b5 build1
Exploratory Tests

The following are the detailed results of the exploratory testing - Web App compatibility

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [Ioana] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [AndreiD] Windows 7: PASS
  • [DONE] [AndreiD] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Mihaela] Ubuntu 11.04: PASS
  • [DONE] [Virgil] Ubuntu 11.04 64-bit: PASS
  • Litmus report
  • [DONE] [gmontagu] Windows 7, es-AR: PASS
Issues Found
  • bug 697696 - Highlight issue in amazon's password box [NEW]
  • bug 697698 - Pdf pages no longer loads if changing application handling pref on Linux [NEW]
  • bug 697725 - Amazon tab favicon flickers when page is loading [NEW]

Bug Fix Verification

Verify 100% QA+ bugs, as many as possible FIXED bugs

  • Bugs: QA+, FIXED
  • Fixed bugs verification spreadsheet Link
  • Highlights
    • Tab Animation has been backed out from all branches. This is visible in bug 690227

Updates

The following are details of update verifications for Firefox 8 beta 5

Betatest:

  • run some manual exploratory tests around updating with add-ons, plugins, session/profile data, etc
  • How to set update channel to betatest:
    • Find the defaults/prefs/channel-prefs.js file in your Firefox application folder
    • Open it with a text editor and change the "beta" channel to "betatest", save and exit
    • Start Firefox and check app.update.channel in about:config to verify it changed
    • NOTE: On Windows Vista and 7 will not let you save the changes unless you run your text editor as an Administrator -- do so by right clicking the launcher for the text editor and select "Run as Administrator"
  • What to test
    • Build a realistic amount of profile and session data on one of the previous betas (4.0b8 through 8.0b5):
    • try installing several add-ons (some out of date)
    • try installing several plugins (some out of date)
    • go to various top sites like Youtube, Banking sites, Social networks, news sites, games, webmail etc
    • put some of the tabs in App Tabs and Panorama groupings
    • Ultimately just play around with your browser for an hour or so like you would on a day to day basis
    • After a while, check for and install the Firefox update to 8.0b5 and restart
  • Result -- check after restart:
    • no loss of session data
    • no loss of profile data
    • no loss of history, bookmarks
    • no loss of add-ons
    • no obvious breakage with Firefox and the sites you use most often
  • What to report
    • File bugs using the [8b5] whiteboard tag

Exploratory testing

  • [DONE] [Ioana] Windows XP:PASS
  • [DONE] [Virgil] Ubuntu 11.10 x86_64:PASS
  • [DONE] [Andrei] Windows 7:PASS
  • [DONE] [VladG] MacOS X 10.6:PASS
    • Exploratory work spreadsheet Link

Releasetest: report

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS

Beta: report

  • [DONE] [juanb] Windows 2000: Pass
  • [DONE] [juanb] Windows XP: Pass
  • [ON TRACK] [tester] Windows Vista: result
  • [DONE] [juanb] Windows 7: Pass
  • [DONE] [tester] Windows 7 64-bit: Pass
  • [DONE] [juanb] Mac OS X 10.6: Pass
  • [DONE] [juanb] Mac OS X 10.7: Pass
  • [DONE] [juanb] Ubuntu 11.04: Pass
  • [DONE] [juanb] Ubuntu 11.04 64-bit: Pass
  • Notes: [juanb] - I ran these manually, maximizing coverage such that I covered OS, partials, completes, fallbacks, and locales with the paths that I tried.
Issues Found

Beta 6

The following tasks need to be completed to sign-off the Beta merge

  • [ON TRACK] [automation] Automated Tests: details
  • [ON TRACK] [softvision] Manual Tests: details
  • [ON TRACK] [softvision] Bug fix verifications details
  • [ON TRACK] [automation] Updates: details

Builds: Firefox 8.0b6(build1)
Litmus:testrun

Automated Tests

The following are the detailed results of the automated BFT test runs.

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
  • Reports: Litmus, Mozmill
Issues Found
  • /testSecurity/testGreyLarry.js fails on all platforms
  • /testSecurity/testIdentityPopupOpenClose.js fails on Windows 2000

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [Simona] Windows XP: PASS
  • [DONE] [Paul] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [Paul] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Ioana] Ubuntu 11.10: PASS
  • [DONE] [Virgil] Ubuntu 11.10 64-bit: PASS
  • Litmus testrun, report
  • Firefox 8.0b6 build1
Exploratory Tests

The following are the detailed results of the exploratory testing - Web App compatibility

  • [DONE] [Simona] Windows 2000: PASS
  • [DONE] [Simona] Windows XP: PASS
  • [DONE] [Paul] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [Paul] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [DONE] [VladG] Mac OS X 10.7: PASS
  • [DONE] [Ioana] Ubuntu 11.10: PASS
  • [DONE] [Virgil] Ubuntu 11.10 64-bit: PASS
  • Litmus report
  • [DONE] [gmontagu] Windows 7, es-AR (daily websites): PASS
Issues Found

Bug Fix Verification

Verify 100% QA+ bugs, as many as possible FIXED bugs

Updates

The following are details of update verifications for Firefox 8 beta 6. This is the last beta (RC) before shipping, so update scenarios must be exercised thoroughly before we ship.

UPDATE SCENARIOS TESTING RESULTS - Link

Issues found

  • Bug 699125 - Error message received when attempting to resume a download after upgrading to a new Firefox version [NEW]
  • Bug 699175 - The current theme is not active when updating from Firefox 7.0b6 to Firefox 8.0b6 [NEW]

Betatest:

  • run some manual exploratory tests around updating with add-ons, plugins, session/profile data, etc
  • How to set update channel to betatest:
    • Find the defaults/prefs/channel-prefs.js file in your Firefox application folder
    • Open it with a text editor and change the "beta" channel to "betatest", save and exit
    • Start Firefox and check app.update.channel in about:config to verify it changed
    • NOTE: On Windows Vista and 7 will not let you save the changes unless you run your text editor as an Administrator -- do so by right clicking the launcher for the text editor and select "Run as Administrator"
  • What to test
    • Build a realistic amount of profile and session data on one of the previous betas (4.0b8 through 8.0b6):
    • try installing several add-ons (some out of date)
    • try installing several plugins (some out of date)
    • go to various top sites like Youtube, Banking sites, Social networks, news sites, games, webmail etc
    • put some of the tabs in App Tabs and Panorama groupings
    • Ultimately just play around with your browser for an hour or so like you would on a day to day basis
    • After a while, check for and install the Firefox update to 8.0b6 and restart
  • Result -- check after restart:
    • no loss of session data
    • no loss of profile data
    • no loss of history, bookmarks
    • no loss of add-ons
    • no obvious breakage with Firefox and the sites you use most often
  • What to report
    • File bugs using the [8b6] whiteboard tag

Feature Sign-Off

Releasetest: report

  • [ON TRACK] [ashughes] Windows 2000: RESULT
  • [ON TRACK] [ashughes] Windows XP: RESULT
  • [ON TRACK] [ashughes] Windows Vista: RESULT
  • [ON TRACK] [ashughes] Windows 7: RESULT
  • [ON TRACK] [ashughes] Windows 7 64-bit: RESULT
  • [ON TRACK] [ashughes] Mac OS X 10.6: RESULT
  • [ON TRACK] [ashughes] Mac OS X 10.7: RESULT
  • [ON TRACK] [ashughes] Ubuntu 11.04: RESULT
  • [ON TRACK] [ashughes] Ubuntu 11.04 64-bit: RESULT

Beta: report

  • [ON TRACK] [tester] Windows 2000: result
  • [ON TRACK] [tester] Windows XP: result
  • [ON TRACK] [tester] Windows Vista: result
  • [ON TRACK] [tester] Windows 7: result
  • [ON TRACK] [tester] Windows 7 64-bit: result
  • [ON TRACK] [tester] Mac OS X 10.6: result
  • [ON TRACK] [tester] Mac OS X 10.7: result
  • [ON TRACK] [tester] Ubuntu 11.04: result
  • [ON TRACK] [tester] Ubuntu 11.04 64-bit: result
Issues Found

Beta 6 (build2)

The following tasks need to be completed to sign-off the Beta merge

  • [ON TRACK] [softvision] Manual Tests: details
  • [ON TRACK] [softvision] Bug fix verifications details
  • [ON TRACK] [geo] Updates: details

Builds: Firefox 8.0b6(build2)
Litmus:testrun

Manual Tests

Smoketests

The following are the detailed results of the manual smoketests.

  • [ON TRACK] [Simona] Windows 2000: RESULT
  • [ON TRACK] [Ioana] Windows XP: RESULT
  • [ON TRACK] [Simona] Windows Vista: RESULT
  • [ON TRACK] [Mihaela] Windows 7: RESULT
  • [ON TRACK] [Mihaela] Windows 7 64-bit: RESULT
  • [ON TRACK] [VladG] Mac OS X 10.6: RESULT
  • [ON TRACK] [VladG] Mac OS X 10.7: RESULT
  • [ON TRACK] [Ioana] Ubuntu 11.04: RESULT
  • [ON TRACK] [Virgil] Ubuntu 11.04 64-bit: RESULT
  • Litmus testrun, report
  • Firefox 8.0b6 build2
Exploratory Tests

Focus on add-ons and web compatibility.

  • [ON TRACK] [Simona] Windows 2000: RESULT
  • [ON TRACK] [Ioana] Windows XP: RESULT
  • [ON TRACK] [Simona] Windows Vista: RESULT
  • [ON TRACK] [Mihaela] Windows 7: RESULT
  • [ON TRACK] [Mihaela] Windows 7 64-bit: RESULT
  • [ON TRACK] [VladG] Mac OS X 10.6: RESULT
  • [ON TRACK] [VladG] Mac OS X 10.7: RESULT
  • [ON TRACK] [Ioana] Ubuntu 11.04: RESULT
  • [ON TRACK] [Virgil] Ubuntu 11.04 64-bit: RESULT
  • Litmus report
Issues Found

Bug Fix Verifications

Verify 100% QA+ bugs, as many as possible FIXED bugs

  • Bugs:

bug 680802 - VERIFIED FIXED

bug 689375 - VERIFIED FIXED

bug 698753 - VERIFIED FIXED

bug 687367 - Mobile

  • Fixed bugs verification spreadsheet Link

Manual spot checks

  • Exploratory work spreadsheet - Link

Issues found

bug 699435 - The third party add-on warning message is misplaced when using a theme [NEW]

Updates

Releasetest: report

  • [ON TRACK] [ashughes] Windows 2000: PASS
  • [ON TRACK] [ashughes] Windows XP: PASS
  • [ON TRACK] [ashughes] Windows Vista: PASS
  • [ON TRACK] [ashughes] Windows 7: PASS
  • [ON TRACK] [ashughes] Windows 7 64-bit: PASS
  • [ON TRACK] [ashughes] Mac OS X 10.6: PASS
  • [ON TRACK] [ashughes] Mac OS X 10.7: PASS
  • [ON TRACK] [ashughes] Ubuntu 11.04: PASS
  • [ON TRACK] [ashughes] Ubuntu 11.04 64-bit: PASS

Beta: report

  • [ON TRACK] [tester] Windows 2000: PASS
  • [ON TRACK] [tester] Windows XP: PASS
  • [ON TRACK] [tester] Windows Vista: PASS
  • [ON TRACK] [tester] Windows 7: PASS
  • [ON TRACK] [tester] Windows 7 64-bit: PASS
  • [ON TRACK] [tester] Mac OS X 10.6: PASS
  • [ON TRACK] [tester] Mac OS X 10.7: PASS
  • [ON TRACK] [tester] Ubuntu 11.04: PASS
  • [ON TRACK] [tester] Ubuntu 11.04 64-bit: PASS
Issues Found

Firefox 8 Final

The following tasks need to be completed to sign-off before final release.

  • [ON TRACK] [automation] Automated Tests: details
  • [ON TRACK] [geo] Updates: details

Builds: Firefox 8.0 RC
Litmus:testrun

Automated Tests

  • [ok] [automation] Windows 2000: PASS
  • [ok] [automation] Windows XP: PASS
  • [ok] [automation] Windows Vista: PASS
  • [ok] [automation] Windows 7: PASS
  • [ok] [automation] Windows 7 64-bit: PASS
  • [ok] [automation] Mac OS X 10.6: PASS
  • [ok] [automation] Mac OS X 10.7: PASS
  • [ok] [automation] Ubuntu 11.04: PASS
  • [ok] [automation] Ubuntu 11.04 64-bit: PASS
  • Reports: Litmus, Mozmill
Issues Found

Bug Fix Verifications

Verify 100% QA+ bugs, as many as possible FIXED bugs

Updates

Releasetest: report

  • [ON TRACK] [automation] Windows 2000: PASS
  • [ON TRACK] [automation] Windows XP: PASS
  • [ON TRACK] [automation] Windows Vista: PASS
  • [ON TRACK] [automation] Windows 7: PASS
  • [ON TRACK] [automation] Windows 7 64-bit: PASS
  • [ON TRACK] [automation] Mac OS X 10.6: PASS
  • [ON TRACK] [automation] Mac OS X 10.7: PASS
  • [ON TRACK] [automation] Ubuntu 11.04: PASS
  • [ON TRACK] [automation] Ubuntu 11.04 64-bit: PASS

Release: report

  • [ON TRACK] [tester] Windows 2000: result
  • [ON TRACK] [tester] Windows XP: result
  • [ON TRACK] [tester] Windows Vista: result
  • [ON TRACK] [tester] Windows 7: result
  • [ON TRACK] [tester] Windows 7 64-bit: result
  • [ON TRACK] [tester] Mac OS X 10.6: result
  • [ON TRACK] [tester] Mac OS X 10.7: result
  • [ON TRACK] [tester] Ubuntu 11.04: result
  • [ON TRACK] [tester] Ubuntu 11.04 64-bit: result
Issues Found

Firefox 8.0.1

As of 11-20 we have build2 candidates. The results from build1 will be reset, but a new testrun will be created for it.

Automated Tests

We will run the automated tests using Mozmill as soon as we get build candidates.

  • [DONE] [ashughes] Windows 2000: PASS
  • [DONE] [ashughes] Windows XP: PASS
  • [DONE] [ashughes] Windows Vista: PASS
  • [DONE] [ashughes] Windows 7: PASS
  • [DONE] [ashughes] Windows 7 64-bit: PASS
  • [DONE] [ashughes] Mac OS X 10.6: PASS
  • [DONE] [ashughes] Mac OS X 10.7: PASS
  • [DONE] [ashughes] Ubuntu 11.04: PASS
  • [DONE] [ashughes] Ubuntu 11.04 64-bit: PASS
  • Reports: Litmus, Mozmill

Issues Found

Manual Tests

Smoketests

Manual smoketests were skipped in favor of focused testing around the fixes, including exploratory testing.

  • [ON TRACK] [Simona] Windows 2000: result
  • [ON TRACK] [Ioana] Windows XP: result
  • [ON TRACK] [Simona] Windows Vista: result
  • [ON TRACK] [Mihaela] Windows 7: result
  • [ON TRACK] [Paul] Windows 7 64-bit: result
  • [ON TRACK] [VladG] Mac OS X 10.6: result
  • [ON TRACK] [VladG] Mac OS X 10.7: result
  • [ON TRACK] [Virgil] Ubuntu 11.10: result
  • [ON TRACK] [Virgil] Ubuntu 11.10 64-bit: result
  • Litmus testrun, report
  • Firefox 8.0.1 (build2) build2
Exploratory Tests

The following are the detailed results of the exploratory testing - Web App compatibility

  • [ON TRACK] [Simona] Windows 2000: result
  • [DONE] [Ioana] Windows XP: PASS
  • [DONE] [Simona] Windows Vista: PASS
  • [DONE] [Mihaela] Windows 7: PASS
  • [DONE] [Paul] Windows 7 64-bit: PASS
  • [DONE] [VladG] Mac OS X 10.6: PASS
  • [ON TRACK] [VladG] Mac OS X 10.7: result
  • [ON TRACK] [Virgil] Ubuntu 11.10: result
  • [DONE] [Virgil] Ubuntu 11.10 64-bit: PASS
  • Litmus report

Issues Found

  • bug 704133 - Shockwave player is not working anymore with Firefox [NEW]

Bug Verifications

The following bugs need verification on all platforms:

Link to etherpad with our testing efforts

Blockers

  • [DONE] - Bug 700835 - Mac crash with Apple's Java update
  • [DONE] - Bug 699134 - Block request for Roboform
    • We are blocking all versions, below 7.6.2. Only 7.6.3 and 7.6.2 should be allowed.
    • All versions of RoboForm downloads here
    • IMO VERIFIED FIXED - I haven't change the status of this bug.

Results

Issues Found

Updates

READ: For 8.0.1, the following applies:

  • Windows/Linux users already in version 8.0 will NOT be offered updates to 8.0.1
  • All Mac users will be offered updates to 8.0.1: 4...8 -> 8.0.1
  • Windows/Linux users in 4...7.0.1 will be offered updates to 8.0.1

Betatest: PASS (report)

  • Win XP (no updates should be offered)
    • 8.0 - en-US, es-ES, it: No updates offered, as expected.
  • Linux (no updates should be offered)
    • 8.0 - en-US, fr, ja: No updates offered, as expected.
  • Win2000 (no updates should be offered)
    • 8.0 - en-US, ru: No updates offered, as expected.
  • Win7
    • 8.0 - en-US: No updates offered, as expected.

Releasetest: PASS (report)

  • [juanb] - I did the type of manual checks as above (same platforms, same paths), but on the releasetest channel. In addition, I checked that older versions did get an update to 8.0.1

Release: PASS (report)

  • [juanb] - Again, similar types of checks on release. No problems found.
  • [ashughes] - some gaps in results caused by slow mirrors triggering timeout

Issues Found:

3.6.24 Major Update

Tester: ashughes Releasetest: report Beta: report Release: report
Windows XP PASS - 2011-11-29 PASS - 2011-11-30 PASS - 2011-11-30
Windows 7 PASS - 2011-11-29 PASS - 2011-11-30 PASS - 2011-11-30
Mac OS X 10.6 PASS - 2011-11-29 PASS - 2011-11-30 PASS - 2011-11-30
Ubuntu 11.04 PASS - 2011-11-29 PASS - 2011-11-30 PASS - 2011-11-30

NOTE: single failure on each testrun is expected due to changing branches of mozmill-tests

Post-Mortem Notes

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_8/Test_Plan&oldid=374825"