QA/Firefox3.5/TestPlan/Beta3

From MozillaWiki
< QA‎ | Firefox3.5‎ | TestPlan
Jump to: navigation, search

« QA/Firefox3.1/TestPlan

Firefox 3.1 beta3 Test Results

The covered testing for this release will focus on smoketests, bfts, l10n, top plugins, top financial sites, updates, and spotchecks.

Related Pages

Firefox 3.1 Tracking Site

Project contacts

PM: Mike Beltzner (beltzner)
Build: Nick Thomas (nthomas)
QA: Tony Chung (tchung)

Schedule

Code Freeze
3/3
Builds hand to QA
3/5
QA test signoff
3/12
  • See Test Coverage below

Build Location and IDs

Download:

Builds:

  • Build 1:
    • Mac 10.5: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
    • Mac 10.4: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
    • Windows: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
    • Linux: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3 (20090304215011)
  • Build 2:
    • Mac 10.5: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
    • Mac 10.4: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
    • Mac 10.5 PPC: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
    • Linux: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
    • Windows XP: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
    • Windows 7: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3

Test Coverage

  • Areas Covered:
    • Smoketests
    • BFTs
    • L10n
    • Top Plugins
    • Top Financial Sites
    • Updates
    • Spotchecks & Webpages
    • Bug Fixes
  • Areas Not Covered:
    • FFTs
    • Topsites (covered in nightly automation)
    • Top Themes
    • Accessibility

Test Results

Types of Tests Windows XP Mac OSX Linux
Smoketests Henrik - PASS
Aakash (Vista) - PASS 		Aakash - PASS
Tracy (PPC) - PASS 		juanb - PASS
BFTs Aakash/Henrik - PASS marcia - PASS juanb - PASS
FFTs Aakash/Henrik - result Tchung/marcia - result Aakash/TBD - result
L10n Tony - PASS Tony - PASS Tony - PASS
Top Plugins Aakash - PASS Murali - MIXED Murali - MIXED
Top Financial sites Tony - PASS Marcia - PASS Tony - PASS
Updates Henrik
Betatest - PASS
releasetest - PASS
Beta - PASS 		Murali
Betatest - PASS
releasetest - result
Beta - PASS 		Aakash
Betatest - PASS
releasetest - PASS
Beta - PASS
Staging/Live Webpages & Spotchecks & Webpages <web tester>, Henrik - PASS <web tester>, Aakash - PASS <web tester>, Tony - PASS

Smoketest Testing

Lite verification tests (or smoketests) will be first ran when the builds are ready. Tests are available on litmus under the Firefox 3.1 test suite. In any case the Smoketests result, we will raise blockers asap before proceeding with the bfts.

Please record your detailed results in the section below.

Note - please also make a smoketest pass against Windows 7 Beta, Mac OSX on PPC, and Windows Vista

BFT / FFT Testing

BFTs are a more comprehensive set of litmus tests that cover the build testing after smoketests result. Tests are available on litmus under the Firefox 3.1 test suite.

FFTs are a full comprehensive set of litmus tests that extend coverage in areas that BFTs dont often get to. These are run at least once during the major milestone timeline.

If you are running a BFT or FFT, Please record your detailed results in the section below.

Updates Testing

We will be doing updates of all Alpha users to beta3. We will be checking complete updates, and failover scenarios.

Will be available on betatest, releasetest, and beta channels.

Testcases and Results are tracked here.

L10N

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to Fx3.1 L10N Requirements for more specifics.

Testcases and Results are tracked here.

Top Plugins Testing

Select a few top plugins from AMO and perform the following testcases. (examples: Flash, Acrobat, Java, Real Player)

Testcases:

  • Enabling extensions and confirming web compatibility
  • Disabling or Uninstalling plugins and confirming proper messaging and behavior
  • Switch profiles and verify plugins are still working properly

Testcases and Results are tracked here.

Top Financial Sites Testing

Testcases:

  • Login/logout, verify functionality
  • Look for any layout discrepencies
  • Make a financial transaction if possible
  • Be sure to test in private browsing mode also

Staging/Live Spotchecks & Webpages

1) Verify the following webpages that gets pushed to staging and then to the live sites. Look for content messaging, spelling mistakes, broken links & imnages, etc..

2) Check live site download links on all platforms. Please cover en-US, and 2 or 3 additional Tier 1 locales as well. Pay attention to the following areas:

  • Check for signed installer and applications (windows only)
  • Firstrun page, whatsnew page, Home page, Search plugin
  • Install, uninstall
  • General spotchecks and browser functionality
  • Confirm Build IDs

Fixed Bugs Verifications

The following bugs are fixed for beta 3 and need verifications:

Testday

A beta3 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found on QMO.

Detailed Results

Smoketests Results

Windows XP [Henrik]

  • Build: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
  • Results
    • Build 1 (20090304233338) PASS - Results in Litmus (Complete)
    • Build 2 (20090305152042) PASS - Results in Litmus (Install/Uninstall)
  • Notes:
    • [Henrik]: filed bug 481714 - Switching to "Always ask" for file types handled by plugins don't work

Windows Vista [Aakash]

  • Build: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
    • 20090305152042
  • Notes:
    • Vista could not find network printers, so no test run there
      • Update: Marcia ran the test case from the Vista Lab PC and it passed

Windows 7 [Henrik]

  • Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
  • Results
    • Build 2 (20090305152042) PASS - Results in Litmus (Complete)
  • Notes
    • bug 482004 - Form history not imported from Internet Explorer 8

Mac [Aakash]

  • Build: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
    • 20090304214853
    • Status: PASS
  • Build: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
    • 20090305133223
    • Status: PASS
  • [tchung] Filed bug bug 481680 - [Mac OSX] Beta 3 build 1 are showing "Mozilla Firefox3.1b3" in window title
    • Blocker filed, fix will land in build 2 respin.
  • [aakash] Filed bug bug 481694 - Home Page not imported from Safari

Mac PPC 10.4 [Tracy]

  • Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
  • Status: PASS

Linux [Juan]

  • Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
    • 20090304215011
  • Status: PASS
  • Bugs:
    • Reopened (eventually re-Resolved) - bug 475066 - unable to detach tabs,
    • Opened new bug and nominated blocking 3.1 - bug 481693 - unable to detach tabs in Linux
    • Cannot (should not?) create web shortcuts. There is an existing bug, but I need to dig it out. It's also related to tabbed browsing (tear off).

Spotchecks [marcia]

  • Filed bug 481687 - Regression that affects only Mac

L10n testing

  • PASS
  • [tchung] bug 475032 - detected by hand that mac ko locale was missing from the /build2 directory. Build Slave glitch, being manually rebuilt.

Financial Testing

  • [tchung]
    • Linux: Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 PASS
    • Windows: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 PASS
    • Ran testcases on ebay, amazon, emigrant, boa, fidelity, gmail, y!mail.
    • No bugs found

Plugins Testing

Linux [Murali]

  • Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
    • 20090305134136
    • Status: MIXED RESULT
  • The Linux build has more leaks when firefox is run from commandline.
  • Verified the JRE bug 474866 as fixed.
  • Filed a bug 481963 which seems to be a DUP of 477110 [ Ref: Tomcat ]
  • The following plugins fail to work on Linux
  • Real Video crashes with Linux build. Known issue.
  • Viewpoint Media player, IPIX are not tested.
  • Final result is PARTIAL

MAC OS X86 [Murali]

  • View Point Media Plugin : Failure
  • IPIX : Not Applicable for MAC
  • Rest of the plugins from testing page are good.
  • Updated the results page for MAC as partial pass.

Win32 [ Aakash ]

  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
    • 20090304233338
    • Status: PASS
  • Abode Reader: PASS
  • Adobe Flash: PASS
  • Java: PASS
  • Quicktime: PASS, but no interaction with firefox on install/uninstall
  • RealPlayer: PASS
  • Shockwave: PASS
  • WMP: PASS
  • Silverlight: PASS
  • VMP: NOT SUPPORTED ON 3.x
  • iPIX: FAIL
  • Divx: PASS

BFT Results

Windows XP [Henrik/Aakash]

  • Build 1: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
  • Results
    • Build 1 (20090304233338) PASS - Results in Litmus (Mostly all)
    • Build 2 (20090305152042) PASS - Results in Litmus (Remaining + Tabs)
  • Notes:
    • [Henrik]: bug 463088, Tearing-off a tab to desktop shows denied cursor
    • [Henrik]: Filed bug 481775 Crash in [@ nsCookieService::RemoveCookieFromList(nsListIter&)]
    • [Henrik] Sometimes after removing a ssl cert the page is still loaded and doesn't show the ssl exception page. No bug filed so far. Have to check why it happens.
    • [Henrik] Leaving Private Browsing mode doesn't show up SSL sites only for each second time (not everytime) - commented in bug 463256
    • [Henrik] bug 468700 - Having "Ask me Everytime" enabled for Cookies during Private Browsing is unusable - asked for blocking1.9.1.
    • [Henrik] Filed bug 481785 - No indication of 'Private Browsing' in window title when having enabled "Always on" Private Browsing Mode
    • [Henrik] Filed bug 481786 - Current session data is getting lost when starting Firefox in "Always On" mode
    • [Henrik] Filed bug 481791 - about:privatebrowsing page doesn't show a button when staying in private browsing mode
    • [Henrik] Updated bug 134002, Content of plugins (flash, java applets) doesn't get rendered for printing or print preview
    • [Henrik] bug 426260, Yahoo doesn't correctly decode the parameters served by the email handler
    • [tchung] Reopened bug 436686, Flash fails to play on windows 7, flash 10.0.22.87
    • [aakash] Filed bug 481762 - Clearing tag via typed-text does not un-check checkbox
    • [aakash] Filed bug 481767 - Tag field overlay issue when 2 or more bookmarks in the same tag
    • [aakash] Filed bug 481904 - Tab dragging in Scroll can not move freely from Left to Right
    • [aakash] Added comments to bug 405511 - Dragging a tab onto its content area reloads that page

Mac [Marcia]

  • PASS
  • Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
  • Flash version is Shockwave Flash 10.0 r12
  • The Following Areas of BFT were run with Build 1:
    • Migration
    • Menu Bar
  • Notes
    • [Marcia/Henrik]: bug 476928 - Slow closing of tabs when entering Private Browsing mode
    • [Henrik]: bug 473467 file picker dialog sometimes doesn't close when Open is clicked
    • [marcia]: Bug 481908 Right clicking on video using Flash 10 causes screen to white out
    • [marcia]: Bug 481915 [Mac] View Certificate "General Tab" looks to be an active field that can be edited
    • [marcia]: Bug 481913 In certain instances "View Cookies" button in Page Info does not open Cookies dialog
    • [marcia]: Bug 481951 Print dialog very slow to open the first time
    • [marcia]: Bug 482286 Cookie dialog does not display properly when visting cnn.com the first time
    • [marcia]: Bug 482286 ESPN search plugin does not show correct favicon
  • Areas for further investigation
    • When you delete a home page and the field is blank, we default to the Firefox Start Page - is that correct?
    • espn.go.com site - the Big Ipod Touch banner ad sometimes moves to the left when it is playing.

Linux [juanb]

  • Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
    • 20090305134136
    • Status: Completed: PASS (as of Monday 6:30pm)
  • Notes
    • [Henrik]: Filed bug 482232 - Using search field in saved password window changes width of search field
    • bug 467565 - Commented on existing bug regarding private browsing autostart preference and its behavior.
    • https://litmus.mozilla.org/show_test.cgi?id=6290 - This test fails, but it also fails with 3.0.x and I need to investigate more.
    • https://litmus.mozilla.org/show_test.cgi?id=5948 - There is no visual distinction between an open Folder and a closed one. Minor nit.
    • I will investigate some of these and find out if there are actually problems. None of these would be blockers if they were confirmed.

FFT Results

Windows [Aakash/Henrik]

  • Build: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 ID:20090305152042
  • Results - PASS
    • Installation (aakash) [4 unclear]
    • Migration (aakash)
    • Default Browser (henrik) [3 unclear]
    • Menu Bar (aakash)
    • Options (aakash) [8 unclear]
    • General (aakash) [2 unclear]
    • Location Bar & Autocomplete (henrik) [11 unclear]
    • Tabbed Browsing (aakash, henrik) [17 unclear]
    • Search (Aakash) [2 unclear]
    • Pop-Up and Annoying Blocking (aakash) [2 unclear]
    • Find in Page (henrik)
    • Bookmarks (henrik) [6 unclear, 5 fail]
    • Library (henrik) [3 unclear, 4 fail]
    • RSS (henrik) [4 unclear, 1 fail]
    • History (aakash) [1 unclear]
    • MicroSummaries (aakash)
    • Add-ons Manager - ongoing (henrik) [7 unclear, 2 fail]
    • Downloading (aakash)
    • Toolbar Customization (henrik) [1 unclear, 3 fail]
    • UI Elements (henrik)
    • Printing (henrik)
    • Password Manager (aakash)
    • Form Manager (henrik)
    • Private Browsing (henrik) [2 unclear]
    • Cookies (aakash)
    • Import (aakash)
    • Plugins (aakash)
    • Layout (aakash)
    • Content Handling (aakash)
    • Safe Browsing (aakash, henrik) [2 unclear]
    • Security (aakash)
    • Session Restore (henrik) [4 unclear, 1 fail]
    • Online/Offline Events (aakash)
    • Technical Tools (aakash)
    • Uninstall (henrik)
    • Software Update (aakash)
  • Notes
    • Filed bug 484315: With browser.tabs.closeWindowWithLastTab set to false popups cannot be closed
    • Filed bug 484790: Copying elements via Ctrl+D&D in trees duplicates the selected entry instead of the clicked one
    • Filed bug 484801 - "Error: uncaught exception: 2147942487" when d&d an item into the empty area of the Library
    • Filed bug 484817 - Tagged bookmarks with a Javascript URI are not suggested via the location bar
    • Filed bug 484823 - Load bookmark in sidebar doesn't work when using auto-complete in location bar
    • Filed bug 485091 "Sort by Name" in Library sorts descending (z-a) instead of ascending (a-z)
    • Filed bug 485100 - Exchanging a letter of a tag name with its big/small equivalent removes tag from bookmark
    • Filed bug 485265 - JSON backup should force .json extension
    • Filed bug 485346 - Location bar doesn't have an icon in customize toolbar dialog
    • Filed bug 485602 - Certification Manager Buttons Enabled but not Usable on Highlighted Sites
    • Update for bug 412655 - Renaming a tag in the library to the name of another tag neither merges tags or rejects rename)
    • Update for bug 394353 - Tag list is not updated when manually adding, renaming, or deleting tags from within the tags field
    • Update for bug 350365 - Malformed feeds look empty
    • Update for bug 482687 - Crash when closing browser while customize toolbar dialog is open [@ nsXULWindow::SavePersistentAttributes()]
    • Update for bug 134002 - Content of plugins (flash, java applets) doesn't get rendered for printing or print preview
    • Update for bug 355739 - Work Offline mode not remembered between browsing sessions
    • Confirmed bug 425645 - Thie site is not removed from Offline Storage after using "Clear Offline Website Data"
    • Filed bug 486169 - Bug for js_SynthesizeFrame Crasher for Win XP
    • Safebrowsing - Currently livehttpheaders extension is not compatible with Fx 3.1b3, so there's a number of test cases that need to be run when it is

Mac [Tchung/Marcia]

  • tchung results
  • 3.1b3 build 20090305133223
  • Notes
    • Filed bug 483885 - Import Dialog has unnecessary vert/hor scrollbars
  • marcia's results
  • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
  • Sections Run on Leopard PPC Mac:
    • Printing
    • Toolbar Customization
  • Bugs Filed
    • bug 484105 - Error code: sec_error_bad_signature received during Litmus test run
    • bug 484112 - [JavaScript Error: "this._storage is null"] error after resetting Master Password
    • bug 484896 - [10.5] When using "Open with" .app shows in application name
    • bug 485076 - Progress meter moves backwards when downloading this site as web page, complete

Linux [Aakash/TBD]

  • Build: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 ID:20090305134136
  • Results
    • All PASS
    • Details: 85% test coverage; 21 bugs found, 93 marked unclear/broken
    • Safebrowsing
      • Can't run the rest of the outstanding test cases as livehttpheaders Addon is still incompatible with Fx 3.1b3.
  • Bugs Filed
    • [aakash] filed bug 484024 - Dropping Tab onto Desktop doesn't Create a Web Shortcut
    • [aakash] Confirmed bug 481904 - Tab dragging in tab bar stops scrolling at the tabs position when dragging to first/last tab and back
    • [aakash] filed bug 484288 - Moving non-Default Search Engines in Manager activates "Restore Default" button
    • [aakash] Confirmed bug 448033 - Keyword javascript bookmarks don't show in the location bar
    • [aakash] filed bug 484442 - Fx crashes on on bringing up safe-mode with any/all of the Options Checked
    • [aakash] filed bug 484489 - Drag and Drop History Item to Bookmarks Toolbar Results in JS Exception
    • [aakash] filed bug 485203 - Leaving Range blank results in abnormal printing behavior
    • [aakash] confirmed bug 416420 - In print preview, back and forward buttons are always enabled
    • [aakash] filed bug 484986 - Private Data Settings button in Privacy Tab is Usable in PB Mode
    • [aakash] filed bug 495369 - Help Menu "Not a Forgery" Option removed after "ignore" hyperlink selected
    • [aakash] filed bug 485440 - Password manager dialog window is not resized on Esc Press
    • [aakash] commented on bug 355739 - Work Offline mode not remembered between browsing sessions
    • [aakash] reproduced bug 485125 - Crash on close [@ libflashplayer.so@0x4110d]

Websites Testing

  • Notes
    • [tchung] bug 482738 opened to track missing locales
    • releasenotes link was pointing to 3.0.7 on the all-beta page
    • Missing robot image on /firstrun , /whatsnew page
    • [aakashd] filed bug 482987 - Other Systems and Languages Hyperlink not Synced to Authstage page

TestDay Results

  • Bugs Verified
  • Number of participants (37)
    • Volunteer Mods (9):
      • Tomcat, aakashd, juanb, timr, tchung, ctalbert, tracy, whimboo, marcia, stephend
    • Moz Employees (5):
      • dietrich, Jesse, abillings, jmaher, dolske
    • Community Members (23):
      • DevilNut, sonny, Coce, JonW, RedHat71, MichaelKohler, GeekShadow, hanthana, Zero456, macgeek23, m-c, firey10may, kushalone, mzz, mtgordon, kbrosnan, boivie, othree, gstrickler, varzan, dustyrayburn, Andrei_Larin, Matti
  • Test Cases Executed

Bugs Filed Post Release from Feedback

  • marcia
    • Bug 483644 Regression from Firefox 3.0.7 - XML Parsing Error when loading this page (works in Firefox 3.0.7)
    • Bug 483270 RadControls Rich Text Editor replaces certain characters with tilde
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox3.5/TestPlan/Beta3&oldid=138240"