Firefox:1.5.0.x:TBMajor Update 1.5.0.x to 2.0.0.x:Test Results:15012

From MozillaWiki
Jump to: navigation, search

Testing results tracking page for major updates. We have tested major updates in the past by following the major software update test cases in Litmus, but currently we can do this type of test on "releasetest" going from 1.5.0.12->2.0.0.6. Bear in mind that the testing environment is temporary.

Use the major software update testing strategy page as reference. In-line Details should be translated for all locales, although it is unclear if we will have these ready for Thunderbird major updates this time around. EULAs will default to the en-US locale. These will not be translated.

Test on betatest or releasetest.

Mac (Intel) - juanb

en-US - PASS

  • Thunderbird BuildIDs: 2007050903-2007072816
  • Tested with a profile containing different accounts and settings:
    • POP, IMAP, RSS Feeds, Newsgroups
    • Toolbar customizations
    • Saved searches
    • Filters
    • Tags, Flags
    • Sorting
    • Address book entries, mailing lists
    • LDAP configuration
    • Compatible and incompatible extensions/themes
  • Accounts and settings were preserved
  • Extensions and Themes were checked for compatibility
  • Functionality remains OK.
  • Software Update History showed successful update entry
  • Fallback update (after a forced failure) worked as well.

es-ES - PASS

  • Checked similar things and all seemed ok
  • Major Software Update dialog box doesn't look as polished as en-US

ja-jp-mac - PASS

fr - PASS

Mac (PPC) - marcia

en-US

  • Scenario 1 = PASS
  • G5 PPC Mac running 10.4.10
  • testing with a 1.5.0.x profile created on PPC 10.3.9
  • betatest and releastest
  • mozilla.com, IMAP, POP3, newsgroup and blog account
  • From version 1.5.0.12 (20070509) to version 2.0.0.6 (20070728)
  • Configured to automatically check for updates. Hit "Later."
  • Installed a theme as was compatible with 1.5 but not with 2.0.0.6 and installed it.
  • Check for updates manually and get the update.
  • Bugs noted during testing: Bug 349594 - "back/go back" button disabled in software update wizard on the license page for major updates (and elsewhere)
  • Scenario 2 - PASS
  • Lab PPC Mac running 10.3.9
  • testing with a 1.5.0.x profile created on PPC 10.3.9
  • releasetest
  • mozilla.com, IMAP, POP3, newsgroup and blog account
  • From version 1.5.0.12 (20070509) to version 2.0.0.6 (20070728)

Windows XP - Tomcat, stephend, marcia, TimK, whimboo

stephend test results

en-US - stephend

  • Using mscott's XML file for Win, I tested the migration of:
    • labels->tags (changed their default colors and names)
    • filters
    • saved searches (both IMAP/POP3)
    • address books
      • cards/mailing lists
    • column ordering (drag and drop)
    • newsgroups
      • regular and SSL w/certificates
  • Tested the suggested 1.5-only/1.5/2 and popular add-ons, and the 1.5-only ones weren't upgraded (as expected). The others were.
  • NOTE: Update history didn't work, but hopefully that's because I'm using mscott's file
  • 1.5.0.12 (2007050903) -> 2.0.0.6 (2007072817)
 * Haven't yet run the BFT, or even the smoketests, but did a lot of ad-hoc; sorry, I'm away at the baseball game, but will resume testing when I return

timk test results

  • En-us (timk)
  • Build ID: version 2.0.0.6 (20070728)
  • Basic upgrade test passed
    • Later, never
    • Extensions, 1.5 only and 2.0
    • Themes upgraded successfully, though I can't be 100% certain everything looked as it should
    • Feeds
    • Address book
    • Pop-imap-newsgroup accounts
    • Prefs

es-es (timk)

  • Build-id: Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.8.1.6) Gecko/2007072817 Thunderbird/2.0.0.6
  • Same tests as en-us, passed
  • The detail pane wasn't localized, but apparently that's not scheduled to happen immediately.

marcia test results

  • All PASS except for mismatch issue noted.
  • en-US (marcia)
  • Build-id:
  • 1.5 profile with mozilla.com IMAP
  • Tested composing mail to send offline. Don't send it, but wait until after upgrade to send. I did get the dialog prompt to send the mail, but I get a mail relay error when the mail tries to be sent.
  • it (marcia)
  • Build-id: versione 2.0.0.6 (20070728)
  • 1.5 profile with mozilla.com IMAP
  • Tested composing mail to send offline. Don't send it, but wait until after upgrade to send. I did get the dialog prompt to send the mail, but I get a mail relay error when the mail tries to be sent.

whimboo test results

en-US (whimboo)

  • Build-id: Gecko/2007072817 Thunderbird/2.0.0.6
  • Compared original 2.0.0.6 application folder with MU application folder - no missing or remaining files.
  • tested update while files are in use. inifinitly restart: bug 340535 happens

Tomcats Test Results

  • en-US (tomcat)
  • Build-id:
  • 1.5 profile with mozilla.com IMAP and yahoo pop
  • Tested some Extensions and Mail profiles with mails and newsgroups and all went as expected.
  • de (tomcat)
  • Build-id: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070728 Thunderbird/2.0.0.6 Mnenhy/0.7.5.0 ID:2007072817
  • 1.5 profile with mozilla.com IMAP and yahoo pop
  • Tested some Extensions and Mail profiles with mails and newsgroups and all went as expected.
  • de (tomcat)
  • Build-id: tested on mac - ppc
  • 1.5 profile with mozilla.com IMAP and yahoo pop
  • Tested some Extensions and Mail profiles with mails and newsgroups and all went as expected.

Windows Vista - abillings, marcia

en-US

  • Tested under various user scenarios, results are available at http://quality.mozilla.org/vista+update+testing
    • NOTE: If you launch Firefox from the shortcut (without "Run as administrator"), software update will fail. You will be able to check for updates, download the patch, and restart... but it will not be applied.)

Windows 2K marcia

en-US

  • Scenario 1 - PASS
  • testing with an existing POP profile stored on the lab machine. No extensions installed.
  • releasetest
  • mozilla.com POP3
  • From version 1.5.0.12 (20070509) to version 2.0.0.6 (20070728)
  • Scenario 2 - PASS
  • testing with an existing POP profile stored on the lab machine. No extensions installed.
  • betatest
  • fail the update and fallover to a full update
  • mozilla.com POP3
  • From version 1.5.0.12 (20070509) to version 2.0.0.6 (20070728)


Windows 98 marcia

en-US

  • Pending reinstallation of OS on lab machine

Linux - ashughes

Profile Details

  • Newsgroups: news@mozilla, news@motzerella (SSL)
  • IMAP: mozilla.com
  • POP3: gmail.com
  • Custom Toolbar:
    • Added Buttons: Prev, Next, File, Seperator
    • Moved Buttons: Address Book, Forward
    • Removed Buttons: Junk
    • Other Settings: Use small icons
  • Search Folders:
    • IMAP - search for "tbird" from SUBJECT stored in "Inbox-TBird"
    • POP - search for "Armen" from FROM stored in "Inbox-Armen"
  • Filters:
    • IMAP - "test"
    • POP - "test"
  • Options Dialog: set to Composition>Addressing tab
  • Cards/List - created 3 "test" cards and populated 1 "test" list
  • Layout - set to vertical view (3-pane)
  • Themes:
    • Pitch Dark - installed but not enabled
    • Black Japan - installed but not enabled
    • Vista Mail - installed but not enabled
    • Noia 2.0 eXtreme - installed but not enabled
  • Extensions:
    • Contacts Sidebar - installed and enabled
    • NewsWorthy - installed and enabled
    • Adblock Plus - installed and enabled
    • FlashGot - installed and enabled
  • RSS Reader: digg
  • Inbox column headings reordered to SENDER, DATE, SUBJECT

en-US

  • Scenario 1:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: MScott's XML method
    • Date: 8 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
  • Scenario 2:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, ran litmus testcases
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Litmus testcases: test 12 failed due to above toolbar problem, test 13 unclear due to below Update History issue, all other tests PASS.
      • Build ID reported in Update History is 2007071612
  • Scenario 3:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, theme set to Black Japan (1.5 only)
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Theme reverted to default theme with no warnings. Reverting back to 1.5.0.12 restores Black Japan theme.
  • Scenario 4:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, theme set to Pitch Dark (1.5/2 compatible)
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Theme Pitch Dark still enabled after update.
  • Scenario 5:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, theme set to Vista Mail (popular)
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Theme Vista Mail still enabled after update.
  • Scenario 6:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, theme set to Noia 2.0 eXtreme (popular)
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Theme Noia 2.0 eXtreme still enabled after update.
  • Scenario 7:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = betatest
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12

fr

  • Scenario 1:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
  • Scenario 2:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = betatest
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
  • Scenario 3:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, set theme to 1.5 only theme
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Theme reverted to default theme after update. Downgrading to 1.5.0.12 reverts the theme to the 1.5 only theme.
  • Scenario 4:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = betatest, set theme to 2.0 compatible theme
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Custom theme still enabled after update.
  • Scenario 5:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, ran litmus tests
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Custom theme still enabled after update.
      • Build ID reported in Update History is 2007071612
      • Litmus testcases: test 12 failed due to above toolbar problem, test 13 unclear due to Update History issue, all other tests PASS.

de

  • Scenario 1:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
  • Scenario 2:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = betatest
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
  • Scenario 3:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, set theme to 1.5 only theme
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Theme reverted to default theme after update. Downgrading to 1.5.0.12 reverts the theme to the 1.5 only theme.
  • Scenario 4:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = betatest, set theme to 2.0 compatible theme
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Custom theme still enabled after update.
  • Scenario 5:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, ran litmus tests
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Custom theme still enabled after update.
      • Build ID reported in Update History is 2007071612
      • Litmus testcases: test 12 failed due to above toolbar problem, test 13 unclear due to Update History issue, all other tests PASS.

jp

  • Scenario 1:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
  • Scenario 2:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = betatest
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
  • Scenario 3:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, set theme to 1.5 only theme
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Theme reverted to default theme after update. Downgrading to 1.5.0.12 reverts the theme to the 1.5 only theme.
  • Scenario 4:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = betatest, set theme to 2.0 compatible theme
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Custom theme still enabled after update.
  • Scenario 5:
    • 1.5.0.12 (20070509) -> 2.0.0.6 (20070728)
    • Update method: channel = releasetest, ran litmus tests
    • Date: 10 Aug 2007
    • Results: PASS
    • Notes:
      • Themes and Extensions that did not have 2.0 compatible versions were checked for updates and none were found. This is expected behaviour.
      • Buttons that were added to the custom toolbar remained, however all other changes were set to defaults.
      • Everything else was migrated and appeared the same as in 1.5.0.12
      • Custom theme still enabled after update.
      • Build ID reported in Update History is 2007071612
      • Litmus testcases: test 12 failed due to above toolbar problem, test 13 unclear due to Update History issue, all other tests PASS.

Focused Testing

Let's try to go through all of the tests in the major updates reference page and test around different kinds of profiles, containing multiple extensions and themes, compatible and incompatible with 2.0x. Keep an eye out for data loss problems, such as mail, labels/tags/settings, and any other preferences that should be preserved after an update.

Bugs Discovered during Testing

Retrieved from "https://wiki.mozilla.org/index.php?title=Firefox:1.5.0.x:TBMajor_Update_1.5.0.x_to_2.0.0.x:Test_Results:15012&oldid=65178"