Stub Installer/QA

From MozillaWiki
Jump to navigation Jump to search

Stub Installer Signoff

Schedule

  • [DONE] 10/12/2012: Funnelcake Beta Test Pass
  • [DONE] 10/15/2012: Aurora Test Pass
  • [DONE] 10/15/2012: Nightly Test Pass
  • 03/07/2013: Aurora 21 Test Pass
  • 03/14/2013: Beta 20 Test Pass

Nightly

Checklist

  • [DONE] Smoke test pass on Nightly
    • Jason will own testing of all platforms
  • [DONE] Bug verifications for blocker bugs
  • [DONE] Spot check installing nightly builds from nightly.mozilla.org
    • Jason will own testing of this
  • [DONE] Peer Sign-off
    • Jason and Juan will own testing of this

Smoke Test Pass on Nightly

Spot check the Nightly builds to make sure nothing serious shows up incorrectly in the builds. To spot check these builds, we will run the Stub Installer - Nightly Signoff test run in Desktop Firefox 19 for each operating system listed.

Bug Verifications for Blocker Bugs

Verify any bug that blocks ship of the stub installer that is possible to verify by QA.

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Spot Check Installing Nightly Builds from nightly.mozilla.org

Spot check nightly.mozilla.org on Windows to ensure that stub installers are being served and installed correctly.

Peer Sign-off

A peer on the Desktop QA team has to signoff on this test plan to declare signoff on the build.

Aurora

Checklist

  • [DONE] Smoke and basic functional test pass on Aurora
    • [DONE] Windows XP: Juan
    • [DONE] Windows 7: Juan
    • [DONE] Windows Vista: Jason
  • [DONE] Exploratory test pass on Aurora
    • [DONE] Windows XP: Juan
    • [DONE] Windows 7: Juan
    • [DONE] Windows Vista: Jason
  • [DONE] Bug verifications for blocker bugs
  • [DONE] Spot check Aurora builds off of https://www.mozilla.org/en-US/firefox/aurora/
  • [DONE] Peer Sign-off

Smoke and Basic Functional Test Pass on Aurora

Do a smoke and basic functional test pass against the Aurora builds to make sure nothing serious shows up incorrectly in the builds. To check these builds, we will run the Stub Installer - Aurora Signoff test run in Desktop Firefox 19 for each operating system listed.

Exploratory Test Pass on Aurora

Exploratory test the stub installer on aurora by trying the following scenarios:

  • After launching the stub installer, change the screen resolution to a few different values on your machine. Verify that the artwork does not get pixelated or patched.
  • Try launching the stub installer multiple times - make sure only one instance of the stub installer is active
  • Try a pave-over install while firefox is running twice in a row. Then, close firefox and try to re-launch it. Then, you should get a prompt to restart your computer. Upon completing a restart, try launching firefox again. It should launch with no restart required prompt.
  • Try installing firefox with the stub installer at different user account levels with the stub installer.

Bug Verifications on Blocker Bugs

Verify any bug that blocks ship of the stub installer that is possible to verify by QA.

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Spot check Aurora builds off of https://www.mozilla.org/en-US/firefox/aurora/

Spot check downloading and installing stub installer builds from https://www.mozilla.org/en-US/firefox/aurora/ to ensure that the right builds are served and installed correctly.

Peer Sign-off

A peer on the Desktop QA team has to signoff on this test plan to declare signoff on the build.

Aurora 21 Sign-off

Three bugs were checked in to nightly, aurora, and beta which add improvements to the instrumentation of the stub installer. Before we release these on Beta for a study, we need to check that there are no issues with the implementation in Aurora. The approach is to go through some basic smoketests of the stub installer and the Firefox build that is set up on the machine as a result of running the stub installer. In addition to this, the bugs that were fixed for this iteration should be verified. These tests should run across Windows XP, Windows 7, Windows Vista, and Windows 8 (in that order of priority).

Builds: FTP

Smoketests

  • Make sure Firefox installs
  • Make sure the First Run page appears correct
  • Make sure the update channel is correct (check about:config and Help>About Firefox)
  • Make sure the About dialog branding is correct
  • Make sure you can browse to and use some top-sites
  • Make sure the crash reporter works
  • Make sure the Mozilla Maintenance Service gets installed
  • Make sure builds off https://www.mozilla.org/en-US/firefox/aurora/ are working

Basic Functional Tests

Exploratory Tests

  • Make sure artwork does not become pixelated or appear with artifacts when changing screen resolution
  • Make sure only one instance of the stub installer can be run at a time
  • Make sure no restart prompt after multiple pave-over installs with Firefox running
    • with Firefox running, do two pave over installs using the stub installer
    • restart Firefox and restart your computer if prompted
    • start Firefox again
  • Make sure Firefox installs using the stub installer with admin and standard user account levels
  • Make sure installing Firefox is not impacted by AV software running
  • Make sure sleep, hibernate, and lock screen do not impact the Stub Installer while running
  • Make sure a system restore does not impact ability to install and run Firefox
  • Make sure Firefox installs and runs with simulating a network disconnect during installation

Bug Fix Verifications

  • [DONE] VERFIED bug 836044 Aurora stub installer doesn't seem to be working (Paul)
  • STATUS bug 811573 Add more data points to the metrics ping for the stub installer (tester)
  • [DONE] VERIFIED bug 829207 Fix Expires header settings for firefox nightly/aurora (Virgil)
  • STATUS Commented bug 797998 In the "download phase" of the stub installer without an internet connection, the stub installer should eventually time out and report an error (Virgil)
  • STATUS Commented bug 848794 Can't re-run Firefox stub installer after cancelling Install (Virgil)

Results

Platform Smoketests Basic Functional Tests Exploratory Tests
Windows XP en-US PASS (Virgil) PASS (Virgil) PASS (Virgil)
Windows 7 32-bit de FAIL(Mihaela) - testing blocked by bug 849130 FAIL(Mihaela) - testing blocked by bug 849130 FAIL(Mihaela) - testing blocked by bug 849130
Windows 7 32-bit en-US PASS (Mihaela) PASS (Mihaela) PASS (Mihaela)
Windows Vista en-US Fail - Bug 848789 (Simona) PASS (Simona) Fail - Bugs: 849803, 849818 (Simona)
Windows 8 64-bit en-US Pass (Cornel) PASS (Cornel) PASS (Cornel)

Issues

  • NEW
    • bug 849803 [Guest] UAC prompts twice when installing Firefox using the stub installer (Simona)
    • bug 849818 [Vista] OS restart required after pave over install while Firefox running (Simona)
  • [ON TRACK]
    • bug 849233 All the builds installed with the stub installer have the same Build ID (Simona)
    • bug 849199 Flicker when reaching the Install phase (Virgil)
    • bug 731299 Create bouncer links for Aurora downloads - Blocks testing localized stub installers
  • [DONE]
    • bug 849130 Localized Stub-Installer installs en-US build (Mihaela) - Duplicate of bug 731299
    • bug 848794 Can't re-run Firefox stub installer after cancelling Install (Virgil) - Verified fixed
  • [DROPPED]
    • bug 848789 First run page not displayed after installing Aurora (Virgil) - Not a Stub Installer bug

Funnelcake / Stub Installer Beta 20.0b4

Builds

Funnelcake Checklist

  • [DONE] Installation and functionality spot check with Stub Installer (Beta)
    • [DONE] Paul: Windows XP
    • [DONE] Cornel: Windows 7
    • [DONE] Simona: Windows Vista
  • [DONE] Installation and functionality spot check with Funnelcake Full Installer v19 (Beta)
    • [DONE] juanb: Windows XP
    • [DONE] Cornel: Windows 7
    • [DONE] Simona: Windows Vista
  • [DONE] Installation and functionality spot check with Funnelcake Full Installer v20 (Beta)
    • [DONE] juanb: Windows XP
    • [DONE] Cornel: Windows 7
    • [DONE] Simona: Windows Vista

Spot Check Stub Installer (Beta)

Use same smoketest methodology as in Aurora.

Spot Check Funnelcake Full Installer (Beta)

Spot check the funnelcake build for beta for the full installer by checking the following:

  • Make sure that firefox installs
  • Make sure the funnelcake string is correct for the first run page
 http://www.mozilla.org/en-US/firefox/20.0/firstrun/?f=19 
 http://www.mozilla.org/en-US/firefox/20.0/firstrun/?f=20
  • Make sure the about:config update channel is correct
 beta
  • Make sure you can load a website, open a new tab, crash firefox with the crash reporter
  • Make sure in About Firefox dialog contains the right information:
 Funnelcake Mar 2013 mozilla19 - 1.0 
 Funnelcake Mar 2013 mozilla20 - 1.0
  • Make sure partner config preference is set correctly in about:config
 app.partner.mozilla19 = mozilla19 
 app.partner.mozilla20 = mozilla20

Issues

  • NEW
    • bug 850192 Stub installer process stays opened after the install is finished (Simona)


Funnelcake / Stub Installer 20.0.1

Builds

Funnelcake Checklist

  • [DONE] Installation and functionality spot check with Stub Installer
    • [DONE] retornam, juanb: Windows XP
    • [DONE] juanb: Windows 7
    • [DONE] juanb: Windows Vista
  • [DONE] Installation and functionality spot check with Funnelcake Full Installer v21
    • [DONE] juanb: Windows 7
    • [DONE] juanb: Windows XP
  • [DONE] Installation and functionality spot check with Funnelcake Full Installer v22
    • [DONE] juanb: Windows 7
    • [DONE] juanb: Windows Vista

Spot Check Funnelcake Full Installer

Spot check the funnelcake build for the full installer by checking the following:

  • Make sure that firefox installs
  • Make sure the funnelcake string is correct for the first run page
 http://www.mozilla.org/en-US/firefox/20.0.1/firstrun/?f=21 
 http://www.mozilla.org/en-US/firefox/20.0.1/firstrun/?f=22
  • Make sure the about:config update channel is correct
 release
  • Make sure you can load a website, open a new tab, crash firefox with the crash reporter
  • Make sure in About Firefox dialog contains the right information:
 Funnelcake April 2013 mozilla21 - 1.0 
 Funnelcake April 2013 mozilla22 - 1.0
  • Make sure partner config preference is set correctly in about:config
 app.partner.mozilla21 = mozilla21 
 app.partner.mozilla22 = mozilla22

Issues

  • NEW
    • bug 850192 Stub installer process stays opened after the install is finished (tester)
Retrieved from "https://wiki.mozilla.org/index.php?title=Stub_Installer/QA&oldid=651242"