Releases/Firefox 34/Test Plan

From MozillaWiki
Jump to navigation Jump to search

Summary

This page is to track testing of Firefox 34 from mozilla-central (Nightly) through to mozilla-release.

QA Dashboard: https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/qa/


How you can help

Would you like to help test Firefox 34? Join us! Here's some things you can do:

Milestones

Milestone Checks Due
Nightly Conduct daily stability and bug triage, ensure features are owned 2014-07-21
Aurora Migration Conduct testing to ensure Firefox 34 Aurora builds are okay 2014-09-01
Aurora Conduct daily stability and bug triage, ensure features are tested 2014-10-13
Beta release date: 2014-12-01

Meetings

Meeting Purpose When Where
Desktop QA Meet the Desktop QA team and hear what they're working on 8:30am PST on Wednesdays Vidyo: DesktopQA, IRC: #qa
QA Team Meeting Meet the entire QA team and hear what they're working on! 1:30am PST on Wednesdays Vidyo: QA, IRC: #qa
Crashkill Raise stability concerns with the Crashkill team 10:00 PST on Monday Vidyo: Stability, IRC: #crashkill
Channel Status Raise quality concerns with the Release Management team 10:00 PST on Tuesday & Thursday Vidyo: ReleaseCoordination, IRC: #planning

Checks

Priority Check Description Frequency Last Checked
1 Stability Review the stability reports to identify new/rising crash signatures Daily 2014-10-13
2 Iteration Triage the iteration bugs, add QA contacts Daily (During Nightly) 2014-09-02
3 QA Needed Bugs Review and assign qawanted bugs for investigation At least twice a week 2014-10-06
4 Unconfirmed Bugs Triage and test unconfirmed bug reports At least once a week 2014-10-06
5 Fixed Bugs Flag fixed bugs for verification At least once a week 2014-10-06
6 Verification Needed Bugs Test verifyme bugs to confirm fixed bugs are fixed At least once a week 2014-10-08
7 Reported Bugs Document and escalated reported bugs as necessary Every Monday 2014-10-06
8 Features Ensure features are owned and on track for release Once a week 2014-10-09
9 Automation Ensure automation bugs are owned and on track for resolution Once a week 2014-10-06
10 Pushlog Review the pushlog to identify fixed bugs needing verification Daily (lower priority) 2014-10-03

Stability

This section tracks bugs representing the most urgent stability issues. Consult this wiki for more information.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1

Notes for followup

Tracking

Tracked for 34 List of bugs being tracked by release managers for Firefox 34. As we approach release, these will either get fixed and may need verification, or will be wontfixed for 34 and pushed back to a future release.

Automation

The following tracks bugs reported from the Mozmill automated tests. For more information consult this wiki.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Features

The following tracks features targeted for Firefox 34. For more information consult this wiki.

  • Ensure all features which do not require testing are whiteboard tagged [qa-]
  • Ensure all features which do require testing are whiteboard tagged [qa+] and have an assigned QA Contact. (new in Aug. 2014: use Flags: qe-verified:+)
  • Ensure all features are either signed-off or disabled before release
  • Check all Core, Firefox, Toolkit bugs with Target Milestone 34, keyword tag "feature": http://mzl.la/1Be6hd1

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


New features for 34

Feature Owner Beta Sign-off Status Potential Blockers
WebIDE Petruta Rasa In progress N/A
App sharing Nils In progress N/A
Loop Anthony In progress N/A
Gaming Anthony/Marc In progress N/A
  • Notification bar to set Fx as your default browser
  • Easily switch themes/personas directly in Customizing mode
  • OMTC needs particular watching.

WebIDE

Forget button

Privacy Tour

Translation

(Bogdan)

Loop

(Anthony) : https://wiki.mozilla.org/Loop/QA

  • Loop will have its button in the toolbar in 34 (so will be more obvious)
  • App sharing is scheduled to go out in Fx34. (bug 1053215)
  • Screensharing and window sharing are scheduled to go out in Fx33.
  • Loop MVP (minimum viable product): https://bugzilla.mozilla.org/show_bug.cgi?id=971986
  • There will be some last minute uplifts in Aurora which we can track separately as we go to beta, since they will have less time to settle in. Test + verify.

Sync

  • New features for Sync (Tracy) (Not yet public)

Ongoing feature work

Mac 10.10

Not a feature, but needs extra testing for 34.

Bug list: https://bugzilla.mozilla.org/buglist.cgi?quicksearch=[10.10]&list_id=11406072

Test updates:

  • Start with a previous-to-latest build of any of the branches (Beta is the top priority for testing) and update Firefox without having upgraded the OS to 10.10.
  • Same, but first upgrade the OS to 10.10 and then upgrade Firefox.

WebIDE

  • Feature owner: Petruta Rasa
  • Status: On Track

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


HTTPS for Wikipedia Search

  • Feature owner: Camelia Badau
  • Status: On Track

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Tabbed Browser

  • Feature owner: Cornel Ionce
  • Status: On Track (Backed out from 33)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Forget Button

  • Feature owner: Bogdan Maris
  • Status: On Track

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Privacy Tour

  • Feature owner: Catalin Varga
  • Status: On Track

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


MacOSX - disable screensaver

  • Feature owner: Catalin Varga
  • Status: On Track (Backed out from 33)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Bug Triage

Pushlog

The following tracks triage of the Mozilla pushlogs. (To triage bug fixes for verification)

  • Beta: https://hg.mozilla.org/releases/mozilla-beta/
  • Review each bug in the pushlog and determine if the bug should be
    • A) Tested by the QA Team --> add the qe-verified:+ flag to the bug and set the status-firefox34 flag to fixed
    • B) Tested by a new volunteer --> add the [good first verify] whiteboard tag
    • C) Tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
    • D) Not tested at all --> add the Flags: qe-verified:+ whiteboard tag
  • If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing
  • Be sure to set the status-firefox34:fixed flag
  • Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures

QA Needed Bugs

The following tracks bugs which have requests for urgent QA assistance.

  • Choose a bug to test from the list below
  • Read it and make sure you understand what information is needed (this may be steps or URLs to reproduce, a regression range, or something else altogether)
  • Use the latest build of this Firefox version and test until you've discovered the information necessary
  • Report this information to the bug and remove the qawanted keyword
  • Document the bug in the Completed list

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Unconfirmed Bugs

The following tracks triage and testing of incoming, unconfirmed bug reports.

  • Choose a bug to test from the list below
  • Use the latest Nightly build to confirm the bug reproduces
  • Flag with reporter with needinfo if you cannot reproduce and need more information
  • Set the status to NEW and ensure the bug is in the appropriate component if you can reproduce the bug (provide as much detail about your testing as possible)
  • If it's a Firefox bug you may want to add the firefox-backlog=? flag to it.

All Unconfirmed Bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Unchanged Since One Week

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Unchanged Since One Month

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Orphaned Fixes

Recent fixes with no target milestone set

https://bugzilla.mozilla.org/buglist.cgi?j_top=OR&list_id=10884420&resolution=FIXED&chfieldto=Now&query_format=advanced&chfieldfrom=-2w&product=Core&product=Firefox&product=Toolkit&target_milestone=---&target_milestone=---&target_milestone=

Recent fixes with status-firefox34 missing.

Testing this query to see if there are problems: https://bugzilla.mozilla.org/buglist.cgi?j_top=OR&f1=cf_status_firefox34&o3=equals&list_id=10884434&v3=---&o1=equals&resolution=FIXED&o2=equals&chfieldto=Now&query_format=advanced&f3=cf_status_firefox32&chfieldfrom=-2w&f2=cf_status_firefox33&v1=---&v2=---&product=Core&product=Firefox&product=Toolkit&target_milestone=Firefox%2034&target_milestone=Firefox%2034

These don't seem to need status-firefox set. They would get picked up in the lists of fixed bugs for verification triage whether or not the status flag is set.

Verification Needed Bugs

  • Review the bugs below in the Pending Triage section and select a bug you want to test
  • Make sure you understand the bug before you begin testing it
  • Test on a previously known broken build to make sure you can reproduce the bug
  • Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
  • Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
  • Reopen the bug if it still reproduces

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Fixed Bugs

The following tracks triage and testing of bugs fixed for this Firefox version.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Reported Bugs

The following tracks bugs reported through day-to-day testing.

  • Once per week, Softvision will send an email identifying their found issues
  • Make sure the issues are documented below
  • Review the NEW issues to make sure they are actionable
  • Actionable bugs can be flagged "firefox-backlog?" according to the backlog triage criteria
  • Any regressions should be nominated to track and be escalated to Release Engineering
  • Review the FIXED issues to make sure they are verified

click [expand] to see this query

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1

Automation

Testrun Reports
Report Test Failures
Add-ons 0 failing tests
Functional bug 999357: /testSessionStore/testUndoTab.js::testUndoTabFromContextMenu
bug 866599: /testPreferences/testPasswordSavedAndDeleted.js::testSaveAndDeletePassword
bug 904610: /restartTests/testAddons_changeTheme/test3.js::testChangedThemeToDefault
Remote bug 1003083 /testSecurity/testSecurityNotification.js::testSecNotification
bug /restartTests/testAddons_InstallAddonWithoutEULA/test2.js::testInstallAddonWithoutEULA
bug 992187 /restartTests/testAddons_InstallAddonWithoutEULA/test1.js::testInstallAddonWithEULA
bug 952034 /restartTests/testAddons_installFromFTP/test2.js::testAddonInstalled
L10n bug 1004827 /testAccessKeys/test1.js::testPrefWindowAccessKeys
Updates 0 failing tests
Endurance 0 failing tests
Memory Usage Report
Platform Min Memory Usage Max Memory Usage Avg Memory Usage Net Compared to Firefox 30
Windows 32-bit 78 MB 99 MB 95 MB +14 MB [17%]
Mac OS X 133 MB 140 MB 135 MB +17 MB [14%]
Linux 32-bit 90 MB 91 MB 90 MB +11 MB [14%]
Linux 64-bit 124 MB 125 MB 125 MB +15 MB [14%]

Memory regression being tracked/investigated in bug 1004608



Mobile

Mobile: Firefox for Android 34 roadmap

Fixed bugs for Firefox for Android:

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Fixed bugs marked tracking-fennec in the last 3 months:

https://bugzilla.mozilla.org/buglist.cgi?list_id=10929374&resolution=FIXED&chfieldto=Now&chfield=cf_blocking_fennec&query_format=advanced&chfieldfrom=-3m&chfieldvalue=%2B

Bugs affected for 34 marked blocking_fennec+

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Automation

The following tracks bugs reported from the Mozmill automated tests. For more information consult this wiki.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1



Features

(To Be Updated 7/22)

Feature Owner Aurora Sign-off Status Potential Blockers
Newtab page Tile Grid Cornel Ionce Ready for Aurora N/A
HTTPS for Yahoo Searches Camelia Badau Ready for Aurora N/A
In-content preferences Camelia Badau Ready for Aurora N/A
Search field in Newtab page Petruta Rasa Ready for Aurora N/A
WebVTT Alexandra Lucinet Ready for Aurora N/A
Plugin whitelist Bogdan Maris ON TRACK N/A

Updates

(To be updated 7-22)

Platform Source Build Target Build Result
Windows Firefox 30.0a en-US 2014-04-06 Firefox 34.0a2 en-US 2014-05-02 PASS
Firefox 30.0a2 eu 2014-04-28 Firefox 34.0a2 eu 2014-05-02 PASS
Mac OS X Firefox 30.0a2 en-US 2014-04-10 Firefox 34.0a2 en-US 2014-05-02 PASS
Firefox 30.0a2 as 2014-04-25 Firefox 34.0a2 as 2014-05-02 PASS
Linux Firefox 30.0a2 en-US 2014-04-01 Firefox 34.0a2 en-US 2014-05-02 PASS
Firefox 34.0a2 zh-TW 2014-04-29 Firefox 34.0a2 zh-TW 2014-05-02 PASS with issues [1]

1. some locales are not updating passed Firefox 34.0a2 2014-05-01

Statistics

Iteration Development

  • 31C-30A-29B.1: 9 bugs verified, 19 bugs qa- (28 bugs total)
  • 31C-30A-29B.2: 17 bugs verified, 28 bugs qa- (46 bugs total)
  • 31C-30A-29B.3: 21 bugs verified, 21 bugs qa- (44 bugs total)
  • 32C-31A-20B.1: 12 bugs verified, 2 bugs qa- (14 bugs total)
  • 34.1: 45 bugs verified, 38 bugs qa- (88 bugs total)
  • 34.2: 24 bugs verified, 42 bugs qa- (109 bugs total)
  • 34.3: 40 bugs verified, (114 bugs total)

Stability

  • 9 / 15 topcrash bugs verified fixed as of 2014-09-02 (Z%)

Reported Bugs

  • 2 / 16 reported bugs verified fixed as of 2014-05-05 (13%)


Beta 1

Read the Beta Coordination Guide for more info Note: betatest should now be beta-localtest, and releasetest is beta-cdntest.

Build2 (20141014134955)

Automated Testing


Manual Testing

Beta 2

Read the Beta Coordination Guide for more info Note: betatest should now be beta-localtest, and releasetest is beta-cdntest.

Build1 (20141020184313)

Automated Testing


Manual Testing
Bugs reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 3

Read the Beta Coordination Guide for more info Note: betatest should now be beta-localtest, and releasetest is beta-cdntest.

Build2 (20141023111813)

Automated Testing
Manual Testing
Bugs reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 4

Read the Beta Coordination Guide for more info Note: betatest should now be beta-localtest, and releasetest is beta-cdntest.

Build1 (several different build ids)

  • linux64: 20141027190746
  • linux32: 20141027190851
  • macosx64: 20141027185351
  • win32: 20141027152126
Automated Testing
Manual Testing
Bugs reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1

Beta 5

Read the Beta Coordination Guide for more info Note: betatest should now be beta-localtest, and releasetest is beta-cdntest.

Build2 (20141030172027)

Automated Testing
Crashes to check

bug 1089413 signature should start falling after beta 5

Manual Testing
Bugs reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 6

Read the Beta Coordination Guide for more info

Build1 (20141103144234)

Automated Testing
Manual Testing
Bugs reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 7

Read the Beta Coordination Guide for more info

Build1 (20141106201515)

Automated Testing


Manual Testing
Bugs reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 8

Read the Beta Coordination Guide for more info

Build1 (20141110195804)

Automated Testing
Manual Testing
Bugs reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 9

Read the Beta Coordination Guide for more info

Build1 (20141113192814)

Automated Testing
Manual Testing
Bugs reported

No new issues.

Build 2
20141114133026

Check build 2, en-US for the lack of the DevEd doorhanger. Check en-US update test.

Beta 10

Build1 (20141117202603)

Automated Testing
Manual Testing
Bugs reported
  • bug 1101427 Street names are not shown in Google Maps on Radeon 3000 devices

Beta 11

gtb will be Thursday evening Nov. 20, release Friday 21st

Build1 (20141120192249)

Automated Testing
Manual Testing
Bugs reported

New Search UI

We have a lot of last minute landings as of Wed Nov. 19, and new bugs related to the new search UI. Here are some specs, questions, and other useful bits of info.

WebQA notes

- the website is being pushed tomorrow [Wed] - we clearly identified which bugs are web-related, and resolved the ones ready to test - Matt and rebecca will go over all of the what's new and firstrun pages today -- this includes all of the different doorhangers, copy changes, yahoo behaviors, etc. - there will be an early meeting Wed. to discuss the web push - ideally it will go out and be ready for Monday's FF release

RC

The plan is to build 34.0 RC and 34.0.5 on Monday the 24th. This may need to be pushed out as late as Wed. Wed is the absolute drop dead date for gtb for RC. We need to take changes by mid week at the latest in order to release on Monday Dec. 1.

We don't know yet which version will be pushed to the beta channel. Whichever version will be on the beta channel should get signed off on first. (We may push 34.0.5 to the beta channel the week of Nov. 24th in order to test it live)

Automated update tests may not work and in that case we need to check en-US builds, and make sure that ru still has yandex and zh locales still have baidu (and haven't been accidentally defaulted to yahoo). Yandex will be the default for other locales like be and kk.

Test plan for new search UI (Kamil): https://etherpad.mozilla.org/34 Bug list for new search UI: https://bugzilla.mozilla.org/buglist.cgi?o5=substring&v5=1101642&f5=blocked&list_id=11596751

RC 34.0

People currently on 33.X will update to 34.0. (This plan may change)

RC 34.0.5

34.0.5 will have yahoo as default search engine in the binaries. We may turn on updates for a very small (1%) user population briefly. Otherwise, 33.X and 34.0 will not update to 34.0.5. (Yet?)

(We may also have an RC 2 this week)


RC1

gtb Nov. 24.

Automation

This is the 34.0 build. We should test updates on beta-cdntest before final sign-off, releasetest after sign-off

(Did not run update tests because we are doing RC2 now)

Manual testing

We should test to make sure that the new search UI functions correctly and consistently. We don't have well defined specs for how it is supposed to behave. UI issues are fine to report but not as much of a priority. Please use (and add to) Kamil's test plan. Test coverage: https://etherpad.mozilla.org/Fx34-0 (also you can try https://etherpad.mozilla.org/34-tests if the first url of test cases doesn't work)

RC1 build 2

This is a second try at the 34.0 build.

Tracking for 34: https://bugzilla.mozilla.org/buglist.cgi?v4=---&f1=cf_tracking_firefox34&o3=equals&v3=affected&j2=OR&o1=equals&f4=cf_status_firefox34&query_format=advanced&f3=cf_status_firefox34&o4=equals&f2=OP&v1=%2B&list_id=11630360


Automation

This is the 34.0 build. We should test updates on beta-cdntest before final sign-off, then beta after conditional signoff and after the push to beta, then on Friday, run tests on releasetest once rel eng pushes there.

Extra Loop tests for RC1 build 2

We need a few extra spot checks for Loop, on RC1 build 2. These can be done tonight (Nov. 26) or Thurs/Fri, on 34.0 which will be on the beta channel by end of day PST nov. 26th. 34.0 and 34.0.5 fresh install tests should be able to be tested as is with the current RC builds on Beta. Anthony will test anything that is left to do on the weekend. 33.0 -> 34 tests will need an update to test. Testing with 33.0rc -> 34.0rc on beta should be sufficient. Please file bugs on the results and note them in the etherpad. If there is anything that looks like a blocker please email lhenry@mozilla.com and ashughes@mozilla.com

Google analytics tests

We need help verifying that the events propagated by Google Analytics are showing up with their correct value-pair in about:healthreport, as outlined here:

34.0.5

This is the build that will only be downloadable. Current Firefox users won't update to it.

Manual testing
Manual testing

We should test to make sure that the new search UI functions correctly and consistently.


34

34.0, release buildid: 20141125180439 34.0.5 release buildid: (20141126041045?)

Post-release update check

Mon. Dec. 8th: We need to test updates in the afternoon Pacific Time.

  • instances that aren't 34.0 should update to 34.0.5.
  • EXCEPT that 34.0 should not update to 34.0.5. (test this on the releasetest channel)
  • test that updates are off (morning) and then once turned on again, updates are enabled at 100%.
    • Updates from a past version to 34.0.5 should be at 100% (afternoon dec. 8)
 ./update.sh 'https://aus3.mozilla.org/update/1/Firefox/33.1.1/20141113143407/WINNT_x86-msvc/en-US/release/update.xml'
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_34/Test_Plan&oldid=1039472"