Releases/Firefox 38/Test Plan/Beta/1

From MozillaWiki
Jump to: navigation, search

«  Firefox 37 | Firefox 38 Beta 1 | Firefox 38 Beta 2  »

Firefox 38 Beta 1 (build 1) Test Plan

QE recommendation regarding the quality of the build

Recommendation based on results from manual testing GOOD FOR RELEASE

Risks with this Beta and ways to mitigate them through testing

Please list any risks related to the quality of this release.

  • 1. Disable Dev edition features in 38 beta 1 build
    • Testing will verify and validate that NO DevEdition features are carried from Aurora
    • Will be covered by basic sanity testing
  • 2. Reader View
    • New feature targeted to release in Firefox 38
    • Testing the feature to verify that it's stable enough for Beta
    • Will be covered by regression testing
  • 3. Reading List
    • New feature targeted to release in Firefox 38
    • Testing the feature to verify that it's stable enough for Beta
    • Will be covered by regression testing
  • 4. Hello with screensharing
    • New feature targeted to release in Firefox 38
    • Testing the feature to verify that it's stable enough for Beta
    • Will be covered by regression testing

Features

See the Feature Release Tracking wiki for a general overview


Feature Owner Status Details Test Plan
Firefox Accounts for Sync users Catalin Varga [ON TRACK] Follow up on additional fixes for Firefox Accounts for Sync users -
Reading List Andrei Vaida [ON TRACK] New feature in Firefox 38 - will be disabled mid-beta Test Plan
Reader View Andrei Vaida [ON TRACK] New feature in Firefox 38 - will be disabled mid-beta Test Plan
Hello screensharing Bogdan Maris [ON TRACK] New feature in Firefox 38 - will be disabled mid-beta Test Plan
In-content preferences Camelia Badau [ON TRACK] New feature since many previous versions - final QA sign-off at the end of Beta 38 Test Plan
Suggested tiles Cornel Ionce [ON TRACK] New feature late in Firefox 38 - final QA sign-off at the end of Beta 38 -
EME Bogdan Maris [ON TRACK] Constantly monitoring for new issues and Verification of important fixes - NO final QA sign-off -
MSE for YoutTube Bogdan Maris [ON TRACK] Constantly monitoring for new issues and Verification of important fixes - NO final QA sign-off -

Results

Bug Summary

  • New bugs
Full Query
ID Summary Status Resolution
1149902 The Reading List's sidebar is sometimes enabled automatically in newly opened private windows RESOLVED WORKSFORME
1150388 Some items from Reading List are doubled after the automated syncing with Firefox Account RESOLVED INVALID
1150422 User should not be able to share the Firefox Sharing Indicator RESOLVED DUPLICATE
1150425 The shared window should be surrounded by a focus light RESOLVED INVALID

4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);

  • New bugs to this Beta
Full Query
ID Summary Status Resolution
1149902 The Reading List's sidebar is sometimes enabled automatically in newly opened private windows RESOLVED WORKSFORME
1150388 Some items from Reading List are doubled after the automated syncing with Firefox Account RESOLVED INVALID
1150422 User should not be able to share the Firefox Sharing Indicator RESOLVED DUPLICATE
1150425 The shared window should be surrounded by a focus light RESOLVED INVALID

4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);

  • Known Issues
Full Query
ID Summary Status Resolution
1089454 crash in DrawingContext::FillRectangle(D2D_RECT_F const*, ID2D1Brush*) RESOLVED FIXED
1090518 Crash during WebGL Conformance Tests - /extensions/webgl-depth-texture.html VERIFIED FIXED
1123519 [ReadingList] Allow adding items to the Reading List via the sidebar RESOLVED INCOMPLETE
1131460 [ReadingList] Add-to-readinglist button in the URLbar should have an animation similar to the bookmarking animation RESOLVED INCOMPLETE
1132656 Reader mode toolbar overlaps content if window becomes too narrow. VERIFIED FIXED
1135009 Printing in reader mode cuts off on the left (sidebar overlays text) VERIFIED FIXED
1135587 Reader Mode sidebar is invisible in high contrast Windows themes RESOLVED WORKSFORME
1135589 The focusring for the controls in reader mode doesn't match the visible borders of the buttons VERIFIED FIXED
1136265 Reading mode desktop images shouldn't be upscaled beyond their _originalWidth, even if they take more than half of the screen. RESOLVED WORKSFORME
1137089 The sidebar header is white on the developer edition theme VERIFIED FIXED
1137211 Font panel in reading view should disappear when clicking on the margins VERIFIED FIXED
1137594 Wrong context menu present for shared screen item and webcam window RESOLVED WORKSFORME
1137597 Video controls will not disappear if I unfocus the video RESOLVED INVALID
1137598 Reversed controls for videos that have a css transform flipping the video (no attribute/option to flip video output) RESOLVED INACTIVE
1137599 Transparency issue in Conversation window RESOLVED INCOMPLETE
1137602 Windows only - Global indicator does not call any pop-ups from Firefox RESOLVED INCOMPLETE
1137603 WebRTC sharing notifications fail to open from the global indicator when the Hello window has been detached VERIFIED FIXED
1137606 Background image from screen sharing while the webcam window of the person who is receiving the call is in fullscreen RESOLVED WORKSFORME
1142298 Fix reader view font/color control glitches VERIFIED FIXED
1143689 Reading List items with a title that can't fit on 2 lines should be cropped with an ellipsis RESOLVED INVALID
1144691 Windows displays a blue glow hover effect for the Reading List button in the Location Bar RESOLVED INVALID
1146348 PageMetadata poorly chooses a preview image when none is specified by the page RESOLVED INVALID
1146373 Images in reader view are sometimes too small VERIFIED FIXED
1146882 Starting tab sharing before the call will not show the tab when someone joins conversation RESOLVED WONTFIX
1146919 Share screen info bar remains visible in fullscreen RESOLVED INVALID
1147429 Missing partner from Hello doorhanger after browser restart RESOLVED DUPLICATE
1147571 Inconsistent thumbnail generation depending on whether you're in reader mode RESOLVED DUPLICATE
1148462 When "Reading List" is disabled (browser.readinglist.enabled = false) CTRL+ALT+R should not open its sidebar. VERIFIED FIXED
1148562 If I right-click the Reader Mode button (in URL bar), the page enters Reader Mode VERIFIED FIXED
1148953 Reading list should remember whether a page was in reader mode when added RESOLVED INCOMPLETE
1149023 unable to remove reading list item VERIFIED FIXED
1149136 Incorrect hover styling displayed for the article remove button available in the Reading List sidebar VERIFIED FIXED
1149144 Page preview images (GIFs) shouldn't animate in the Reading List sidebar RESOLVED WORKSFORME

33 Total; 0 Open (0%); 20 Resolved (60.61%); 13 Verified (39.39%);


Regression Tests

  • Tests are run from a Full Regression set, created before Beta 2, to cover areas most affected by fixes in current Beta (Firefox 38)
  • Regression tests will be run over the interval Beta 2 -> Release
  • Tests to run for each Beta are picked to cover areas most affected by fixes since previous Beta (if there are no such areas, tests are picked randomly from the Full Beta Regression set).
  • Tests will be available in Mozilla Moztrap.
  • In any case of failure in any regression tests, bugs need to be raised ASAP before proceeding to the exploratory testing.

Regression testing areas (final list to be finalized before Beta 2):

  • Desktop Firefox Add-on Compatibility
  • Breakpad End-to-End Functionality
  • Desktop Firefox Plugin Compatibility
  • Desktop Firefox Web Compatibility
  • Mac OSX Signing Verification
  • Verify update applies with root owned files
  • Mac OSX 32-bit Mode

Exploratory Testing

  • Testing is performed around areas most affected by fixes since the previous Beta (in cases when there are no Regression tests to cover these areas, or those tests have already been run as part of the Full Beta Regression set)
Platform Assignee Result: report
Windows 7 64-bit (GPU: ATI Radeon 3000) Andrei Vaida [DONE]
Windows 8.1 32-bit (GPU: AMD Radeon HD 7700) Vasilica Mihasca [DONE]
Mac OS X 10.9.5 (GPU: Intel Iris Pro 1024 MB) Cornel Ionce [DONE]
Ubuntu 14.04 64-bit (GPU: AMD Radeon HD 6450) Bogdan Maris [DONE]
Tested Area Testing Type Results (report)
Disable Dev edition features Basic Sanity testing PASS
Reader View Regression testing CONDITIONAL PASS
Reading List Regression testing CONDITIONAL PASS
Hello with screensharing Regression testing CONDITIONAL PASS

Branch Bug Fix Verifications

  • Verified bugs
Full Query
ID Summary Status Resolution
1053783 New window gets out of focus after exiting full screen mode VERIFIED FIXED
1123974 geoIP request should not override existing isUS cached values VERIFIED FIXED
1129597 Unable to edit keyword VERIFIED FIXED
1131584 Add option to screen share dropdown to share tabs VERIFIED FIXED
1146565 [EME] EME-free Firefox fails to uninstall any pre-existing CDMs VERIFIED FIXED
1146883 nightly crashes when deleting quoted html elements in a gmail reply VERIFIED FIXED
1146938 Prepare backout patch for screensharing in Fx38 VERIFIED FIXED

7 Total; 0 Open (0%); 0 Resolved (0%); 7 Verified (100%);


Status tags

  • [NOT STARTED] - {{mnotstarted}} - little or no work has started on this task
  • [ON TRACK] - {{mok}} - task is actively being worked on and those assigned feel comfortable and confident in the quality of the build.
  • [DONE] - {{mdone}} - task is completed and those assigned feel comfortable and confident in the quality of the build.
  • [AT RISK] - {{mrisk}}- blocking bug / high number of bugs / other confidence-shakers makes us feel not comfortable with the quality of the build.
  • [IN PROGRESS] - {{mprog}} - actively being tested on
  • [INVESTIGATION NEEDED] - {{minvest}} - more investigation needed