QA/Photon Performance

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
08/11/2017 1.0 Adrian Florinescu Initial Draft
08/17/2017 2.0 Adrian Florinescu RAC/TO updates

Purpose

Overview

Photon is the codename for the Firefox UI refresh on the Firefox front end to be the user facing component of Quantum. Photon Performance is front-end specific performance improvements that complement Quantum work. Summing up, Photon Performance consists of performance improvements and user perceived performance improvements.

Ownership

Engineering lead
  Justin Dolske
Engineering Team
  Florian Quèze
  Mike Conley
QA 
  Adrian Florinescu

References


Testing summary

Scope of Testing

In Scope

  • what's expected from the manual QA point of view is to ensure the Photon Performance changes do not introduce regressions and the areas impacted are covered by exploratory/smoke/regression testing;
  • the depth of testing vary from case to case, based on the FF area that shall be checked;
  • validating user perceived improvements;

Out of Scope

  • validating the actual performance improvements - excepting the visual ones (user perceived improvements)

Environments

Windows 7, 8.1, 10
Ubuntu 16.04
Mac 10.12, 10.10


Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 (Start-up)Captive Portal detection is not done at all or in fashionable time TO-1, TO-2, TO-3 3-High 2-Possible 3-High 18
RAC-2 (Start-up) Captive Portal detection and OS native captive portal agents(MAC/Win) do not integrate TO-1, TO-2, TO-3 2-Moderate 2-Possible 3-High 12
RAC-3 FF cannot be set as default from preferences, at startup or by console command TO-4, To-5, TO-6 3-High 2-Possible 3-High 18
RAC-4 Tab labels are not correctly displayed TO-7, TO-8, TO-9 2-Moderate 2-Possible 3-High 12
RAC-5 Window animation are enabled for FF (win/mac) TO-10, TO-11, TO-12 2-Moderate 2-Possible 2-Medium 8
RAC-6 Firefox scroll is not correctly enabled and handled TO-14 3-High 2-Possible 2-Medium 12
RAC-7 Firefox scrolling is not working smoothly when using trackpad TO-15 2-Moderate 2-Possible 1-Low 4
RAC-8 Firefox scrolling is not working smoothly when using touch TO-15 2-Moderate 2-Possible 1-Low 4
RAC-9 Search panel position is not correctly calculated or remembered on basic scenarios TO-16 3-High 1-Unlikely 2-Medium 6
RAC-10 Search panel position is not correctly calculated or remembered on HiDPI, customized position, FF resizes, drag and drop, multi screen TO-17, TO-18, TO-19, TO-20, To-21 3-High 2-Possible 1-Low 6
RAC-11 Windows size and position not remembered correctly after session restore TO-23 2-Moderate 3-Almost Certain 2-Medium 12

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

TO Main area Sub-areas Additional Details Enviroments TestSuite Coverage (%) RAC
1 Captive Portal basic functionality w-l-m 100% RAC1, RAC2
2 Active/Inactive CP detection w-l-m 100% RAC1, RAC2
3 CP managers(OSX/Win) focused on startup w-m 100% RAC1, RAC2
4 Set FF Default w-l-m 0% RAC3
5 set from preferences / startup w-l-m 0% RAC3
6 set from command line w-l-m 0% RAC3
7 Tab labels behavior tab label display and transitions during operations w-l-m 60% RAC4
8 session restore w-l-m 40% RAC4
9 general tab label transitions w-l-m 60% RAC4
10 Window animation suppression w-l-m 0% RAC5
11 FF open with params OS animation is suppresed w-l-m 0% RAC5
12 FF open from external OS animation is suppresed w-l-m 0% RAC5
13 FF scroll & layout w-l-m 0%
14 FF basic scroll UI not content (tab strip, library, long autocomplete pop-ups -eg. autofill) w-l-m 70% RAC6
15 touch/trackpads scrolling w-l-m 70% RAC7
16 Search panel pozition and size of search panel is recalculated corectly on FF actions w-l-m 0% RAC8
17 Customized position w-l-m 0% RAC10
18 FF resize w-l-m 0% RAC10
19 drag and drop with resize w-l-m 0% RAC10
20 DPI changes w-l-m 0% RAC10
21 HiDPI w-l-m 0% RAC10
22 MousePos flush (1307134 ) status bar is swaped from L to R and other way around w-l-m 0%
23 Restore session - windows size and positions w-l-m 60% RAC11
24 Themes Activate/deactivate window/tab w-l-m 70%
25 Tab switch w-l-m 0%
26 Startup w-l-m 0%

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 03/30/1017
Study documentation/specs received from developers 03/30/1017
QA - Test plan creation 08/11/2017
QA - Test cases/Env preparation -
QA - Nightly Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management [ Bugzilla]

Bug Work

Meta bug: 1348289 - (photon-performance-triage) [meta Photon - Improve perceived UI performance in Firefox]


Full Query
ID Priority Component Assigned to Summary Status Target milestone
669603 -- Session Restore large sessionStorage data causes session restore to block the UI RESOLVED ---
691507 P5 Places [meta] Periodic hangs in the UI due to Places NEW ---
854464 -- Graphics: Layers Florian Quèze [:florian] Incredible lag doing a download with lots of windows open RESOLVED mozilla61
887887 P4 Form Manager Use Sqlite.jsm in Form History NEW ---
906076 P2 Tabbed Browser u462496 Virtual tabs - lazily create linkedBrowser and other dependent elements for tabbrowser tabs to improve startup performance RESOLVED Firefox 55
1054740 P1 Session Restore Dão Gottwald [::dao] When restoring a session, "about:home" is displayed in the selected tab before the tab is restored RESOLVED Firefox 57
1071513 P1 Bookmarks & History Mark Banner (:standard8) Enable async PlacesTransactions in Nightly RESOLVED Firefox 56
1094812 P1 Bookmarks & History Mark Banner (:standard8) Use Bookmarks.jsm in browser-places.js RESOLVED Firefox 57
1095406 P1 Places Marco Bonardo [::mak] Remove getBookmarksForURI and getMostRecentBookmarkForURI from PlacesUtils RESOLVED mozilla60
1095408 P4 Places Remove PlacesUtils.getMostRecentBookmarkForURI RESOLVED ---
1095425 P1 Places Marco Bonardo [::mak] Convert PlacesTransactions to the new Bookmarks.jsm API RESOLVED mozilla56
1095426 P1 Places Mark Banner (:standard8) Convert JSON backups code to the new Bookmarks.jsm API VERIFIED mozilla56
1095427 P1 Places Mark Banner (:standard8) Convert BookmarkHTMLUtils code to the new Bookmarks.jsm API RESOLVED mozilla58
1296507 -- Menus password and control center doorhangers janky when closing under OSX RESOLVED ---
1307134 P1 DOM: Core & HTML David Baron :dbaron: 🇫🇷 ⌚UTC+2 MousePosTracker flushes layout on mouse move (via mozInnerScreenX/Y) VERIFIED mozilla55
1320534 P5 Bookmarks & History Speed and performance of importing bookmarks from JSON or HTML files is awful NEW ---
1336230 -- Widget: Win32 Mike Conley (:mconley) (:⚙️) When a window is maximized, it should not show the window opening transition RESOLVED mozilla54
1337432 -- Widget: Win32 browser_new_content_window_chromeflags.js should be updated to account for renamed CHROME_SUPPRESS_ANIMATION flag RESOLVED mozilla54
1339859 -- Tabbed Browser [meta] Very slow width resize action when multiple tabs are pinned using compact themes RESOLVED Firefox 55
1349742 P1 Tours Florian Quèze [:florian] Investigate UITour's performance impact during window opening and closing RESOLVED Firefox 55
1349761 P1 Tabbed Browser Florian Quèze [:florian] Some code from browser-ctrlTab.js is showing up in profiles of opening browser windows RESOLVED Firefox 55
1351413 P3 General Disable animations/transitions based on Windows Experience Index NEW ---
1352501 P1 Reader Mode Mike Conley (:mconley) (:⚙️) Remove Reader Mode feature promotion panel VERIFIED mozilla55
1352518 P5 Tours Avoid calling getComputedStyle() in isElementVisible() in UITour.jsm NEW ---
1353533 P1 Session Restore Tim Taubert [:ttaubert] (inactive) Creating empty maps while deleting non-existent cookies from SessionCookies.jsm's CookieStore causes jank near startup RESOLVED Firefox 55
1353563 P1 Address Bar Florian Quèze [:florian] Displaying awesomebar items is janky (especially adjustSiteIconStart) RESOLVED Firefox 55
1353708 P1 Autocomplete Marco Bonardo [::mak] richlistbox's ensureElementIsVisible causes a layout flush that contributes to the jank while opening the awesomebar panel RESOLVED mozilla55
1353725 P3 Address Bar displaying the awesomebar panel could be smoother NEW ---
1353744 P3 Widget: Cocoa Incorrect font size while opening a new window on an external screen NEW ---
1354562 P4 Theme Consider moving some <style> elements to separate sheets NEW ---
1354564 -- XUL Neil Deakin Only update commands once per tab switch RESOLVED mozilla55
1354781 -- Tabbed Browser Dão Gottwald [::dao] Filter tab strip scrollbox overflow and underflow events more accurately RESOLVED Firefox 55
1354782 -- Tabbed Browser Dão Gottwald [::dao] Avoid layout flush and invalidation loop in _positionPinnedTabs RESOLVED Firefox 55
1354789 -- Tabbed Browser Dão Gottwald [::dao] Cache layout information needed in _positionPinnedTabs RESOLVED Firefox 55
1354956 -- General Neil Deakin TooltipTextProvider can cause synchronous reflow when computing element direction RESOLVED mozilla55
1355056 P1 General Florian Quèze [:florian] Replace (function(args) { /* do stuff using this */ }).bind(this) with arrow functions RESOLVED Firefox 55
1355861 P4 New Tab Page PageThumbsProtocol.js' newChannel2 implementation is slow RESOLVED ---
1355956 -- General [meta] Photon - Make Firefox start more quickly NEW ---
1355978 P2 Tabbed Browser show tab favicons immediately UNCONFIRMED ---
1356587 P1 Menus Neil Deakin Updating commands takes a lot of CPU time RESOLVED Firefox 55
1356593 P1 Search Florian Quèze [:florian] nsISearchEngine.speculativeConnect and nsISearchEngine.getSubmission are slow RESOLVED Firefox 55
1356655 P1 Tabbed Browser Dão Gottwald [::dao] adjustTabstrip is slow, due to flushing layout RESOLVED Firefox 55
1356663 P1 Tabbed Browser Dão Gottwald [::dao] 3.5ms uninterruptible reflow at _calcMouseTargetRect@chrome://browser/content/tabbrowser.xml:7763:27 when the status text changes RESOLVED Firefox 55
1356862 P4 General 1.2ms uninterruptible reflow at setDefaultFocus@resource://gre/modules/CommonDialog.jsm:246:17 RESOLVED ---
1356911 P1 Toolbars and Customization Dão Gottwald [::dao] CustomizableUI.unregisterBuildWindow is slow when closing windows RESOLVED Firefox 55
1356920 P4 Tabbed Browser Mike Conley (:mconley) (:⚙️) 6.33ms uninterruptible reflow at rect@chrome://browser/content/browser-tabsintitlebar.js:106:23 ASSIGNED ---
1357061 P1 Tabbed Browser Florian Quèze [:florian] 0.98ms uninterruptible reflow caused by underflow event at get_scrollClientRect@chrome://global/content/bindings/scrollbox.xml:131:18 RESOLVED Firefox 55
1357481 P4 General 1.03ms uninterruptible reflow at appendPrompt@chrome://browser/content/browser.js:8426:5 NEW ---
1358173 P1 Toolbars and Customization Florian Quèze [:florian] Removing an "addon-alert" notification that doesn't exist is 8% of _delayedStartup RESOLVED Firefox 55
1358382 P1 Search Florian Quèze [:florian] 1.22ms uninterruptible reflow at onxblpopupshowing@chrome://browser/content/search/search.xml:1087:1 VERIFIED Firefox 55
1358388 P4 Toolbars and Customization :Gijs (he/him) 1.03ms uninterruptible reflow at _moveItemsBackToTheirOrigin@resource:///modules/CustomizableUI.jsm:4203:1 RESOLVED Firefox 61
1358390 P4 Inspector 1.24ms uninterruptible reflow at useLandscapeMode@chrome://devtools/content/inspector/inspector.js:460:27 NEW ---
1358391 P4 Tabbed Browser 2.24ms uninterruptible reflow at handleEvent@chrome://browser/content/tabbrowser.xml:6224:19 RESOLVED ---
1358393 P4 Console 1.64ms uninterruptible reflow at resize@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/client/webconsole/webconsole.js:661:1 RESOLVED ---
1358397 P4 New Tab Page 1.48ms uninterruptible reflow at showLearn@chrome://browser/content/newtab/newTab.js:2613:5 RESOLVED ---
1358408 P4 Tabbed Browser 1.37ms uninterruptible reflow at BrowserOpenAddonsMgr/<@chrome://browser/content/browser.js:6577:7 RESOLVED ---
1358415 P1 Frontend Kris Maglione [:kmag] 1.08ms uninterruptible reflow at get width@chrome://browser/content/ext-utils.js:528:5 RESOLVED mozilla55
1358648 P1 Sync Edouard Oger [:eoger] Weave.js is expensive during early startup due to importing resource://services-sync/util.js RESOLVED Firefox 55
1358695 P4 Find Toolbar 1.43ms uninterruptible reflow at focusContent@resource://gre/modules/RemoteFinder.jsm:169:5 RESOLVED ---
1358719 P3 Bookmarks & History Mike Conley (:mconley) (:⚙️) 1.26ms uninterruptible reflow at PT__updateChevronTimerCallback@chrome://browser/content/places/browserPlacesViews.js:1205:22 RESOLVED Firefox 61
1358722 P4 Toolbars and Customization 1.43ms uninterruptible reflow at updateLWTStyling@resource:///modules/CustomizeMode.jsm:635:23 RESOLVED ---
1358724 P3 Bookmarks & History 1.15ms uninterruptible reflow at updateOverflowStatus@chrome://browser/content/places/browserPlacesViews.js:1163:1 NEW ---
1358728 P1 Find Toolbar 1.14ms uninterruptible reflow at startFind@chrome://global/content/bindings/findbar.xml:1142:13 RESOLVED mozilla55
1358729 P4 Source Editor 1.75ms uninterruptible reflow at onResize@chrome://devtools/content/sourceeditor/codemirror/codemirror.bundle.js:3783:1 NEW ---
1358730 P4 Tabbed Browser 2.41ms uninterruptible reflow at stop@chrome://global/content/bindings/browser.xml:99:13 NEW ---
1358731 P1 Tabbed Browser 0.95ms uninterruptible reflow at removePrompt@chrome://browser/content/browser.js:8515:7 RESOLVED Firefox 55
1358733 P1 Toolbars and Customization 1.09ms uninterruptible reflow at _customizationEnding@chrome://browser/content/browser-customization.js:98:5 RESOLVED Firefox 55
1358809 P1 DOM: Core & HTML Dão Gottwald [::dao] 0.94ms uninterruptible reflow at ssi_getWindowDimension@resource:///modules/sessionstore/SessionStore.jsm:4265:7 VERIFIED mozilla55
1358810 P4 General 0.91ms uninterruptible reflow at create@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/client/framework/toolbox-hosts.js:86:5 NEW ---
1358811 P1 Session Restore 0.91ms uninterruptible reflow at ssi_undoCloseTab@resource:///modules/sessionstore/SessionStore.jsm:2429:5 RESOLVED Firefox 55
1358813 -- Tabbed Browser :Gijs (he/him) 0.92ms uninterruptible reflow at select@chrome://global/content/bindings/textbox.xml:115:11 RESOLVED Firefox 64
1358815 P4 Find Toolbar :Gijs (he/him) 0.90ms uninterruptible reflow at getFindBar@chrome://browser/content/tabbrowser.xml:178:11 RESOLVED mozilla61
1358823 P1 Tabbed Browser 0.92ms uninterruptible reflow at updateBrowserRemoteness@chrome://browser/content/tabbrowser.xml:1795:15 RESOLVED Firefox 55
1358899 P4 JavaScript: GC Opening a link from an external application while Firefox is idle is slow NEW ---
1359111 P1 General Florian Quèze [:florian] BrowserUtils.jsm's makeURI method just adds overhead RESOLVED mozilla55
1359130 P1 Search Florian Quèze [:florian] Remove NetUtil.jsm overhead from the search service initialization RESOLVED Firefox 55
1359408 -- Frontend 0.29ms uninterruptible reflow at _handleDOMChange@chrome://extensions/content/ext-browser-content.js:215:24 NEW ---
1359989 P4 General 1.00ms uninterruptible reflow at openPopupAtScreen@chrome://global/content/bindings/popup.xml:70:15 RESOLVED ---
1360028 P4 General 0.99ms uninterruptible reflow at showNotification@chrome://browser/content/browser-addons.js:412:22 NEW ---
1360043 P1 XUL Widgets 1.57ms uninterruptible reflow at doFocus@chrome://global/content/bindings/popup.xml:334:13 RESOLVED mozilla55
1360078 P1 General 1.13ms uninterruptible reflow at PopupNotifications_dismiss@resource://gre/modules/PopupNotifications.jsm:690:7 RESOLVED Firefox 55
1360279 P1 Address Bar Marco Bonardo [::mak] Typing in the location bar after a keyword causes an icon to flicker VERIFIED Firefox 55
1361087 P1 OS.File Florian Quèze [:florian] Loading osfile_shared_allthreads.jsm does main thread IO on Windows RESOLVED mozilla55
1361115 P4 Tabbed Browser 1.42ms uninterruptible reflow at onResize@chrome://global/content/tabprompts.xml:254:1 NEW ---
1361347 P3 Site Identity and Permission Panels Can refreshIdentityBlock be optimized? NEW ---
1362529 P4 Tabbed Browser 0.97ms uninterruptible reflow at @chrome://browser/content/tab-content.js:85:43 NEW ---
1362533 P3 Bookmarks & History 1.0ms uninterruptible reflow at PT__onDragOver@chrome://browser/content/places/browserPlacesViews.js:1600:11 NEW ---
1362604 -- General 1.58ms uninterruptible reflow at openModalWindow@jar:file:///C:/Program%20Files/Nightly/omni.ja!/components/nsPrompter.js:363:5 RESOLVED ---
1362605 -- Session Restore 0.93ms uninterruptible reflow at restoreHistory@resource:///modules/sessionstore/ContentRestore.jsm:123:5 RESOLVED ---
1362808 P1 Printing 4.64ms uninterruptible reflow at onEntered@chrome://global/content/printUtils.js:669:7 RESOLVED mozilla55
1364127 -- Session Restore Dão Gottwald [::dao] When restoring a tab, don't change its title to "New Tab" and then back to the previous title VERIFIED Firefox 55
1364290 P4 Toolbars and Customization 1.89ms uninterruptible reflow at insertPlaceholder@resource://app/modules/DragPositionManager.jsm:181:7 NEW ---
1364323 -- Sync 0.91ms uninterruptible reflow at waitForSyncCallback@resource://gre/modules/services-common/async.js:98:7 RESOLVED ---
1364324 P4 Toolbars and Customization 0.98ms uninterruptible reflow at AreaPositionManager@resource://app/modules/DragPositionManager.jsm:24:23 NEW ---
1364468 P3 DOM: Push Notifications Push notifications can still be received during shutdown NEW ---
1364725 -- Toolbars and Customization 2.68ms uninterruptible reflow at updateLWTStyling@resource://app/modules/CustomizeMode.jsm:669:23 RESOLVED ---
1365605 P1 Address Bar 1.36ms uninterruptible reflow at _loadURL@chrome://browser/content/urlbarBindings.xml:593:11 RESOLVED Firefox 55
1372599 P1 Widget: Cocoa Josh Aas nsAppShell::Init can wait quite a while calling CGSSetDebugOptions RESOLVED mozilla56
1373694 P3 DOM: Events Don't block on beforeunload events from third-party tracking domain scripts NEW ---
1382012 -- General snippets loading pushes up the search box and nightly logo on about:home RESOLVED ---
1383444 P4 XUL Widgets 3.01ms uninterruptible reflow at nextStep@chrome://global/content/bindings/progressmeter.xml:83:19 NEW ---
1383445 -- Notifications and Alerts 5.14ms uninterruptible reflow at PopupNotifications_showPanel@resource://gre/modules/PopupNotifications.jsm:932:9 RESOLVED ---
1387045 P4 General Suppress window animation for profile manager NEW ---
1387142 P4 General Suppress window animation for New Window NEW ---
1388758 P3 Session Restore Restoring sessions with multiple windows is very slow NEW ---
1446027 -- Frontend Luca Greco [:rpl] 2.00ms uninterruptible reflow at attach@resource:///modules/ExtensionPopups.jsm:488:5 NEW ---
1462373 P3 Menus 1.39ms uninterruptible reflow at _calculateMaxHeight@resource:///modules/PanelMultiView.jsm:1032:9 NEW ---
1462375 -- Add-ons uBlock causes 1.08ms uninterruptible reflow at resize@moz-extension://58e2758a-efad-4ad8-8551-36cb12e93495/js/popup.js:604:1 NEW ---
1462377 -- Bookmarks & History 3.86ms uninterruptible reflow at PT__rebuild/<@chrome://browser/content/places/browserPlacesViews.js:1062:13 RESOLVED ---
1462388 -- Add-ons 1.14ms uninterruptible reflow at draw/<@moz-extension://2a497ae9-5020-43f3-b1bb-83008feef0c6/lib/Chart.min.js:13:742 RESOLVED ---
1482581 P3 Address Bar 1.08ms uninterruptible reflow at _openAutocompletePopup@chrome://browser/content/urlbarBindings.xml:2229:11 NEW ---
1482654 P3 Tabbed Browser 1.43ms uninterruptible reflow at onxblunderflow@chrome://browser/content/tabbrowser.xml:47:11 NEW ---
1482669 P5 Toolbars and Customization 2.25ms uninterruptible reflow at _getDragItemSize@resource:///modules/CustomizeMode.jsm:2216:16 NEW ---
1482670 P3 Tours 1.38ms uninterruptible reflow at recreatePopup@resource:///modules/UITour.jsm:1443:5 NEW ---
1482671 P3 Tours 2.32ms uninterruptible reflow at showHighlightElement@resource:///modules/UITour.jsm:1046:24 NEW ---
1482734 P5 Tabbed Browser 2.09ms uninterruptible reflow at onxbldragover@chrome://browser/content/tabbrowser.xml:1391:9 NEW ---
1482735 P5 Tabbed Browser 3.58ms uninterruptible reflow at _positionPinnedTabs@chrome://browser/content/tabbrowser.xml:532:33 RESOLVED ---

117 Total; 41 Open (35.04%); 69 Resolved (58.97%); 7 Verified (5.98%);


Sign off Criteria

TBD



Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
Lawrence Mandel/Ryan VanderMeulen QA Reviewer Product Integrity Date Email
Florian Quèze Engineering Engineering Date Email
Marco Mucci Product Management Product Management Date Email