Performance Triage

From MozillaWiki
Jump to: navigation, search


Idea.png
If you have any feedback/suggestions/questions regarding the performance triage process, you can share them in #perf-triage, or reach out to Dave Hunt or Frank Doty.

Nomination

Bugzilla

To (re)nominate a bug for triage, set the Performance Impact flag in Bugzilla to ?

This can be found by clicking Show Advanced Fields followed by Set bug flags when entering a new bug:

Bugzilla performance nomination on new bug form.png

Or by expanding the Tracking section when editing an existing bug:

Screenshot 2022-02-24 at 19.53.54.png

GitHub

To nominate a bug for triage, add the Performance label to an issue. This can be done by filing an new issue with the "Performance issue" template:

Screenshot of file a "Performance issue" template on GitHub

Or by opening an existing issue on GitHub and selecting the label from the right-hand bar:

Screenshot of adding a performance label on GitHub

Currently, only the following GitHub repositories are supported:

Queries

Performance triage

Full Query
ID Summary Status
1838865 Viewing large PDF causes persistent high CPU and makes Firefox UI unresponsive REOPENED
1866145 Delay opening new windows due to long GC slices in the parent process NEW
1878133 PDF takes excessively long to render NEW
1887129 Slow responses on trivia game in Fenix Nightly vs. Chrome on same device & network connection NEW
1888064 Firefox takes a long time to start prior to Load a Website UNCONFIRMED
1888439 Memory leak on Business Insider UNCONFIRMED
1888525 helpdeskgeek site runs a lot more JS in Firefox than in Chrome (checkerboards during scrolling) NEW
1889461 Ozon.ru eats up RAM UNCONFIRMED
1889648 Fenix restarts itself soon after getting killed by the user NEW
1889722 FirefoxCP Isolated Content Using Almost 10GB RAM UNCONFIRMED
1890537 Firefox nearly froze when I zoomed in in this PDF NEW
1890599 LCP end timestamp doesn't include composition time NEW
1891345 Enourmous RAM usage (60Gig) UNCONFIRMED
1891411 Very high memory usage(95%) by a single website UNCONFIRMED
1891878 I believe that Mozilla Firefox has memory leak issues. UNCONFIRMED
1891950 Android idle battery drain due to Firefox UNCONFIRMED
1891979 Very High Ram usage UNCONFIRMED
1892367 3%-5% regression on multiple AWFY-SP2 benchmark suites around 17Apr2024 (mostly on todo-mvc-Deleting- XXXX type benchmarks) NEW
1892850 Bug Report - Scrolling lags on about:preferences in Firefox Snap (lubuntu 24.04 alpha) UNCONFIRMED
1893080 Uses a lot of ram, literally a lot even for less number of tabs opened UNCONFIRMED
1893263 Mozilla Firefox Freezes When Viewing Web Page Source UNCONFIRMED
1893459 Memory Leak (Isolated Web Container) UNCONFIRMED
1893481 Firefox uses all RAM when run for several days UNCONFIRMED
1893551 Firefox commit growing by 14MiB per second when used while Windows system disk is out of space NEW
1893881 randomly massive memory usage with browser freeze UNCONFIRMED
1893884 Previewing a PDF file from within GitLab's UI is extremely slow, affects interaction with apparent hangs / freezes UNCONFIRMED
1894163 Firefox takes up more than 1 GB of RAM even with only 7 tabs open UNCONFIRMED
1894194 Tab switches to/from tabs with tab-specific infobars cause viewport height to change, resulting in layout jank NEW
1894286 Multiple gigabytes of tiny detached windows UNCONFIRMED
1894695 Large FCP on MacOS in CI when using webdriver since v124.0.2 UNCONFIRMED
1894906 2x performance regression when rendering PDFs with smask NEW
1895113 Crash in [@ OOM | small] NEW
1895131 Intermittent raptor-browsertime Critical: WebDriver session does not exist, or is not active NEW
1895499 Canvas filters on Mac are very slow, which badly affects rendering PDFs with smask NEW
1895646 Expanding the bookmarks bar will cause page freezes and reduce FPS UNCONFIRMED
1896308 Firefox 125 window frozes on Win11 UNCONFIRMED
1896500 Memory Leak on newest version. 14 task manager occurrences from one tab open using over a gig of memory. Laptop is overheating and only fix is reset computer. UNCONFIRMED
1897637 Firefox consumes 65 GB RAM UNCONFIRMED
1897881 Firefox is having Memory Leaks since v126 UNCONFIRMED
1898645 FIrefox Uses more RAM the longer which it stays on regarless of activity. UNCONFIRMED

40 Total; 40 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Performance triage (pending-needinfo)

Full Query
ID Summary Status
1606631 Inefficient z-ordering of elements in the browser UI, causes large layers with current WebRender heuristics ASSIGNED
1805255 Firefox has 50% more soft page faults than Chrome running Speedometer 2 NEW
1870532 Firefox 120.0.1 grabs 100% of cpu resources and freezes UNCONFIRMED
1872368 Since the last update 121.0 Firefox uses insane amount of Ram 99% of my 32GB without any real reason and if it gets closed it buges out for 10 seconds spikes over the ram limit just to slowly close UNCONFIRMED
1872414 Firefox lags after latest update, high memory use (121.0 12-2023) UNCONFIRMED
1875547 Slow (single-threaded) PDF rendering for documents with lots of vector shapes NEW
1876708 firefox using 3.4gb of memory with a few tabs open UNCONFIRMED
1877007 extremely high CPU usage even with one tab open on the splash screen UNCONFIRMED
1879177 3 youtube tabs are using 7GB of RAM UNCONFIRMED
1879905 firefox consuming 2 GB of RAM with 8 tabs open UNCONFIRMED
1882916 Firefox consumes a lot of CPU and makes me Mac M2 hang for a good 15 seconds UNCONFIRMED
1882952 Firefox Browser Freezes when Opening Multiple Tabs UNCONFIRMED
1883246 firefox using large amount of memory UNCONFIRMED
1883475 huge amount of memory storage (constantly) idk why pls help/................................................................................................................................................. UNCONFIRMED
1883507 Memory usage is too high 23GB and up to 60% CPU usage. UNCONFIRMED
1883669 Memory Problem UNCONFIRMED
1884296 Firefox is using lots of memory when only having several tabs open UNCONFIRMED
1885796 Use to many Ram UNCONFIRMED

18 Total; 18 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Recently opened bugs with performance keywords in the summary

Full Query
ID Summary Status
1896305 Add performance status arrows to CTR values NEW
1896319 Nightly is 1.8x slower than Chrome on a Codepen demo with css-doodle.js (https://codepen.io/aragakey/pen/eYaObKJ) NEW
1896709 Implement on-memory cache of JS::Stencil based on SharedSubResourceCache ASSIGNED
1896722 Android Firefox, pdfs slow down and freeze as I scroll down, requiring browser restart UNCONFIRMED
1896729 15.35 - 9.97% perf_reftest_singletons external-string-pass.html / perf_reftest_singletons external-string-pass.html (OSX) regression on Thu May 9 2024 NEW
1896843 GPURenderPassEncoder: setBindGroup() argument 2 should be nullable UNCONFIRMED
1896864 Attempt to override an existing message - "select-translations-panel-translating-placeholder-text" from preview/select-translations.ftl NEW
1896952 System gets slammed with all cores at 100% initializing hand gesture detection on macOS NEW
1897054 CPU usage spikes when opening links in a new tab causes unresponsive touchpad scrolling UNCONFIRMED
1897097 Error message flashes briefly on quora.com after performing a search using the “Enter” key NEW
1897104 Consider adding idle-time memory checking NEW
1897105 Scan read-only memory for badram NEW
1897131 Determine why `webgpu:shader,validation,expression,unary,arithmetic_negation:scalar_vector:*` isn't `FAIL`ing for `f16` variants on non-MacOS platforms ASSIGNED
1897254 JS demo (https://www.9patch.online/editor/) is 1.8x slower in Nightly NEW
1897294 [REGRESSION] [LINUX] 100% CPU use with zero web pages open UNCONFIRMED
1897302 60 fps YouTube video is jankier in Firefox than in Chrome NEW
1897518 Add perf test/tests for float clearance change in bug 1874079 NEW
1897785 Update API on `TranslationsAction.OperationRequestedAction` To Accept an Optional TabID ASSIGNED
1897843 HLS OutOfMemory Crash in [@ java.lang.OutOfMemoryError: at java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java) ] NEW
1897868 Crashes when I want to translate text through the Android context menu UNCONFIRMED
1897892 Firefox headless is slow under Apple Silicon Virtualization UNCONFIRMED
1898039 Consider on-demand mapping of stack memory for lazy growing NEW
1898040 Excessive thread creation during Google Meet call (maybe excessive CPU too) NEW
1898164 Display problems with Beta 127 on a Windows ARM PC (Adreno 685 GPU) UNCONFIRMED
1898191 Improving HTTPS RR Handling to Mitigate Performance Regression NEW
1898245 Migrate HTTP3_PERF_FIRST_CONTENTFUL_PAINT_MS, H3P_PERF_FIRST_CONTENTFUL_PAINT_MS, and EH_PERF_FIRST_CONTENTFUL_PAINT_MS probes to glean ASSIGNED
1898319 firefox slows down, needs relaunch to restore throughput UNCONFIRMED
1898327 Google Meet "instant meeting"s sometimes pause at "Loading..." NEW
1898340 Vulkan validation layer not available in WebGPU CI NEW
1898440 Fenix opening itself and pulling 100% CPU in the background UNCONFIRMED
1898520 Update AC `isTranslateProcessing` NEW
1898545 Bryntum Components are 3.5x+ slower in Firefox than Chrome. NEW
1898567 Video and audio synced perfectly initially, but become VERY slightly desynced for the rest of the video unless paused again UNCONFIRMED
1898597 High CPU cost during scrolling from onScrollChange notifications NEW
1898647 Seperate the cache of unallocated memory into: dirty, clean, unmapped. NEW
1898708 Enable DisableExtensionPoints mitigation on GPU process UNCONFIRMED
1898757 Firefox Translate seems miss-lead by datetime timestamps. NEW
1898785 Staging folder compression step with nsIZipWriter janks the parent process main thread NEW
1898861 WebGPU crash - failed to create read-only memfd: Too many open files UNCONFIRMED

39 Total; 39 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Triage process

Introduction

The goal of performance triage is to identify the extent to which bugs impact the performance of our products, and to move these bugs towards an actionable state. The goal is not to diagnose or fix bugs during triage. We triage bugs that have been nominated for triage and bugs in the Core::Performance component that do not have the performance impact project flag set.

During triage we may do any/all of the following:

  • Request further information from the reporter (such as a profile)
  • Set the performance impact project flag
  • Add performance keywords
  • Move the bug to a more appropriate component

Who is responsible for triage?

Everyone is welcome to take part in triage. By default, everyone on the performance team is enrolled in triage rotation, but we also have participants from outside the team.

How do I schedule a triage meeting?

If you are on triage duty, you will receive an invitation as a reminder to schedule the triage meeting on the shared performance calendar with the nominated sheriffs invited at a time that works for them. The responsibility of scheduling the meeting falls to the lead sheriff. Once a triage meeting has been scheduled, it’s a good idea to remove the reminder event from the calendar to avoid confusion. It’s a good idea to use the shared calendar, as this increases the visibility of the performance triage and allows other members of the team to contribute or observe the process.

What if a sheriff is unavailable?

The rotation script is not perfect, and doesn’t know when people are on PTO or otherwise unavailable. If the lead sheriff is available, it is their responsibility to either schedule the triage with the remaining available sheriff or to identify a suitable substitute for the unavailable sheriff(s). If the lead sheriff is unavailable, this responsibility passes onto the remaining available sheriffs.

How do I run a triage meeting?

The following describes the triage process to follow during the meeting:

  1. Ask if others would prefer you to share your screen. This can be especially helpful for those new to triage.
  2. Open the first triage query to show bugs nominated for triage or in the Core::Performance component without the performance impact project flag set. The bugs are sorted from oldest to newest. For each bug in the list, follow these steps:
    • Bugs that look like tasks that were filed by members of the Performance team will generally need to be moved to the Core::Performance Engineering component.
    • For defects: Determine if the bug is reproducible and actionable. If not, add a needinfo for the reporter asking for more information, set the performance impact project flag to pending-needinfo, and then move onto the next bug. We have a template that you can modify as needed.
    • For all bugs (including enhancements):
  3. Open the second triage query to show bugs that are waiting further information to determine the performance impact. The bugs are sorted from oldest to newest. For each bug in the list, follow these steps:
    • If the performance impact project flag was set to pending-needinfo less than 2 weeks ago, move onto the next bug.
    • If the performance impact project flag was set to pending-needinfo more than 2 weeks ago but less than 2 months ago, consider adding a needinfo for either: another reporter of the issue, someone with access to the appropriate platform(s) to attempt to reproduce the issue, or a relevant subject matter expert.
    • If the performance impact project flag was set to pending-needinfo more than 2 months ago, close the bug as inactive. You can modify the inactive bug template as needed.
  4. If time permits, open the third triage query to show recently opened bugs with performance related keywords in the summary. If any of these look like performance bugs, they can either be triaged the same way as bugs in the initial query or they can be nominated for triage in a subsequent meeting.

What if things don't go as expected?

Don't panic! The triage process is not expected to be perfect, and can improve with your feedback. Maybe the result of the triage calculator doesn't feel right, or you find a scenario that's not covered in these guidelines. In this case we recommend that you bring it up in #perf-triage, or consider scheduling a short meeting with some triage leads (you can see some recent leads in the triage rotation). If in doubt, leave a comment on the bug with your thoughts and move on. There's a chance someone will respond, but if not the next performance triage sheriffs may have some other ideas.

How do I determine the performance impact project flag?

The performance impact project flag is used to indicate a bug’s relationship to the performance of our products. It can be applied to all bugs, and not only defects. The triage calculator should be used to help determine the most appropriate value for this flag. In addition to setting the performance impact project flag, make sure to use the “Copy Bugzilla Comment” button and paste this as a comment on the bug.

If you do not have enough information to set the performance impact project flag, open a needinfo request against an appropriate individual (such as a reporter), and set the performance impact project flag to pending-needinfo.

For more information about what this flag, and it's settings mean see this blog post.

How do I determine the performance keywords?

There are several performance related keywords, which can be helpful to understand how our performance issues are distributed, or whenever there’s a concerted effort to improve a particular aspect of our products. The triage calculator may recommend keywords to set, and by typing “perf:” in the keywords field in Bugzilla, you will see the available options. Select all that apply to the bug.

How do I determine the correct Bugzilla component?

Ideally we would only have bugs in the Core::Performance component that are the responsibility of the engineers in the performance team. For performance bugs to have the best chance of being fixed, it's important to assign them to the correct component. In some cases the correct component will be obvious from the bug summary, description, or steps to reproduce. In other cases, you may need to do a bit more work to identify the component. For example, if there's a profile associated with the bug, you could see where the majority of time is being spent using the category annotations.

How do I read a performance profile?

It's useful to be able to understand a profile generated by the Firefox Profiler, and hopefully someone in the triage meeting will be able to help. If you find an interesting profile, or just want to understand how to use them to analyse a performance problem, we encourage you to post a link to the profile (or bug) in #joy-of-profiling where someone will be happy to help. The profile may even be analysed during one of the regular "Joy of Profiling" open sessions that can be found on the Performance Office Hours calendar.

Triage calculator

The Performance Impact Calculator was developed to assist in identifying and applying the performance impact project flag and performance keywords consistently. If you have feedback or would like to suggest changes to this tool, please share these in the #perf-triage Matrix channel.

Triage rotation

The sheriffs are allocated on a weekly basis, which is published here. The rotation is generated by this script.

Templates

New bug

This template is included in the description for new bugs opened in the Core::Performance component. If a bug is opened in another component and then moved to Core::Performance, this template can be used as needed to request additional information from the reporter.

### Basic information

Steps to Reproduce:


Expected Results:


Actual Results:


---

### Performance recording (profile)

Profile URL:
(If this report is about slow performance or high CPU usage, please capture a performance profile by following the instructions at https://profiler.firefox.com/. Then upload the profile and insert the link here.)

#### System configuration:

OS version:
GPU model:
Number of cores: 
Amount of memory (RAM): 

### More information

Please consider attaching the following information after filing this bug, if relevant:

 - Screenshot / screen recording
 - Anonymized about:memory dump, for issues with memory usage
 - Troubleshooting information: Go to about:support, click "Copy text to clipboard", paste it to a file, save it, and attach the file here.

---

Thanks so much for your help.

Moved to Core::Performance

This bug was moved into the Performance component. Reporter, could you make sure the following information is on this bug?

 - For slowness or high CPU usage, capture a profile with http://profiler.firefox.com/ , upload it and share the link here.
 - For memory usage issues, capture a memory dump from about:memory and attach it to this bug.
 - Troubleshooting information: Go to about:support, click "Copy raw data to clipboard", paste it into a file, save it, and attach the file here.

Thank you.

No longer able to reproduce

This bug doesn’t seem to happen anymore in current versions of Firefox. Please reopen or file a new bug if you see it again.

No response from reporter

With no answer from the reporter, we don’t have enough data to reproduce and/or fix this issue. Please reopen or file a new bug with more information if you see it again.

Expected behaviour

This is expected behavior. Please reopen or file a new bug if you think otherwise.

Website issue

According to the investigation, this is a website issue. Please reopen or file a new bug if you think otherwise.