Performance Triage

From MozillaWiki
Jump to: navigation, search


Idea.png
If you have any feedback/suggestions/questions regarding the performance triage process, you can share them in #perf-triage, or reach out to Dave Hunt or Frank Doty.

Nomination

Bugzilla

To (re)nominate a bug for triage, set the Performance Impact flag in Bugzilla to ?

This can be found by clicking Show Advanced Fields followed by Set bug flags when entering a new bug:

Bugzilla performance nomination on new bug form.png

Or by expanding the Tracking section when editing an existing bug:

Screenshot 2022-02-24 at 19.53.54.png

GitHub

To nominate a bug for triage, add the Performance label to an issue. This can be done by filing an new issue with the "Performance issue" template:

Screenshot of file a "Performance issue" template on GitHub

Or by opening an existing issue on GitHub and selecting the label from the right-hand bar:

Screenshot of adding a performance label on GitHub

Currently, only the following GitHub repositories are supported:

Queries

Performance triage

Full Query
ID Summary Status
1811750 Performance.mark/measure takes a long time to execute REOPENED
1834526 Large fetch in Web Worker in extension causes hang in Firefox extension process NEW
1850822 Firefox, with only one tab is taking up a lot of RAM is OSX UNCONFIRMED
1852121 Memory Leak UNCONFIRMED
1854623 Incredibly high GPU RAM Usage when Hardware Accel is on UNCONFIRMED
1854657 Firefox Leaking memory while using ChatGPT and or Netflix UNCONFIRMED
1855804 RAM eater UNCONFIRMED
1855871 Delay when using search bar before loading occurs, likely caused by Kaspersky UNCONFIRMED
1856617 terrible performance on https://arc.net/ UNCONFIRMED
1857832 excessive memory usage UNCONFIRMED
1858026 A memory leak has occured in Firefox for Windows UNCONFIRMED
1858535 Speedometer 3 score is ~1 point lower than Chrome on Windows 11 Surface Pro 7+ device, when on battery power NEW
1860663 Firefox sometimes freezes with constant CPU usage when bringing up the Back history list UNCONFIRMED
1860895 High Memory Usage (up to 1050 MB) UNCONFIRMED
1861717 Browser Loads Slowly After Update UNCONFIRMED
1861810 Firefox locks up often when trying to load a new page. UNCONFIRMED
1861822 Firefox takes a while to cold boot UNCONFIRMED
1861856 [Linux] Native Wayland loads the DOM A LOT slower than XWayland? UNCONFIRMED
1861870 Memory leak in firefox UNCONFIRMED
1861991 Bad mutex contention when allocating strings in JSON.parse() in TodoMVC-Javascript-ES6-Webpack on Android NEW
1862741 nsHtml5Tokenizer::appendStrBuf isn't inlined on Android NEW
1863557 GPU process is disabled for android PGO profile generation NEW
1863839 High ram usage NEW
1864178 LCP shows up in sp3 profiles NEW
1864322 100% CPU usage on parent process after resume from hibernate NEW
1864370 Three minutes to start rendering tabs after automatic update UNCONFIRMED
1865437 Firefox frequently hangs and sometimes does not recover NEW
1865886 Use aarch64/arm32 builds for PGO profile generation on Android NEW
1866034 High memory usage when running Firefox UNCONFIRMED
1866066 Firefox freezes completely at irregular intervals UNCONFIRMED
1866145 Half-second to one-second delay on opening new windows NEW
1867074 60% CPU usage of content process because of memory leak in gc-heap on Amazon product page NEW
1867361 Experiment with implementing the js self profiling API NEW
1867468 My firefox takes a long time to boot with hardware acceleration on UNCONFIRMED
1867581 When run through profileserver.py, speedometer3 hangs at 57/610 test case and hits 404 NEW
1867784 DoorDash site fails to load. Last working in version 104 NEW
1868326 Use aarch64 instrumented build for PGO profile generation for aarch64 macos NEW
1868643 Memory leak and high CPU consumption at marketwatch.com NEW
1868885 High Ram Usage UNCONFIRMED

39 Total; 39 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Performance triage (pending-needinfo)

Full Query
ID Summary Status
1805255 Firefox has 50% more soft page faults than Chrome running Speedometer 2 NEW

1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Recently opened bugs with performance keywords in the summary

Full Query
ID Summary Status
1866460 crash wgpu_core::track::metadata::ResourceMetadata<A,I,T>::tracker_assert_in_bounds NEW
1866499 camelCase() is 2.8x faster in V8 than SM NEW
1866606 TaggedAnonymousMemory can support all Linux ASSIGNED
1866677 Erroneous translations break paypal UNCONFIRMED
1866710 clicking on the address bar janks the browser for 500ms if you have copied large text outside of the browser NEW
1866823 Add option to "Translate Page to [Language]..." inside the page context menu NEW
1866827 Firefox translation feature should remember the last destination language NEW
1866853 Translations UI Migrate static_strings.xml to strings.xml for Core Translation Flows NEW
1866997 URL status bar persists in normal/theather mode NEW
1867118 Slack is frozen/unresponsive when returning from sleep mode NEW
1867191 Set the PHC memory area size from a pref ASSIGNED
1867249 View Source is slow on specific markdown page on Github NEW
1867278 m.youtube.com - Performance issues causing drop in FPS NEW
1867283 Firefox is 2x slower on an online PNG optimiser NEW
1867291 Firefox is 1.7x slower than Chrome on an online tool and spends tons of time in Baseline (time spent in array:splice?) NEW
1867359 React's getElem with internalInstanceKey is 3x slower in SM (150ms) than V8 (50ms) and JSC (37ms) NEW
1867574 A webgpu tab in the foreground will crash if you manually do "Terminate GPU process" NEW
1867624 Massive main process jank when some spanish websites are used and you have the DNIe card reader software installed NEW
1867734 Enable Raptor performance tests for IndexedDB on Safari NEW
1867787 Translation does not translate article title UNCONFIRMED
1867791 Firefox WebGL performance is poor *UNLESS* backdrop-filter: blur is used UNCONFIRMED
1867937 Firefox launch extremely slow on my Linux system recently UNCONFIRMED
1868045 Clicking on tabs doesn't work/switches slowly UNCONFIRMED
1868105 Translations UI Refactor ListItem to support Radio Button use case. NEW
1868219 GeckoView Update to fit Change in Behavior of `TranslationsParent:LanguageState` Event NEW
1868239 <https://webgpufundamentals.org/webgpu/lessons/webgpu-fundamentals.html> fails to render red triangle NEW
1868285 Nightly is slower than Chrome on a Codepen demo (https://codepen.io/ccprog/pen/rNPoNxP?editors=0010) NEW
1868349 Zooming on netz24.vvs.de is janky NEW
1868460 [wpt-sync] Sync PR 43527 - Fix memory leak in mousemove_prevent_default_action. NEW
1868523 slow startup with system themes UNCONFIRMED
1868601 Enable the entry trampolines and IC frame pointers only when IONPERF is set NEW
1868641 Ctrl+A selecting text in Slack's "New message" "To:" field and typing appends next characters instead of replacing the selected text ASSIGNED
1868672 wgpu update (December 7th) ASSIGNED
1868679 Add a pref for browser.translations.select.enable NEW
1868689 Slow `CanvasRenderingContext2D.drawImage()` with SVGs in a large rect UNCONFIRMED
1868724 After setting setting up a language using translate function user is facing issues to NEW
1868770 PerformanceEventTimeing.cpp may not be clamping/jittering in all circumstances when RFP is enabled NEW
1868912 Screenshot "Save" function is mislabeled "Download" UNCONFIRMED
1868914 Search option "Match Diacritics" is mislabeled UNCONFIRMED

39 Total; 39 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Triage process

Introduction

The goal of performance triage is to identify the extent to which bugs impact the performance of our products, and to move these bugs towards an actionable state. The goal is not to diagnose or fix bugs during triage. We triage bugs that have been nominated for triage and bugs in the Core::Performance component that do not have the performance impact project flag set.

During triage we may do any/all of the following:

  • Request further information from the reporter (such as a profile)
  • Set the performance impact project flag
  • Add performance keywords
  • Move the bug to a more appropriate component

Who is responsible for triage?

Everyone is welcome to take part in triage. By default, everyone on the performance team is enrolled in triage rotation, but we also have participants from outside the team.

How do I schedule a triage meeting?

If you are on triage duty, you will receive an invitation as a reminder to schedule the triage meeting on the shared performance calendar with the nominated sheriffs invited at a time that works for them. The responsibility of scheduling the meeting falls to the lead sheriff. Once a triage meeting has been scheduled, it’s a good idea to remove the reminder event from the calendar to avoid confusion. It’s a good idea to use the shared calendar, as this increases the visibility of the performance triage and allows other members of the team to contribute or observe the process.

What if a sheriff is unavailable?

The rotation script is not perfect, and doesn’t know when people are on PTO or otherwise unavailable. If the lead sheriff is available, it is their responsibility to either schedule the triage with the remaining available sheriff or to identify a suitable substitute for the unavailable sheriff(s). If the lead sheriff is unavailable, this responsibility passes onto the remaining available sheriffs.

How do I run a triage meeting?

The following describes the triage process to follow during the meeting:

  1. Ask if others would prefer you to share your screen. This can be especially helpful for those new to triage.
  2. Open the first triage query to show bugs nominated for triage or in the Core::Performance component without the performance impact project flag set. The bugs are sorted from oldest to newest. For each bug in the list, follow these steps:
    • Bugs that look like tasks that were filed by members of the Performance team will generally need to be moved to the Core::Performance Engineering component.
    • For defects: Determine if the bug is reproducible and actionable. If not, add a needinfo for the reporter asking for more information, set the performance impact project flag to pending-needinfo, and then move onto the next bug. We have a template that you can modify as needed.
    • For all bugs (including enhancements):
  3. Open the second triage query to show bugs that are waiting further information to determine the performance impact. The bugs are sorted from oldest to newest. For each bug in the list, follow these steps:
    • If the performance impact project flag was set to pending-needinfo less than 2 weeks ago, move onto the next bug.
    • If the performance impact project flag was set to pending-needinfo more than 2 weeks ago but less than 2 months ago, consider adding a needinfo for either: another reporter of the issue, someone with access to the appropriate platform(s) to attempt to reproduce the issue, or a relevant subject matter expert.
    • If the performance impact project flag was set to pending-needinfo more than 2 months ago, close the bug as inactive. You can modify the inactive bug template as needed.
  4. If time permits, open the third triage query to show recently opened bugs with performance related keywords in the summary. If any of these look like performance bugs, they can either be triaged the same way as bugs in the initial query or they can be nominated for triage in a subsequent meeting.

What if things don't go as expected?

Don't panic! The triage process is not expected to be perfect, and can improve with your feedback. Maybe the result of the triage calculator doesn't feel right, or you find a scenario that's not covered in these guidelines. In this case we recommend that you bring it up in #perf-triage, or consider scheduling a short meeting with some triage leads (you can see some recent leads in the triage rotation). If in doubt, leave a comment on the bug with your thoughts and move on. There's a chance someone will respond, but if not the next performance triage sheriffs may have some other ideas.

How do I determine the performance impact project flag?

The performance impact project flag is used to indicate a bug’s relationship to the performance of our products. It can be applied to all bugs, and not only defects. The triage calculator should be used to help determine the most appropriate value for this flag. In addition to setting the performance impact project flag, make sure to use the “Copy Bugzilla Comment” button and paste this as a comment on the bug.

If you do not have enough information to set the performance impact project flag, open a needinfo request against an appropriate individual (such as a reporter), and set the performance impact project flag to pending-needinfo.

For more information about what this flag, and it's settings mean see this blog post.

How do I determine the performance keywords?

There are several performance related keywords, which can be helpful to understand how our performance issues are distributed, or whenever there’s a concerted effort to improve a particular aspect of our products. The triage calculator may recommend keywords to set, and by typing “perf:” in the keywords field in Bugzilla, you will see the available options. Select all that apply to the bug.

How do I determine the correct Bugzilla component?

Ideally we would only have bugs in the Core::Performance component that are the responsibility of the engineers in the performance team. For performance bugs to have the best chance of being fixed, it's important to assign them to the correct component. In some cases the correct component will be obvious from the bug summary, description, or steps to reproduce. In other cases, you may need to do a bit more work to identify the component. For example, if there's a profile associated with the bug, you could see where the majority of time is being spent using the category annotations.

How do I read a performance profile?

It's useful to be able to understand a profile generated by the Firefox Profiler, and hopefully someone in the triage meeting will be able to help. If you find an interesting profile, or just want to understand how to use them to analyse a performance problem, we encourage you to post a link to the profile (or bug) in #joy-of-profiling where someone will be happy to help. The profile may even be analysed during one of the regular "Joy of Profiling" open sessions that can be found on the Performance Office Hours calendar.

Triage calculator

The Performance Impact Calculator was developed to assist in identifying and applying the performance impact project flag and performance keywords consistently. If you have feedback or would like to suggest changes to this tool, please share these in the #perf-triage Matrix channel.

Triage rotation

The sheriffs are allocated on a weekly basis, which is published here. The rotation is generated by this script.

Templates

New bug

This template is included in the description for new bugs opened in the Core::Performance component. If a bug is opened in another component and then moved to Core::Performance, this template can be used as needed to request additional information from the reporter.

### Basic information

Steps to Reproduce:


Expected Results:


Actual Results:


---

### Performance recording (profile)

Profile URL:
(If this report is about slow performance or high CPU usage, please capture a performance profile by following the instructions at https://profiler.firefox.com/. Then upload the profile and insert the link here.)

#### System configuration:

OS version:
GPU model:
Number of cores: 
Amount of memory (RAM): 

### More information

Please consider attaching the following information after filing this bug, if relevant:

 - Screenshot / screen recording
 - Anonymized about:memory dump, for issues with memory usage
 - Troubleshooting information: Go to about:support, click "Copy text to clipboard", paste it to a file, save it, and attach the file here.

---

Thanks so much for your help.

Moved to Core::Performance

This bug was moved into the Performance component. Reporter, could you make sure the following information is on this bug?

 - For slowness or high CPU usage, capture a profile with http://profiler.firefox.com/ , upload it and share the link here.
 - For memory usage issues, capture a memory dump from about:memory and attach it to this bug.
 - Troubleshooting information: Go to about:support, click "Copy raw data to clipboard", paste it into a file, save it, and attach the file here.

Thank you.

No longer able to reproduce

This bug doesn’t seem to happen anymore in current versions of Firefox. Please reopen or file a new bug if you see it again.

No response from reporter

With no answer from the reporter, we don’t have enough data to reproduce and/or fix this issue. Please reopen or file a new bug with more information if you see it again.

Expected behaviour

This is expected behavior. Please reopen or file a new bug if you think otherwise.

Website issue

According to the investigation, this is a website issue. Please reopen or file a new bug if you think otherwise.