Performance/MemShrink: Difference between revisions

No edit summary
m (added zoom number)
 
(168 intermediate revisions by 20 users not shown)
Line 1: Line 1:
MemShrink aims to reduce Firefox's memory consumption.  There are three potential benefits:
MemShrink is a project that aims to reduce the memory consumption of Firefox (on desktop and mobile).  There are three potential benefits.


# Speed: less cache and VM pressure.
# Speed. Firefox will be faster due to less cache pressure, less paging, and fewer/smaller GC and CC pauses.  Changes that reduce memory consumption but make Firefox slower are not desirable.
# Stability: fewer OOMs, whether due to address space exhaustion or otherwiseThis results in fewer crashes (due to mishandling of OOM) or aborts.
# Stability. Firefox will suffer fewer aborts/crashes due to virtual or physical memory exhaustion.  The former is mostly a problem on 32-bit Windows builds with a 2GB or 4GB virtual memory limit, the latter is mostly a problem on mobile devices that lack swap space.
# Perception: fewer people will complain about Firefox being a memory hog.
# Reputation. Fewer people will complain that Firefox is a memory hog and that Mozilla ignores memory usage.


== areweslimyet.com ==
[http://blog.mozilla.com/nnethercote/2012/02/08/the-benefits-of-reducing-memory-consumption-2/ This blog post] describes these benefits in more detail.


jmuizelaar set up [http://areweslimyet.com/ AWSY], inspired by [http://arewefastyet.com AWFY].  It currently points to a [http://graphs-stage2.mozilla.org Perf-o-matic] Talos graph (Tp4 private bytes).
There are two main ways to reduce memory consumption.


Why was AWFY so successful?  Some key characteristics.
# "Slim down" memory usage, e.g. make data structures more space-efficient.
* The benchmarks were easy to choose:  everybody already used SS and V8, and then Mozilla pushed out KrakenThey run quickly, too, which is nice for devs, and allows expensive tools (like Cachegrind) to be used.
# Avoid "leaks"This loose use of the term (which is used throughout this document) includes:
* The metrics were easy to choose.  SS time, V8 time, Kraken time. They can be measured easily, precisely, and fairly repeatably. You can break the total time down into per-benchmark times, which really helps with understanding improvements and regressions. They're easy for devs to run on their own machine.
#* True leaks, where memory is lost forever.
* There was a well-defined goal:  match or beat the other browsers.
#* Lifetime issues, where memory is not reclaimed until you close the page/tab/window/process.
* Understanding where the time went was fairly easy, because there are good time-oriented profilers:  Shark, Cachegrind, etc.
#* Collection heuristic issues (e.g. GC is too infrequent).
#* Bad cache algorithms and poorly tuned caches.
#* Fragmentation.


In comparison, for AWSY there is no good and/or standard benchmark suite, metrics are less clear and/or harder to measure, there's no well-defined goal ("better than it currently is" isn't enough), and the memory profiling tools aren't very goodLet's consider each of those in turns.
Leaks are generally more important, because they are more likely to lead to horrible performance.   


== Better Benchmarks ==
== Meetings ==
Meetings happen every second Wednesday at 11am Pacific time ([http://arewemeetingyet.com/Los%20Angeles/2014-10-14/16:00/b/MemShrink%20Meeting countdown in your timezone])


Talos benchmarks currently load a bunch of popular (Alexa top 100) pages sequentially, take measurements periodically, and somehow average those measurements into a single number.
{{conf|95346}}
* zoom: 310 455 675
* IRC: [irc://irc.mozilla.org/memshrink #memshrink]


Opening only one tab at a time is silly.  Sayre's [https://bugzilla.mozilla.org/show_bug.cgi?id=631705 Membuster] benchmark looks promising;  it opens a heap of new tabs.
== Bug Tracking ==


Repeatability is a big issue.  If we load pages from the actual internet, most of them will change frequently.  Using a webpage recorder would be good, if it can be done.  It might be worth redoing the recordings every 6 months to keep them up-to-date with the web's evolution.
Bugs are prioritized during MemShrink meetings.


== Better Metrics ==
If you want a bug that is related to memory usage to be triaged in a MemShrink meeting, add "[MemShrink]" to the bug whiteboard. After prioritization, this whiteboard marking will be changed to one of "[MemShrink:P1]", "[MemShrink:P2]", "[MemShrink:P3]" or "[MemShrink:meta]". The meaning of these markings, and links to all such bugs, is as follows.


Picking good metrics is difficult, especially if we want to compare against other browsers.
* [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=regexp&query_format=advanced&list_id=577353&status_whiteboard=MemShrink%5b%5e%3A%5d&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&resolution=---&resolution=DUPLICATE Unprioritized MemShrink bugs].  These are prioritized regularly in meetings.
* [https://bugzilla.mozilla.org/buglist.cgi?list_id=463753&resolution=---&resolution=DUPLICATE&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=MemShrink%3AP1&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&order=bugs.bug_id MemShrink:P1 bugs]. Once prioritized, these will be revisited and discussed occasionally in meetings.
* [https://bugzilla.mozilla.org/buglist.cgi?list_id=463753&resolution=---&resolution=DUPLICATE&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=MemShrink%3AP2&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&order=bugs.bug_id MemShrink:P2 bugs]. Once prioritized, these will be revisited and discussed rarely in meetings. P2 is the default priority.
* [https://bugzilla.mozilla.org/buglist.cgi?list_id=463753&resolution=---&resolution=DUPLICATE&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=MemShrink%3AP3&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&order=bugs.bug_id MemShrink:P3 bugs]. Once prioritized, these are unlikely to be revisited and discussed again in meetings.
* [https://bugzilla.mozilla.org/buglist.cgi?list_id=463753&resolution=---&resolution=DUPLICATE&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=MemShrink%3Ameta&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&order=bugs.bug_id MemShrink:meta bugs]. Once prioritized, these will be revisited and discussed occasionally in meetings.


* Total memory is only moderately relevant, because lots of mapped pages aren't touched and so don't cost anything. (Thread stacks are a good example -- they're 8MB each on 64-bit Linux.)  The exception is on 32-bit machines where address space exhaustion is possible.
Some other interesting bug lists that overlap with the ones above.
* RSS (resident set size) is more relevant from a performance point of view. But it's controlled by some more-or-less opaque kernel page discard algorithm, and depends on the behaviour of all processes in the system, which isn't great. But if tests were always run on the same machine and the browser was the only thing running, this might not be too bad. (No good for measuring once you get past peak, because jemalloc holds onto pages containing freed heap blocks? Not sure about that.)
* [https://bugzilla.mozilla.org/buglist.cgi?list_id=463753&resolution=---&resolution=DUPLICATE&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=MemShrink&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&order=bugs.bug_id All MemShrink bugs].
* We could do something like the size of all heap allocations plus other notable non-heap allocations (like code pages allocated by the JITs).  This requires some internal accounting and so is hard to compare against other browsers.
* [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&list_id=1252875&status_whiteboard=MemShrink&bug_status=UNCONFIRMED&resolution=---&resolution=DUPLICATE Unconfirmed MemShrink bugs]. These are problems reported by users and generally require some kind of additional work to confirm.
* What do we do about multiple processes? Measure them all and sum the numbers?
* [https://bugzilla.mozilla.org/buglist.cgi?f1=bug_mentor&list_id=12543462&o1=isnotempty&resolution=---&resolution=DUPLICATE&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=MemShrink&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED MemShrink bugs with a "mentor" annotation]. These are bugs that someone has identified as reasonable easy, and that person is willing to help a newcomer fix the bug.
* There seems to be a lot of non-determinism in browser memory usage, which makes pinpointing small regressions/improvements difficult.
* [https://bugzilla.mozilla.org/buglist.cgi?list_id=1252950&resolution=---&resolution=DUPLICATE&emailtype1=regexp&status_whiteboard_type=allwordssubstr&emailassigned_to1=1&query_format=advanced&status_whiteboard=MemShrink&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&email1=nobody%40%7Cgeneral%40js.bugs Unassigned MemShrink bugs]. These need someone to work on them.
* Do we only measure peak usage?  Or do we want to open and close lots of tabs to try to catch leaks?
* jseward's [http://blog.mozilla.com/jseward/2011/01/27/profiling-the-browsers-virtual-memory-behaviour/ VM-sim extension to Cachegrind] takes a different tack;  it measures page faults on a quasi-idealized machine.  It's slow, though.


== A Clear Goal ==
== Tools ==


Some ideas:
All the tools that we use to measure and improve memory usage are described [https://developer.mozilla.org/en-US/docs/Mozilla/Performance on MDN], under "Memory profiling and leak detection tools".
* A certain percentage better than current, for whatever metric we choose. Arbitrary, but clear.
* Within a certain percentage of other browsers (possibly including Firefox 3.6).  Requires a metric that can be measured in other browsers easily.
* No obvious leaks.  Could be aimed for if we measure after closing lots of tabs.
* Keep going until we can open N copies of TechCrunch on some standard phone? (I.e. until the mobile guys stop complaining)
* Keep going until commenters on slashdot/Ars/reddit/etc stop complaining??


== Better Profiling ==
== Meeting Minutes, Progress Reports and Presentations ==


nnethercote has plans for greatly improving about:memory. See [http://blog.mozilla.com/nnethercote/2011/02/09/a-vision-for-better-memory-profiling-with-aboutmemory/ this blog post] for details.  These will make it much easier for everyone to understand how Firefox is using memory.
We no longer take meeting minutes nor make progress reports, but the meetings still happen. Brief MemShrink updates are given in the minutes of the weekly Platform meeting ([https://wiki.mozilla.org/Platform#Meeting_Notes wiki], [https://blog.mozilla.org/meeting-notes/archives/tag/mozillaplatform blog]). And nnethercote still [https://blog.mozilla.org/nnethercote/category/memshrink/ blogs] about major MemShrink occurrences.


== A Straw-man Proposal ==
Here is the historical record from when we did take minutes and make progress reports.


Just to get discussion going...
{| border="1" cellpadding="2"
! Date !! Minutes !! Progress report
|-
| 2014-06-16 || - || [https://blog.mozilla.org/nnethercote/2014/06/16/memshrinks-3rd-birthday/ MemShrink's 3rd Birthday] (includes the fourth "Big Ticket Items" list)
|-
| 2013-11-26 || - || [https://blog.mozilla.org/nnethercote/2013/11/27/memshrink-progress-final/ The final progress report]
|-
| 2013-10-29 || - || [https://blog.mozilla.org/nnethercote/2013/10/30/memshrink-progress-week-121-124/ Week 121--124]
|-
| 2013-09-30 || - || [https://blog.mozilla.org/nnethercote/2013/10/01/memshrink-progress-week-117-120/ Week 117--120]
|-
| 2013-09-03 || - || [https://blog.mozilla.org/nnethercote/2013/09/04/memshrink-progress-week-113-116/ Week 113--116]
|-
| 2013-08-06 || - || [https://blog.mozilla.org/nnethercote/2013/08/07/memshrink-progress-week-109-112/ Week 109--112]
|-
| 2013-07-09 || - || [https://blog.mozilla.org/nnethercote/2013/07/10/memshrink-progress-week-105-108/ Week 105--108]
|-
| 2013-06-14 || - || [https://blog.mozilla.org/nnethercote/2013/06/15/memshrinks-2nd-birthday/ MemShrink's 2nd Birthday] (includes the third "Big Ticket Items" list)
|-
| 2013-06-11 || - || [https://blog.mozilla.org/nnethercote/2013/06/12/memshrink-progress-week-103-104/ Week 103--104]
|-
| 2013-05-28 || - || [http://jlebar.com/2013/5/8/MemShrink_process%2C_weeks_98-102.html Week 99--102] (guest post)
|-
| 2013-04-30 || - || [http://jlebar.com/2013/5/8/MemShrink_process%2C_weeks_97-98.html Week 97--98] (guest post)
|-
| 2013-04-16 || - || [https://blog.mozilla.org/nnethercote/2013/04/17/memshrink-progress-week-95-96/ Week 95--96]
|-
| 2013-04-02 || - || [https://blog.mozilla.org/nnethercote/2013/04/03/memshrink-progress-week-93-94/ Week 93--94]
|-
| 2013-03-19 || - || [https://blog.mozilla.org/nnethercote/2013/03/20/memshrink-progress-week-91-92/ Week 91--92]
|-
| 2013-03-05 || - || [https://blog.mozilla.org/nnethercote/2013/03/07/memshrink-progress-week-89-90/ Week 89--90]
|-
| 2013-02-19 || - || [https://blog.mozilla.org/nnethercote/2013/02/20/memshrink-progress-week-87-88/ Week 87--88]
|-
| 2013-02-05 || - || [https://blog.mozilla.org/nnethercote/2013/02/06/memshrink-progress-week-85-86/ Week 85--86]
|-
| 2013-01-22 || - || [https://blog.mozilla.org/nnethercote/2013/01/23/memshrink-progress-week-83-84/ Week 83--84]
|-
| 2013-01-08 || - || [https://blog.mozilla.org/nnethercote/2013/01/09/memshrink-progress-week-79-82/ Week 79--82]
|-
| 2012-12-11 || - || [https://blog.mozilla.org/nnethercote/2012/12/12/memshrink-progress-week-77-78/ Week 77--78]
|-
| 2012-11-27 || - || [https://blog.mozilla.org/nnethercote/2012/11/28/memshrink-progress-week-75-76/ Week 75--76]
|-
| 2012-11-13 || - || [https://blog.mozilla.org/nnethercote/2012/11/14/memshrink-progress-week-73-74/ Week 73--74]
|-
| 2012-10-30 || - || [https://blog.mozilla.org/nnethercote/2012/10/31/memshrink-progress-week-71-72/ Week 71--72]
|-
| 2012-10-16 || - || [https://blog.mozilla.org/nnethercote/2012/10/17/memshrink-progress-week-69-70/ Week 69--70]
|-
| 2012-10-02 || - || [https://blog.mozilla.org/nnethercote/2012/10/03/memshrink-progress-week-67-68/ Week 67--68]
|-
| 2012-09-18 || - || [https://blog.mozilla.org/nnethercote/2012/09/19/memshrink-progress-week-65-66/ Week 65--66]
|-
| 2012-09-04 || - || [https://blog.mozilla.org/nnethercote/2012/09/05/memshrink-progress-week-63-64/ Week 63--64]
|-
| 2012-08-21 || - || [https://blog.mozilla.org/nnethercote/2012/08/23/memshrink-progress-week-61-62/ Week 61--62]
|-
| 2012-08-07 || - || [https://blog.mozilla.org/nnethercote/2012/08/08/memshrink-progress-week-57-60/ Week 57--60]
|-
| 2012-07-10 || - || [https://blog.mozilla.org/nnethercote/2012/07/11/memshrink-progress-week-55-56/ Week 55--56] (includes the second "Big Ticket Items" list)
|-
| 2012-06-26 || - || [http://blog.mozilla.org/nnethercote/2012/06/27/memshrink-progress-week-53-54/ Week 53--54]
|-
| 2012-06-14 || - || [http://blog.mozilla.org/nnethercote/2012/06/15/memshrinks-1st-birthday/ MemShrink's 1st Birthday]
|-
| 2012-06-12 || - || [http://blog.mozilla.org/nnethercote/2012/06/13/memshrink-progress-week-51-52/ Week 51--52]
|-
| 2012-05-29 || - || [http://blog.mozilla.org/nnethercote/2012/05/30/memshrink-progress-week-49-50/ Week 49--50]
|-
| 2012-05-15 || - || [http://blog.mozilla.org/nnethercote/2012/05/16/memshrink-progress-week-47-48/ Week 47--48]
|-
| 2012-05-01 || - || [http://blog.mozilla.org/nnethercote/2012/05/02/memshrink-progress-week-45-46/ Week 45--46]
|-
| 2012-04-17 || - || [http://blog.mozilla.org/nnethercote/2012/04/18/memshrink-progress-week-43-44/ Week 43--44]
|-
| 2012-04-03 || - || [http://blog.mozilla.org/nnethercote/2012/04/04/memshrink-progress-week-42/ Week 42]
|-
| 2012-03-27 || - || [http://blog.mozilla.com/nnethercote/2012/03/28/memshrink-progress-week-41/ Week 41]
|-
| 2012-03-20 || - || [http://blog.mozilla.com/nnethercote/2012/03/21/memshrink-progress-week-40/ Week 40]
|-
| 2012-03-13 || - || [http://blog.mozilla.com/nnethercote/2012/03/14/memshrink-progress-week-39/ Week 39]
|-
| 2012-03-06 || - || [http://blog.mozilla.com/nnethercote/2012/03/07/memshrink-progress-week-38/ Week 38]
|-
| 2012-02-28 || - || [http://blog.mozilla.com/nnethercote/2012/02/29/memshrink-progress-week-37/ Week 37]
|-
| 2012-02-21 || - || [http://blog.mozilla.com/nnethercote/2012/02/22/memshrink-progress-week-36/ Week 36]
|-
| 2012-02-14 || - || [http://blog.mozilla.com/nnethercote/2012/02/15/memshrink-progress-week-35/ Week 35]
|-
| 2012-02-07 || - || [http://blog.mozilla.com/nnethercote/2012/02/08/memshrink-progress-week-34/ Week 34]
|-
| 2012-01-31 || - || [http://blog.mozilla.com/nnethercote/2012/02/01/memshrink-progress-week-33/ Week 33]
|-
| 2012-01-24 || - || [http://blog.mozilla.com/nnethercote/2012/01/25/memshrink-progress-week-32/ Week 32] (includes the first "Big Ticket Items" list)
|-
| 2012-01-17 || - || [http://blog.mozilla.com/nnethercote/2012/01/18/memshrink-progress-week-31/ Week 31]
|-
| 2012-01-10 || - || [http://blog.mozilla.com/nnethercote/2012/01/11/memshrink-progress-week-30/ Week 30]
|-
| 2012-01-03 || - || [http://blog.mozilla.com/nnethercote/2012/01/04/memshrink-progress-weeks-28-29/ Weeks 28--29]
|-
| 2011-12-20 || - || [http://blog.mozilla.com/nnethercote/2011/12/21/memshrink-progress-week-27/ Week 27]
|-
| 2011-12-13 || [[Performance/MemShrink/Meetings/2011-12-13|minutes]] || [http://blog.mozilla.com/nnethercote/2011/12/14/memshrink-progress-week-26/ Week 26]
|-
| 2011-12-06 || [[Performance/MemShrink/Meetings/2011-12-06|minutes]] ||  [http://blog.mozilla.com/nnethercote/2011/12/07/memshrink-progress-week-25/ Week 25]
|-
| 2011-11-29 || [[Performance/MemShrink/Meetings/2011-11-29|minutes]] ||  [http://blog.mozilla.com/nnethercote/2011/11/30/memshrink-progress-report-week-24/ Week 24]
|-
| 2011-11-22 || - ||  [http://blog.mozilla.com/nnethercote/2011/11/23/memshrink-progress-report-week-23/ Week 23]
|-
| 2011-11-15 || - ||  [http://blog.mozilla.com/nnethercote/2011/11/16/memshrink-progress-report-week-22/ Week 22]
|-
| 2011-11-08 || [[Performance/MemShrink/Meetings/2011-11-08|minutes]] || [http://blog.mozilla.com/nnethercote/2011/11/09/memshrink-progress-week-21/ Week 21]
|-
| 2011-11-01 || [[Performance/MemShrink/Meetings/2011-11-01|minutes]] || [http://blog.mozilla.com/nnethercote/2011/11/02/memshrink-progress-week-20/ Week 20]
|-
| 2011-10-25 || [[Performance/MemShrink/Meetings/2011-10-25|minutes]] || [http://blog.mozilla.com/nnethercote/2011/10/26/memshrink-progress-week-19/ Week 19]
|-
| 2011-10-18 || [[Performance/MemShrink/Meetings/2011-10-18|minutes]] || [http://blog.mozilla.com/nnethercote/2011/10/19/memshrink-progress-weeks-13-18/ Weeks 13--18]
|-
| 2011-10-11 || [[Performance/MemShrink/Meetings/2011-10-11|minutes]] || -
|-
| 2011-10-04 || [[Performance/MemShrink/Meetings/2011-10-04|minutes]] || -
|-
| 2011-09-27 || - || -
|-
| 2011-09-20 || [[Performance/MemShrink/Meetings/2011-09-20|minutes]] || -
|-
| 2011-09-13 || - || -
|-
| 2011-09-06 || [[Performance/MemShrink/Meetings/2011-09-06|minutes]] || [http://blog.mozilla.com/nnethercote/2011/09/07/memshrink-progress-week-12/ Week 12]
|-
| 2011-08-30 || [[Performance/MemShrink/Meetings/2011-08-30|minutes]] || [http://blog.mozilla.com/nnethercote/2011/08/31/memshrink-progress-week-11/ Week 11]
|-
| 2011-08-23 || [[Performance/MemShrink/Meetings/2011-08-23|minutes]] || [http://blog.mozilla.com/nnethercote/2011/08/24/memshrink-progress-week-10/ Week 10]
|-
| 2011-08-16 || [[Performance/MemShrink/Meetings/2011-08-16|minutes]] || [http://blog.mozilla.com/nnethercote/2011/08/17/memshrink-progress-week-9/ Week 9]
|-
| 2011-08-09 || [[Performance/MemShrink/Meetings/2011-08-09|minutes]] || [http://blog.mozilla.com/nnethercote/2011/08/10/memshrink-progress-week-8/ Week 8]
|-
| 2011-08-02 || - || [http://blog.mozilla.com/nnethercote/2011/08/03/memshrink-progress-week-7/ Week 7]
|-
| 2011-07-26 || [[Performance/MemShrink/Meetings/2011-07-26|minutes]] || [http://blog.mozilla.com/nnethercote/2011/07/27/memshrink-progress-week-6/ Week 6]
|-
| 2011-07-19 || [[Performance/MemShrink/Meetings/2011-07-19|minutes]] || [http://blog.mozilla.com/nnethercote/2011/07/20/memshrink-progress-week-5/ Week 5]
|-
| 2011-07-12 || [[Performance/MemShrink/Meetings/2011-07-12|minutes]] || [http://blog.mozilla.com/nnethercote/2011/07/13/memshrink-progress-week-4/ Week 4]
|-
| 2011-07-05 || [[Performance/MemShrink/Meetings/2011-07-05|minutes]] || [http://blog.mozilla.com/nnethercote/2011/07/06/memshrink-progress-week-3/ Week 3]
|-
| 2011-06-28 || - || [http://blog.mozilla.com/nnethercote/2011/06/29/memshrink-progress-week-2/ Week 2]
|-
| 2011-06-21 || - || [http://blog.mozilla.com/nnethercote/2011/06/22/memshrink-progress-week-1/ Week 1]
|-
| 2011-06-14 || - || -
|}


=== Benchmark ===
Something like Membuster:  open a bunch of popular and/or interesting (eg. memory intensive, ones that caused problems in the past) pages, one per tab. 
Not too many though;  I suspect that opening 100 won't give us much more information that 10 would, and it should be runnable by devs easily.
Use recorded versions for consistent results, if possible, and make those recorded versions easy to use by devs.


=== Metrics ===
Presentations.
* Peak RSS:  easy to measure (on all platforms -- I think?) across all browsers, has some reflection on actual performance.
* Peak total: easy to measure across all browsers, interesting for the address space exhaustion issue.


So, only two metrics per browser per run.  Leaks aren't addressed;  might be able to included them if we measure at startup and shutdown (after all tabs have closed).
*2012-01-16 [[media:LCA2012.pdf|"Notes on Reducing Firefox's Memory Consumption"]], presented at linux.conf.au Browser MiniConf
 
=== Goal ===
Reduce peak RSS by 1/3.
 
(Who knows?  Probably need to measure other browsers to get a sense of what the goal should be.)
 
== Pushing This Forward ==
nnethercote and jmuizelaar have volunteered to co-lead the MemShrink effort.
MemShrink could be modelled on CrashKill and CritSmash.  One problem with that is that neither of them are in Mountain View -- jmuizelaar is in the Toronto office (not too bad), nnethercote works at home in a PST+17/+19 timezone. So having CrashKill/CritSmash-style meetings will be difficult.
 
The MemShrink effort can also include leaks and quasi-leaks in its jurisdiction, even if areweslimyet.com doesn't demonstrate them.
 
In terms of tracking bugs, now that we're on a 3 month release cycle, I like the idea of having one meta-bug tracking all memory use reductions for the next release.  I personally find meta-bugs like this to be quite useful so long as they have a clear end-point.

Latest revision as of 20:17, 10 July 2019

MemShrink is a project that aims to reduce the memory consumption of Firefox (on desktop and mobile). There are three potential benefits.

  1. Speed. Firefox will be faster due to less cache pressure, less paging, and fewer/smaller GC and CC pauses. Changes that reduce memory consumption but make Firefox slower are not desirable.
  2. Stability. Firefox will suffer fewer aborts/crashes due to virtual or physical memory exhaustion. The former is mostly a problem on 32-bit Windows builds with a 2GB or 4GB virtual memory limit, the latter is mostly a problem on mobile devices that lack swap space.
  3. Reputation. Fewer people will complain that Firefox is a memory hog and that Mozilla ignores memory usage.

This blog post describes these benefits in more detail.

There are two main ways to reduce memory consumption.

  1. "Slim down" memory usage, e.g. make data structures more space-efficient.
  2. Avoid "leaks". This loose use of the term (which is used throughout this document) includes:
    • True leaks, where memory is lost forever.
    • Lifetime issues, where memory is not reclaimed until you close the page/tab/window/process.
    • Collection heuristic issues (e.g. GC is too infrequent).
    • Bad cache algorithms and poorly tuned caches.
    • Fragmentation.

Leaks are generally more important, because they are more likely to lead to horrible performance.

Meetings

Meetings happen every second Wednesday at 11am Pacific time (countdown in your timezone)


  • Dial-in: Audio-only conference# 95346
    • People with Mozilla phones or softphones please dial x4000 Conf# 95346
    • US/Toll-free: +1 800 707 2533, (pin 4000) Conf# 95346
    • US/California/Mountain View: +1 650 903 0800, x4000 Conf# 95346
    • US/California/San Francisco: +1 415 762 5700, x4000 Conf# 95346
    • US/Oregon/Portland: +1 971 544 8000, x4000 Conf# 95346
    • CA/British Columbia/Vancouver: +1 778 785 1540, x4000 Conf# 95346
    • CA/Ontario/Toronto: +1 416 848 3114, x4000 Conf# 95346
    • UK/London: +44 (0)207 855 3000, x4000 Conf# 95346
    • FR/Paris: +33 1 84 88 37 37, x4000 Conf# 95346
    • Gmail Chat (requires Flash and the Google Talk plugin): paste +1 650 903 0800 into the Gmail Chat box that doesn't look like it accepts phone numbers
    • SkypeOut is free if you use the 800 number
  • zoom: 310 455 675
  • IRC: #memshrink

Bug Tracking

Bugs are prioritized during MemShrink meetings.

If you want a bug that is related to memory usage to be triaged in a MemShrink meeting, add "[MemShrink]" to the bug whiteboard. After prioritization, this whiteboard marking will be changed to one of "[MemShrink:P1]", "[MemShrink:P2]", "[MemShrink:P3]" or "[MemShrink:meta]". The meaning of these markings, and links to all such bugs, is as follows.

  • Unprioritized MemShrink bugs. These are prioritized regularly in meetings.
  • MemShrink:P1 bugs. Once prioritized, these will be revisited and discussed occasionally in meetings.
  • MemShrink:P2 bugs. Once prioritized, these will be revisited and discussed rarely in meetings. P2 is the default priority.
  • MemShrink:P3 bugs. Once prioritized, these are unlikely to be revisited and discussed again in meetings.
  • MemShrink:meta bugs. Once prioritized, these will be revisited and discussed occasionally in meetings.

Some other interesting bug lists that overlap with the ones above.

Tools

All the tools that we use to measure and improve memory usage are described on MDN, under "Memory profiling and leak detection tools".

Meeting Minutes, Progress Reports and Presentations

We no longer take meeting minutes nor make progress reports, but the meetings still happen. Brief MemShrink updates are given in the minutes of the weekly Platform meeting (wiki, blog). And nnethercote still blogs about major MemShrink occurrences.

Here is the historical record from when we did take minutes and make progress reports.

Date Minutes Progress report
2014-06-16 - MemShrink's 3rd Birthday (includes the fourth "Big Ticket Items" list)
2013-11-26 - The final progress report
2013-10-29 - Week 121--124
2013-09-30 - Week 117--120
2013-09-03 - Week 113--116
2013-08-06 - Week 109--112
2013-07-09 - Week 105--108
2013-06-14 - MemShrink's 2nd Birthday (includes the third "Big Ticket Items" list)
2013-06-11 - Week 103--104
2013-05-28 - Week 99--102 (guest post)
2013-04-30 - Week 97--98 (guest post)
2013-04-16 - Week 95--96
2013-04-02 - Week 93--94
2013-03-19 - Week 91--92
2013-03-05 - Week 89--90
2013-02-19 - Week 87--88
2013-02-05 - Week 85--86
2013-01-22 - Week 83--84
2013-01-08 - Week 79--82
2012-12-11 - Week 77--78
2012-11-27 - Week 75--76
2012-11-13 - Week 73--74
2012-10-30 - Week 71--72
2012-10-16 - Week 69--70
2012-10-02 - Week 67--68
2012-09-18 - Week 65--66
2012-09-04 - Week 63--64
2012-08-21 - Week 61--62
2012-08-07 - Week 57--60
2012-07-10 - Week 55--56 (includes the second "Big Ticket Items" list)
2012-06-26 - Week 53--54
2012-06-14 - MemShrink's 1st Birthday
2012-06-12 - Week 51--52
2012-05-29 - Week 49--50
2012-05-15 - Week 47--48
2012-05-01 - Week 45--46
2012-04-17 - Week 43--44
2012-04-03 - Week 42
2012-03-27 - Week 41
2012-03-20 - Week 40
2012-03-13 - Week 39
2012-03-06 - Week 38
2012-02-28 - Week 37
2012-02-21 - Week 36
2012-02-14 - Week 35
2012-02-07 - Week 34
2012-01-31 - Week 33
2012-01-24 - Week 32 (includes the first "Big Ticket Items" list)
2012-01-17 - Week 31
2012-01-10 - Week 30
2012-01-03 - Weeks 28--29
2011-12-20 - Week 27
2011-12-13 minutes Week 26
2011-12-06 minutes Week 25
2011-11-29 minutes Week 24
2011-11-22 - Week 23
2011-11-15 - Week 22
2011-11-08 minutes Week 21
2011-11-01 minutes Week 20
2011-10-25 minutes Week 19
2011-10-18 minutes Weeks 13--18
2011-10-11 minutes -
2011-10-04 minutes -
2011-09-27 - -
2011-09-20 minutes -
2011-09-13 - -
2011-09-06 minutes Week 12
2011-08-30 minutes Week 11
2011-08-23 minutes Week 10
2011-08-16 minutes Week 9
2011-08-09 minutes Week 8
2011-08-02 - Week 7
2011-07-26 minutes Week 6
2011-07-19 minutes Week 5
2011-07-12 minutes Week 4
2011-07-05 minutes Week 3
2011-06-28 - Week 2
2011-06-21 - Week 1
2011-06-14 - -


Presentations.