Performance/MemShrink

MemShrink aims to reduce Firefox's memory consumption. There are three potential benefits:

1. Speed: less cache and VM pressure.
2. Stability: fewer OOMs, whether due to address space exhaustion or otherwise. This results in fewer crashes (due to mishandling of OOM) or aborts.
3. Perception: fewer people will complain about Firefox being a memory hog.

These factors are doubly important for Firefox Mobile.

areweslimyet.com

jmuizelaar set up AWSY, inspired by AWFY. It currently points to a Perf-o-matic Talos graph (Tp4 private bytes).

Why was AWFY so successful? Some key characteristics.

• The benchmarks were easy to choose: everybody already used SS and V8, and then Mozilla pushed out Kraken. They run quickly, too, which is nice for devs, and allows expensive tools (like Cachegrind) to be used.
• The metrics were easy to choose. SS time, V8 time, Kraken time. They can be measured easily, precisely, and fairly repeatably. You can break the total time down into per-benchmark times, which really helps with understanding improvements and regressions. They're easy for devs to run on their own machine.
• There was a well-defined goal: match or beat the other browsers.
• Understanding where the time went was fairly easy, because there are good time-oriented profilers: Shark, Cachegrind, etc.

In comparison, for AWSY there is no good and/or standard benchmark suite, metrics are less clear and/or harder to measure, there's no well-defined goal ("better than it currently is" isn't enough), and the memory profiling tools aren't very good. Let's consider each of those in turns.

Better Benchmarks

Talos benchmarks currently load a bunch of popular (Alexa top 100) pages sequentially, take measurements periodically, and somehow average those measurements into a single number.

Opening only one tab at a time is silly. Sayre's Membuster benchmark looks promising; it opens a heap of new tabs.

Repeatability is a big issue. If we load pages from the actual internet, most of them will change frequently. Using a webpage recorder would be good, if it can be done. It might be worth redoing the recordings every 6 months to keep them up-to-date with the web's evolution.

Better Metrics

Picking good metrics is difficult, especially if we want to compare against other browsers.

• Total memory is only moderately relevant, because lots of mapped pages aren't touched and so don't cost anything. (Thread stacks are a good example -- they're 8MB each on 64-bit Linux.) The exception is on 32-bit machines where address space exhaustion is possible.
• RSS (resident set size) is more relevant from a performance point of view. But it's controlled by some more-or-less opaque kernel page discard algorithm, and depends on the behaviour of all processes in the system, which isn't great. But if tests were always run on the same machine and the browser was the only thing running, this might not be too bad. (No good for measuring once you get past peak, because jemalloc holds onto pages containing freed heap blocks? Not sure about that.)
• We could do something like the size of all heap allocations plus other notable non-heap allocations (like code pages allocated by the JITs). This requires some internal accounting and so is hard to compare against other browsers.
• What do we do about multiple processes? Measure them all and sum the numbers?
• There seems to be a lot of non-determinism in browser memory usage, which makes pinpointing small regressions/improvements difficult.
• Do we only measure peak usage? Or do we want to open and close lots of tabs to try to catch leaks?
• jseward's VM-sim extension to Cachegrind takes a different tack; it measures page faults on a quasi-idealized machine. It's slow, though.

An alternative approach

Good metrics are more difficult for memory usage than they are for cpu time. Mostly this is bad, but we may be able to use it to our advantage because it means it's harder for people to show that Firefox is using more memory than another browser. i.e. you can't load up the equivalent to sunspider in both browsers and immediately see which is better. This means that impact of bad memory usage is based on experience during regular browsing.

I wonder if we could add something to Firefox input like "Report Firefox using too much memory" we could then upload the contents of about:memory. This can help us get a better picture of when our users have the perception that we're using too much memory. It also gives us a better opportunity to discover memory leaks that are happening in the field.

A Clear Goal

Some ideas:

• A certain percentage better than current, for whatever metric we choose. Arbitrary, but clear.
• Within a certain percentage of other browsers (possibly including Firefox 3.6). Requires a metric that can be measured in other browsers easily.
• No obvious leaks. Could be aimed for if we measure after closing lots of tabs.
• Keep going until we can open N copies of TechCrunch on some standard phone? (I.e. until the mobile guys stop complaining)
• Keep going until commenters on slashdot/Ars/reddit/etc stop complaining??

Better Profiling

nnethercote has plans for greatly improving about:memory. See this blog post for details. These will make it much easier for everyone to understand how Firefox is using memory.

Chrome also has a JS heap snapshot tool, which looks pretty useful. It'd be great to have one in Firefox.

A Straw-man Proposal

Just to get discussion going...

Benchmark

Something like Membuster: open a bunch of popular and/or interesting (eg. memory intensive, ones that caused problems in the past) pages, one per tab. Not too many though; I suspect that opening 100 won't give us much more information that 10 would, and it should be runnable by devs easily. Use recorded versions for consistent results, if possible, and make those recorded versions easy to use by devs.

Metrics

• Peak RSS: easy to measure (on all platforms -- I think?) across all browsers, has some reflection on actual performance.
• Peak total: easy to measure across all browsers, interesting for the address space exhaustion issue.

So, only two metrics per browser per run. Leaks aren't addressed; might be able to included them if we measure at startup and shutdown (after all tabs have closed).

Goal

Reduce peak RSS by 1/3.

(Who knows? Probably need to measure other browsers to get a sense of what the goal should be.)

Pushing This Forward

nnethercote and jmuizelaar have volunteered to co-lead the MemShrink effort. MemShrink could be modelled on CrashKill and CritSmash. One problem with that is that neither of them are in Mountain View -- jmuizelaar is in the Toronto office (not too bad), nnethercote works at home in a PST+17/+19 timezone. So having CrashKill/CritSmash-style meetings will be difficult.

The MemShrink effort can also include leaks and quasi-leaks in its jurisdiction, even if areweslimyet.com doesn't demonstrate them.

In terms of tracking bugs, now that we're on a 3 month release cycle, I like the idea of having one meta-bug tracking all memory use reductions for the next release. I personally find meta-bugs like this to be quite useful so long as they have a clear end-point. So I opened bug 640452 for leaks in Firefox 5 and and bug 640457 for other memory use reductions in Firefox 5.