QA/Fennec/PerformanceBenchmark

Summary

We want to benchmark

Environment

• Maemo 5. N900
• Android 2.1 or 2.2. Vibrant, Droid, Nexus One, Captivate, Droid 2

Resources

• Waverley. <tester>

Test Plan

1. Fennec startup Time

• Record startup time from cold start to home page
• Record startup time after restarting after applying an addon

2. Pageload time

• Pick 10 top US sites and record pageload times
• Pick 10 top International sites and record pageload times
• PR 1.2 vs. PR 1.3

3. Page navigation time

• Pick 10 top US sites and switch to 2-3 links
• Pick 10 top International sites and switch to 2-3 links
• Record page navigation time

4. Measure Memory Usage (ie. Process Manager)

• Record memory size at startup (home page only)
• Open 5 tabs and record memory size
• Play an Ogg video and record memory size
• Launch a news RSS feed and record memory size
• Close Fennec and record memory size

Results

• Report results in the spreadsheet. but describe observations here.

References

• bug 606574 needs to be added to an extension to measure responsiveness. Need resources.

1) Maemo: Maemo 5 on N900. Fennec 1.1 vs. Fennec 4 beta - Record fennec startup time on the N900 - Pick 5 topsites and compare pageloads on the N900 - PR 1.2 vs. PR 1.3 2) Android: Assorted devices against Android 2.2 - Record fennec startup time against Vibrant, Droid 2 - Pick 5-10 topsites and record pageload times. Compare against other other device browsers (Android, Opera Mobile? Dolphin?) - Restart time against the Vibrant, Droid 2 (install/uninstall addons, update to a new nightly, switch locales) When recording startup time, you want to consider "warm" and "cold" speeds. pageload of 5 or 10 is a very small sample. I would argue that you need to pick the top 50 or top 100 US and top 50 international sites. There has been alot of thought into this space, a good primer is here: http://weblogs.mozillazine.org/hyatt/archives/2004_05.html We need a responsiveness test. How do we, on android, compare to 1.1, default, and beta. We should try to make it not subjective. High speed video might be a way to go.

Memory usage. We should compare memory usage (f.e. RSS, X, GL) of the browser during this page loads. We want to know how much memory each browser takes up before any pages are loaded, how much they take up after they load a bunch of pages, how much they take up after the windows/tabs are closed. CPU Usage. We should compare cpu usage of the browsers during the page load and idle.