QA/Fennec/PerformanceBenchmark: Difference between revisions
m (Created page with "= Summary = = Environment = = Resources = = Test Plan = = Results = = References =") |
mNo edit summary |
||
| Line 1: | Line 1: | ||
= Summary = | = Summary = | ||
We want to benchmark | |||
= Environment = | = Environment = | ||
* Maemo 5. N900 | |||
* Android 2.1 or 2.2. Vibrant, Droid, Nexus One, Captivate, Droid 2 | |||
= Resources = | = Resources = | ||
* Waverley. <tester> | |||
= Test Plan = | = Test Plan = | ||
1. Fennec | |||
1. Fennec startup Time | |||
* Reco | |||
* Record startup time from cold start to | |||
2. Pageload time | |||
* Pick 10 top US sites and record pageload times | |||
* Pick 10 top International sites and record pageload times | |||
* PR 1.2 vs. PR 1.3 | |||
3. Page navigation time | |||
* Pick 10 top US sites and switch to 2-3 links | |||
* Pick 10 top International sites and switch to 2-3 links | |||
* Record page navigation time | |||
4. Measure Memory Usage | |||
Memory usage. We should compare memory usage (f.e. RSS, X, GL) of the browser during this page loads. We want to know how much memory each browser takes up before any pages are loaded, how much they take up after they load a bunch of pages, how much they take up after the windows/tabs are closed. | |||
CPU Usage. We should compare cpu usage of the browsers during the page load and idle. | |||
* Record memory size at startup | |||
* Open 5 tabs and record memory size | |||
* Close Fennec and record memory size | |||
= Results = | = Results = | ||
* Report results in the spreadsheet. but describe observations here. | |||
= References = | = References = | ||
* {{bug|606574}} needs to be added to an extension to measure responsiveness. Need resources. | |||
1) Maemo: Maemo 5 on N900. Fennec 1.1 vs. Fennec 4 beta | |||
- Record fennec startup time on the N900 | |||
- Pick 5 topsites and compare pageloads on the N900 | |||
- PR 1.2 vs. PR 1.3 | |||
2) Android: Assorted devices against Android 2.2 | |||
- Record fennec startup time against Vibrant, Droid 2 | |||
- Pick 5-10 topsites and record pageload times. Compare against other other device browsers (Android, Opera Mobile? Dolphin?) | |||
- Restart time against the Vibrant, Droid 2 (install/uninstall addons, update to a new nightly, switch locales) | |||
When recording startup time, you want to consider "warm" and "cold" speeds. | |||
pageload of 5 or 10 is a very small sample. I would argue that you need to pick the top 50 or top 100 US and top 50 international sites. There has been alot of thought into this space, a good primer is here: http://weblogs.mozillazine.org/hyatt/archives/2004_05.html | |||
We need a responsiveness test. How do we, on android, compare to 1.1, default, and beta. We should try to make it not subjective. High speed video might be a way to go. | |||
Memory usage. We should compare memory usage (f.e. RSS, X, GL) of the browser during this page loads. We want to know how much memory each browser takes up before any pages are loaded, how much they take up after they load a bunch of pages, how much they take up after the windows/tabs are closed. | |||
CPU Usage. We should compare cpu usage of the browsers during the page load and idle. | |||
Revision as of 23:18, 12 November 2010
Summary
We want to benchmark
Environment
- Maemo 5. N900
- Android 2.1 or 2.2. Vibrant, Droid, Nexus One, Captivate, Droid 2
Resources
- Waverley. <tester>
Test Plan
1. Fennec
1. Fennec startup Time
- Reco
- Record startup time from cold start to
2. Pageload time
- Pick 10 top US sites and record pageload times
- Pick 10 top International sites and record pageload times
- PR 1.2 vs. PR 1.3
3. Page navigation time
- Pick 10 top US sites and switch to 2-3 links
- Pick 10 top International sites and switch to 2-3 links
- Record page navigation time
4. Measure Memory Usage Memory usage. We should compare memory usage (f.e. RSS, X, GL) of the browser during this page loads. We want to know how much memory each browser takes up before any pages are loaded, how much they take up after they load a bunch of pages, how much they take up after the windows/tabs are closed. CPU Usage. We should compare cpu usage of the browsers during the page load and idle.
- Record memory size at startup
- Open 5 tabs and record memory size
- Close Fennec and record memory size
Results
- Report results in the spreadsheet. but describe observations here.
References
- bug 606574 needs to be added to an extension to measure responsiveness. Need resources.
1) Maemo: Maemo 5 on N900. Fennec 1.1 vs. Fennec 4 beta - Record fennec startup time on the N900 - Pick 5 topsites and compare pageloads on the N900 - PR 1.2 vs. PR 1.3 2) Android: Assorted devices against Android 2.2 - Record fennec startup time against Vibrant, Droid 2 - Pick 5-10 topsites and record pageload times. Compare against other other device browsers (Android, Opera Mobile? Dolphin?) - Restart time against the Vibrant, Droid 2 (install/uninstall addons, update to a new nightly, switch locales) When recording startup time, you want to consider "warm" and "cold" speeds. pageload of 5 or 10 is a very small sample. I would argue that you need to pick the top 50 or top 100 US and top 50 international sites. There has been alot of thought into this space, a good primer is here: http://weblogs.mozillazine.org/hyatt/archives/2004_05.html We need a responsiveness test. How do we, on android, compare to 1.1, default, and beta. We should try to make it not subjective. High speed video might be a way to go.
Memory usage. We should compare memory usage (f.e. RSS, X, GL) of the browser during this page loads. We want to know how much memory each browser takes up before any pages are loaded, how much they take up after they load a bunch of pages, how much they take up after the windows/tabs are closed. CPU Usage. We should compare cpu usage of the browsers during the page load and idle.