QA/Fennec/PerformanceBenchmark
Summary
The objective here is to run a few performance tests and measure various startup and page load times on fennec betas. We will trend the performance against previous versions as well as stock browsers. We will measure CPU and memory usage when the application has been functionally operating over time. We will also measure performance when operating on sites that provide standard javascript and css supported pages.
Environment and Setup
- Environment
- Maemo 5. N900
- Compare against Stock Maemo 5 browser
- compare against Fennec 1.1
- Android 2.1 or 2.2. Vibrant, Droid, Nexus One, Captivate, Droid 2
- Compare against Stock Android browser
- Watching CPU and Memory
- There are various tools you can use to watch the CPU and memory usage of Fennec. The easiest way is to SSH into the device and use various tools to monitor the progression.
- On Maemo, you can download OpenSSH from the application Manager, connect to a computer, and use top to watch the cpu and memory
- On Android, you can download Process Manager from the Market, and run that in the background. You can also launch Eclipse's ADT plugin and connect via USB
Resources
- Waverley. <Ioana>
Test Plan
Refer to Performance Spreadsheet for details and reporting.
1. Fennec startup Time
- Record startup time from cold start to home page
- Record startup time after restarting after applying an addon
2. Pageload time
- Pick 10 top US sites and record pageload times
- Pick 10 top International sites and record pageload times
- PR 1.2 vs. PR 1.3
3. Measure Memory Usage (Process Manager App)
- Record memory size at startup (home page only)
- Open 5 tabs and record memory size
- Play an Ogg video and record memory size
- Launch a news RSS feed and record memory size
- Close Fennec and record memory size
Test Process (details)
- Start-up and Page load times
- each test was measured from the perspective of human perception
- used Rob Keir's millisecond timer app
- for each browser version, measured start up time at least 3 times and computed an average.
- for each site in Top 10 US list, measured load time at least 3 times and computed an average
- for top International sites, picked 10 sites that were among top 10 sites in Romania, Germany and France and were not already in Top 10 US list. Measured load times as described above for sites in Top 10 US list.
- Memory usage
| Test | Fennec 1.1 | Fennec beta 2 | Native Browser |
|---|---|---|---|
| Startup | - used the following script:
#!/bin/sh
while [ true ]; do
- computed an average using numbers from resulting file |
- started browser manually and while doing that ran a script containing the following:
#!/bin/sh while [ true ]; do
- computed an average using numbers from resulting file | |
| 5 Tabs | - opened in separate tabs: google.com, yahoo.com, wikipedia.com, amazon.com, twitter.com
- in parallel, ran the following script: while [ true ]; do
- computed an average using numbers from resulting file |
- ran the following script:
#!/bin/sh
while [ true ]; do
- computed an average using numbers from resulting file |
- opened in separate windows: google.com, yahoo.com, wikipedia.com, amazon.com, twitter.com
- in parallel, ran the following script: while [ true ]; do
- computed an average using numbers from resulting file |
Results
- Report results in the spreadsheet. but describe observations here.
References
- bug 606574 needs to be added to an extension to measure responsiveness. Need resources.
- Top US websites (via Alexa.com)
- Top Countries websites (via Alexa.com)