QA/Fennec/PerformanceBenchmark

Summary

The objective here is to run a few performance tests and measure various startup and page load times on fennec betas. We will trend the performance against previous versions as well as stock browsers. We will measure CPU and memory usage when the application has been functionally operating over time. We will also measure performance when operating on sites that provide standard javascript and css supported pages.

Environment and Setup

Environment

• Maemo 5. N900
  • Compare against Stock Maemo 5 browser
  • compare against Fennec 1.1
• Android 2.1 or 2.2. Vibrant, Droid, Nexus One, Captivate, Droid 2
  • Compare against Stock Android browser
  • Compare against Alpha 2

Watching CPU and Memory

• There are various tools you can use to watch the CPU and memory usage of Fennec. The easiest way is to SSH into the device and use various tools to monitor the progression.
  
• On Maemo, you can download OpenSSH from the application Manager, connect to a computer, and use top to watch the cpu and memory
  
• On Android, you can download Process Manager from the Market, and run that in the background. You can also launch Eclipse's ADT plugin and connect via USB

Resources

• Waverley. <tester>

Test Plan

Refer to Performance Spreadsheet for details and reporting.

1. Fennec startup Time

• Record startup time from cold start to home page
• Record startup time after restarting after applying an addon

2. Pageload time

• Pick 10 top US sites and record pageload times
• Pick 10 top International sites and record pageload times
• PR 1.2 vs. PR 1.3

3. Page navigation time

• Pick 10 top US sites and switch to 2-3 links
• Pick 10 top International sites and switch to 2-3 links
• Record page navigation time

4. Measure Memory Usage (Process Manager App)

• Record memory size at startup (home page only)
• Open 5 tabs and record memory size
• Play an Ogg video and record memory size
• Launch a news RSS feed and record memory size
• Close Fennec and record memory size

Results

• Report results in the spreadsheet. but describe observations here.

References

• bug 606574 needs to be added to an extension to measure responsiveness. Need resources.
• Top US websites (via Alexa.com)
• Top Countries websites (via Alexa.com)