QA/Fennec/PerformanceBenchmark

Summary

The objective here is to run a few performance tests and measure various startup and page load times on fennec betas. We will trend the performance against previous versions as well as stock browsers. We will measure CPU and memory usage when the application has been functionally operating over time. We will also measure performance when operating on sites that provide standard javascript and css supported pages.

Environment and Setup

Environment

• Maemo 5. N900
  • Compare against Stock Maemo 5 browser
  • compare against Fennec 1.1
• Android 2.1 or 2.2. Vibrant, Droid, Nexus One, Captivate, Droid 2
  • Compare against Stock Android browser

Watching CPU and Memory

• There are various tools you can use to watch the CPU and memory usage of Fennec. The easiest way is to SSH into the device and use various tools to monitor the progression.
  
• On Maemo, you can download OpenSSH from the application Manager, connect to a computer, and use top to watch the cpu and memory
  
• On Android, you can download Process Manager from the Market, and run that in the background. You can also launch Eclipse's ADT plugin and connect via USB

Resources

• Waverley. <Ioana>

Test Plan

Refer to Performance Spreadsheet for details and reporting.

1. Fennec startup Time

• Record startup time from cold start to home page
• Record startup time after restarting after applying an addon

2. Pageload time

• Pick 10 top US sites and record pageload times
• Pick 10 top International sites and record pageload times
• PR 1.2 vs. PR 1.3

3. Measure Memory Usage (Process Manager App)

• Record memory size at startup (home page only)
• Open 5 tabs and record memory size
• Play an Ogg video and record memory size
• Launch a news RSS feed and record memory size
• Close Fennec and record memory size

Test Process (details)

Start-up and Page load times

• each test was measured from the perspective of human perception.
• used Rob Keir's millisecond timer app
• used a Wi-Fi connection
• for each browser version, measured start up time at least 3 times. Any obvious outliers were removed and replaced (the variance between times is 1 second at most). Computed an average from those times.
• for each site in Top 10 US list, measured load time at least 3 times. Any obvious outliers were removed and replaced (the variance between times is 1 second at most). Computed an average from those times.
• for top International sites, picked 10 sites that were among top 10 sites in Romania, Germany and France and were not already in Top 10 US list. Measured load times as described above for sites in Top 10 US list.

Note: For Nokia N900, also tried to measure start-up times using sp-startup-time and syslog utilities, but the results obtained were approximately 5 times smaller than the ones in spreadsheet.

Memory usage

Nokia N900

Test	Fennec 1.1	Fennec 4 beta 2	Native Browser
Startup	- used the following script: #!/bin/sh /usr/bin/fennec >/dev/null & while [ true ]; do   top -b -n 1| grep fennec | awk '{totuse=totuse + $7, totuseMem=totuseMem+$6} END { print totuse,totuseMem }' >> /home/user/StartupFennec    sleep 1 done - computed an average using numbers from resulting file		- started browser manually and while doing that ran a script containing the following: #!/bin/sh while [ true ]; do   top -b -n 1| grep browser | awk '{totuse=totuse + $7, totuseMem=totuseMem+$6} END { print totuse,totuseMem }' >> /home/user/StartupBrowser    sleep 1 done - computed an average using numbers from resulting file
5 Tabs	- opened in separate tabs: google.com, yahoo.com, wikipedia.com, amazon.com, twitter.com - in parallel, ran the following script: while [ true ]; do   top -b -n 1| grep fennec | awk '{totuse=totuse + $7, totuseMem=totuseMem+$6} END { print totuse,totuseMem }' >> /home/user/StartupFennec    sleep 10 done - computed an average using numbers from resulting file	- ran the following script: #!/bin/sh /usr/bin/fennec http://www.google.com http://yahoo.com http://wikipedia.org http://amazon.com http://twitter.com >/dev/null & while [ true ]; do   top -b -n 1| grep fennec | awk '{totuse=totuse + $7, totuseMem=totuseMem+$6} END { print totuse,totuseMem }' >> /home/user/StartupFennec    sleep 1 done - computed an average using numbers from resulting file	- opened in separate windows: google.com, yahoo.com, wikipedia.com, amazon.com, twitter.com - in parallel, ran the following script: while [ true ]; do   top -b -n 1| grep browser | awk '{totuse=totuse + $7, totuseMem=totuseMem+$6} END { print totuse,totuseMem }' >> /home/user/StartupBrowser    sleep 10 done - computed an average using numbers from resulting file
Ogg video	- used the following script: #!/bin/sh /usr/bin/fennec http://videos.mozilla.org/firefox/3.6/getpersonas.ogv >/dev/null & while [ true ]; do   top -b -n 1| grep fennec | awk '{totuse=totuse + $7, totuseMem=totuseMem+$6} END { print totuse,totuseMem }' >> /home/user/oggFennec    sleep 10 done - computed an average using numbers from resulting file		- browser does not open .ogg files
Close	- started browser and opened 2-3 tabs/windows - in parallel, ran the following: Fennec #!/bin/sh while [ true ]; do   top -b -n 1| grep fennec | awk '{totuse=totuse + $7, totuseMem=totuseMem+$6} END { print totuse,totuseMem }' >> /home/user/CloseFennec    sleep 1 done Native Browser #!/bin/sh while [ true ]; do   top -b -n 1| grep browser | awk '{totuse=totuse + $7, totuseMem=totuseMem+$6} END { print totuse,totuseMem }' >> /home/user/CloseFennec    sleep 1 done - computed an average using numbers from resulting file

Android

• for each browser version, measured the start up CPU at least 3 times and computed an average.
• for the 5 tabs test, the steps were the following: open all 5 tabs at the same time (tabs from last time) and measured the CPU at least 3 times and computed an average.
• for video test I used: http://videos.mozilla.org/firefox/3.6/getpersonas.ogv
• for the CPU measurement:
  • used CPU Usage by TwistByte LLC
• for the memory usage :
  • used Process Manager by CurveFish
• opened in separate tabs: google.com, yahoo.com, wikipedia.com, amazon.com, twitter.com

Results

Start-up and Page load times

• Nokia N900
  • time needed for cold start is almost equal among Fennec builds, but considerably bigger compared to native browser's time
  • pages load more quickly on Fennec 1.1

Memory usage

• Nokia N900
  • Fennec 4 beta 2 uses more resources than Fennec 1.1

References

• bug 606574 needs to be added to an extension to measure responsiveness. Need resources.
• Top US websites (via Alexa.com)
• Top Countries websites (via Alexa.com)