Performance/Snappy/Testing:BFCache Sprint

From MozillaWiki
Jump to: navigation, search

Overview

The purpose of this sprint is to gather some initial baseline data to determine how often Firefox is actually using BFCache

Background

BF cache is what makes pages appear instantly when pressing back (ie. there is no incremental page layout, image loading, etc). Other types of cache include disk cache and memory cache.

For our needs, we just need to make sure disk cache is empty at startup (rebooting OS before-hand) so that we get a fair comparison. Once we have a stable test for comparing network traffic, we should also profile disk cache using xperf.

For more background, read this blog post.

Testing

Baseline Testing

Overview

  • testing click-through perf on several pages
  • time to click link to throbber stopping
    • click next link when throbber stops
  • screencast of testing (cell phone, or software)
  • running test in parallel
    • 2 people run the test on the same hardware, differing browsers
  • clear cache between testing
  • ebay, amazon, or something that is not so dynamic
  • looking for one browser being slower than another, interest is in timing
  • start with a couple QA then branch to community
  • timeline ~2 weeks from now
  • timings in spreadsheet, screencasts stored in a people account

Hardware

  • low-end on Windows 7 (ie. netbook) -- x120e is the best candidate
  • make sure harddrive light is off
  • indexing / scans should not be running (ie. Antivirus)
  • use ethernet on the same network
  • run in QA lab

Reference Test Case

The following should be run twice, once on Firefox Nightly, once on Opera.

Preliminary

  1. Install browser
  2. Start with a new profile
  3. Create a bookmark for Amazon.com, Huffingtonpost.com and Tomshardware.com
  4. Clear the cache
  5. Make sure no hard drive activity
  6. Report all results in this spreadsheet (times in ms) -- if possible provide a link to a screencast

What to Test

  1. Start the timer (use your cellphone if it helps)
  2. Open one of the websites via bookmark
  3. Pause the timer once the throbber stops and record the time
  4. Resume the timer and click the specified link
  5. Pause the timer once the throbber stops and record the time
  6. Repeat steps 4 and 5 for each link specified

How to Test

  1. Run the test for all three sites in Firefox
  2. Quit Firefox and start Opera
  3. Run the test for all three sites in Opera
  4. Quit Opera and start Firefox
  5. Run the test for all three sites in Firefox
  6. Quit Firefox and start Opera
  7. Run the test for all three sites in Opera
  8. Quit Opera

Amazon.com Checkpoints

  1. Initial load of Amazon.com
  2. Help
  3. Shipping Rates & Information
  4. International
  5. Other Help Areas > Kindle
  6. Kindle Keyboard
  7. Wireless Coverage Map
  8. Careers
  9. Code Ninjas
  10. FAQ

Huffingtonpost.com Checkpoints

  1. Initial load of Huffingtonpost.com
  2. User Agreement (bottom of the page)
  3. Privacy (bottom of the page)
  4. About Us (bottom of the page)
  5. Front Page
  6. Politics
  7. Business
  8. World
  9. Living
  10. Tech

Tomshardware.com Checkpoints

  1. Initial load of Tomshardware.com
  2. Articles
  3. Components
  4. How to
  5. Modding
  6. "In Pictures: 19 Ways To Recycle Old, Unused Hardware" from Oct 16, 2010
  7. Cycle through all of the 19 pictures (use the > button above the picture)
  8. Legal (at the bottom of the page)
  9. Charts > HDD Charts 2012
  10. Desktop Performance benchmark

Sprints

Preparing your System
  • Download and install the Windows SDK (this will give you the Windows Performance Toolkit which includes xperf.exe)
  • Download Wireshark
Phase 1 - xperf on Windows 7
  • Start by restarting your computer
  • Run cmd.exe as Administrator
  • Run xperf -on latency+FILE_IO+FILE_IO_INIT+DISK_IO
  • Start Firefox
    • Navigate to your test page
    • Press the back button
    • Press the forward button
  • Run xperf -d report.etl
  • Run xperf report.etl to view the report

Need somewhere for testers to post their reports

Phase 2 - wireshark
  • Record a wireshark dump of your browsing session
  • Filter out the non-web traffic
  • Submit reports comparing Firefox to Opera
Sites to Test

Top 10 sites not requiring authentication

  • www.google.com
  • www.youtube.com
  • www.yahoo.com
  • www.baidu.com
  • www.wikipedia.com
  • www.amazon.com
  • www.qq.com
  • www.msn.com
  • www.ebay.com
  • www.bing.com
Retrieved from "https://wiki.mozilla.org/index.php?title=Performance/Snappy/Testing:BFCache_Sprint&oldid=385176"