Benchmarks

Information related to commonly used benchmarks and compliance measures for the web

Benchmarks

Bold means commonly used in media

Compliance

• HTML5Test
  • Targeted for "HTML5 Ready" apps/usage
  • Tests existence of certain APIs but not really compliance; easy to game and being gamed.
  • We disagree that the FileSystem API should be part of this suite, no vendor other than Google implements and they seem to have given up on this spec too.
• Ringmark
  • Targeted for mobile HTML5 app capabilities
  • Initiated by Facebook
  • https://bugzilla.mozilla.org/show_bug.cgi?id=748248
  • includes some WebKit-specific tests
  • Still maturing through W3C Coremob
• Acid3
  • All major browsers pass this
  • Should not be used as a performance test
• Test262
  • Javascript compliance
• CSS3 Selectors Test
  • All major browsers should pass
• Browserscope
  • community driven project initiated by Google
  • includes Ringmark
  • the rich-text numbers are somewhat arbitrary and a higher score doesn't necessarily mean a better implemenation
• W3C CSS Test Suites
  • hard to run, but results from others, for some of the test suites, are available (but separated by browser engine but not by version)
  • test suites in-development aren't separated from the stable ones
• Content Security Policy Compliance Test Suite
  • CSP is a way for documents to restrict where their content can come from
  • test suite is hosted at http://csptesting.herokuapp.com/
  • test suite source at https://github.com/eoftedal/csp-testing
  • Webkit and Gecko have implementations of CSP, IE 10 only implements the CSP sandbox directive (optional in CSP 1.0, required in CSP 1.1)
  • https://bugzilla.mozilla.org/show_bug.cgi?id=746978
  • CSP 1.0 Browser Compliance Testing - http://csptesting.herokuapp.com/?disable_old_headers=true and http://csptesting.herokuapp.com/

Performance

General

• Futuremark Peacekeeper
• BrowserMark
• Vellamo
  • Targeted for Android
  • Only tests built-in browser
  • No source code or detailed description of tests available
• RoboHornet
  • Brand new, seems to be in an alpha state

Graphics

• GUIMark2
  • Vector Charting Test
    • Dominated by line stroking performance
    • 2D canvas
  • Bitmap Gaming Test
    • 2D canvas
  • Text Column Test
• GUIMark3
  • A basic test of Canvas and video performance, largely focusing on comparing similar workloads between Flash and HTML5.
  • The result of each test is intended to be a framerate, with frames rendered via setInterval.
  • On a fast machine, results are meaningless -- likely to hit the throttled rate given the simple workload.
  • Bitmap Test
    • 2D canvas
  • Vector Test
    • 2D canvas
    • Dominated by radial gradient performance when not running with Direct2D
  • Compute Test
  • Video Test
• FishIE
  • Only tests drawImage()
  • http://muizelaar.blogspot.ca/2011/02/drawing-sprites-canvas-2d-vs-webgl.html
• PenguinMark

Javascript

• V8
• Octane
  • Essentially V8 version 8
    • Repackages various parts of V8
    • https://hacks.mozilla.org/2012/08/no-single-benchmark-for-the-web/
  • Uses WebBox2D that is slower than emscripten generated version
    • Alon is working on an emscripten generated alternative
• Spinning Balls
  • Used to test effectiveness of garbage collection

• Kraken
  • Mozilla developed
  • Essentially unmaintained
• Sunspider
  • Tests run very fast which makes measurement error significant and means that the tasks are not scaled to a significant amount of work.
    • Has 26 tests that run in 200-300ms, depending on the machine, so about 10ms per test. The differences between current browsers are now on the order of running a test in 9ms instead of 10ms, so practical significance is limited. Another problem with SunSpider is that with 10ms per tests, making a more advanced JIT doesn't improve your score, because the compile time ends up outweighing the improved run time. This doesn't matter so much for comparing browsers, but it means SunSpider doesn't really drive JS engines to get faster on big apps. If anything, it tells browsers to create new startup modes so they can run small programs a shade faster. (dmandelin)

DOM

• Dromaeo
  • Mozilla developed.
  • Maintenance status unclear; no reasonable contact info for how to get the deployed benchmark updated.
  • Source lives at https://github.com/jeresig/dromaeo
  • Microbenchmarks that are not representative of any real workloads and in fact are often completely bogus.
  • Easily gamed (and has been actively gamed in the past by UAs).
  • Incentivizes making already fast stuff faster more than making slow stuff faster.
  • Often doesn't measure what it claims to be measuring.
  • Best (because pretty much only, not because it's so good) DOM benchmark around.

Page Load

• WebPageTest
  • seems very good
  • cross-browser
  • simulates different networks
• iBench
  • uses onLoad() which is not credible for performance
  • not available anymore.

Pan/Zoom

• Eideticker
  • Mozilla developed
• Chalkboard
  • Does CSS transforms on a single SVG image
    • Tests rasterization of SVG images
  • Not that interesting for real pan and zoom

Other

• MazeSolver
  • bug 776190
  • Not clear it tests anything useful for end users
• JSGamebench
  • Initiated by Facebook
• Asteroids

Benchmark Aggregators

• Tom's Hardware Guide