Benchmarks

Information related to commonly used benchmarks and compliance measures for the web

Benchmarks

Bold means commonly used in media

Compliance

• HTML5Test
  • Targeted for "HTML5 Ready" apps/usage
  • Tests existence of certain APIs but not full compliance
  • We disagree that the FileSystem API should be part of this suite, no vendor other than Google implements and they seem to have given up on this spec too.
• Ringmark
  • Targeted for mobile HTML5 app capabilities
  • Initiated by Facebook
  • https://bugzilla.mozilla.org/show_bug.cgi?id=748248
  • includes some WebKit-specific tests
  • Still maturing through W3C Coremob
• Acid3
  • All major browsers pass this
  • Should not be used as a performance test
• Test262
  • Javascript compliance
• CSS3 Selectors Test
  • All major browsers should pass
• Browserscope
  • community driven project initiated by Google
  • includes Ringmark
  • the rich-text numbers are somewhat arbitrary and a higher score doesn't necessarily mean a better implemenation
• W3C CSS Test Suites
  • hard to run, but results from others, for some of the test suites, are available (but separated by browser engine but not by version)
  • test suites in-development aren't separated from the stable ones
• Content Security Policy Compliance Test Suite
  • CSP is a way for documents to restrict where their content can come from
  • test suite is hosted at http://csptesting.herokuapp.com/
  • test suite source at https://github.com/eoftedal/csp-testing
  • Webkit and Gecko have implementations of CSP, IE 10 only implements the CSP sandbox directive (optional in CSP 1.0, required in CSP 1.1)
  • https://bugzilla.mozilla.org/show_bug.cgi?id=746978
  • CSP 1.0 Browser Compliance Testing - http://csptesting.herokuapp.com/?disable_old_headers=true and http://csptesting.herokuapp.com/

Performance

General

• Futuremark Peacekeeper
• BrowserMark
• Vellamo
  • Targeted for Android
  • Only tests built-in browser
  • No source code or detailed description of tests available

Graphics

• GUIMark2
  • Vector Charting Test
    • Dominated by line stroking performance
    • 2D canvas
  • Bitmap Gaming Test
    • 2D canvas
  • Text Column Test
• GUIMark3
  • Bitmap Test
    • 2D canvas
  • Vector Test
    • 2D canvas
    • Dominated by radial gradient performance.
  • Compute Test
  • Video Test
• FishIE
  • Only tests drawImage()
  • http://muizelaar.blogspot.ca/2011/02/drawing-sprites-canvas-2d-vs-webgl.html

Javascript

• V8
• Octane
  • Essentially V8 version 8
    • Repackages various parts of V8
    • https://hacks.mozilla.org/2012/08/no-single-benchmark-for-the-web/
  • Uses WebBox2D that is slower than emscripten generated version
    • Alon is working on an emscripten generated alternative

• Kraken
  • Mozilla developed
  • Essentially unmaintained
• Sunspider
  • Tests run very fast which makes measurement error significant and means that the tasks are not scaled to a significant amount of work.
    • Has 26 tests that run in 200-300ms, depending on the machine, so about 10ms per test. The differences between current browsers are now on the order of running a test in 9ms instead of 10ms, so practical significance is limited. Another problem with SunSpider is that with 10ms per tests, making a more advanced JIT doesn't improve your score, because the compile time ends up outweighing the improved run time. This doesn't matter so much for comparing browsers, but it means SunSpider doesn't really drive JS engines to get faster on big apps. If anything, it tells browsers to create new startup modes so they can run small programs a shade faster. (dmandelin)

DOM

• Dromaeo
  • Mozilla developed

Page Load

• iBench
  • uses onLoad() which is not credible for performance
  • not available anymore.

Pan/Zoom

• Eideticker
  • Mozilla developed

Other

• MazeSolver
  • bug 776190
  • Not clear it tests anything useful for end users
• JSGamebench
  • Initiated by Facebook
• Asteroids

Benchmark Aggregators

• Tom's Hardware Guide