Changes

All tests have an automatic version number, meaning that when the contents of a test change its results will no longer be used for comparison against other mis-matched results. This is taken care of, automatically, by the suite. For example, if you run the the test "3D Mesh Transformation v115" and then (at some point later, after changes ) run "3D Mesh Transformation v116" - and try to compare the results - no comparison will be allowed.

All test results are automatically saved to the server and stored in a database for later retrieval and analysis. This is important for a couple reasons:* Encoding test result data in the URL (such as via a query string) is not a scalable solution - and provides too little granularity.* Server-side storage provides limitless information store and can even store additional information to be used later (such as min, max, and deviation).* The simple URLs are easy to pass around are are quickly identifiable.* Simple test IDs make test comparisons trivial for the end user. This solution is much more scalable, and highly usable, providing a good long-term solution.

* sssWhen analyzing JavaScript performance it's important to make sure that all that you're testing is just the immediate performance of the script. Accidentally including factors that can lead to rendering effects can cause slowdowns from unrelated aspects of the browser. For example, in the SunSpider suite they loaded tests within iFrames, initiating timestamp-logging calls before and after the test completed. When [http://blog.vlad1.com/2008/02/28/finding-the-os-x-turbo-button/ coalesced updates] landed for Firefox 3 a significant performance improvement to the SunSpider results was found - even though the change had absolutely nothing to do with JavaScript performance, only actual rendering speed. To combat this issue all tests are pre-loaded by the browser and are run on-demand. This allows for no extra overhead and much more accurate results.