Dromaeo is currently considered to be a "work in progress." It's been brought forward for review to the other browser vendors and some concerns were raised, namely:
- The number of test iterations that are run. Currently each test is run, at least, 5 times - it's been requested that this be increased.
- The calculation of error ranges. Currently the data is fit to a t-distribution, it's been requested that it use a two-variable t-test.
How to Use It
Visiting the main Dromaeo page you are given a collection of tests which can be run. Generally these tests are designed to be more "real world" in nature (testing a number of features simultaneously). Each test should have a full description explaining what the test is achieving along with an indication of what is being tested.
If you don't wish to run all of the available tests, and only run a sub-section of them, you can filter tests via the URL, like so:
- http://dromaeo.com/?3d (this will only run 3d-related tests)
- http://dromaeo.com/?string (this will only run String-related tests)
You have full regular expression support so something like the following will work, as well:
- http://dromaeo.com/?(3d|string) (only run 3d or String-related tests)
Running the Tests
To begin you can run the tests by simply hitting the 'Run' button. It should be capable of running in Firefox 2+, Safari 3+, and Opera 9+, and Internet Explorer 6+. Please report any problems that you may have.
Note: Currently Internet Explorer 6 and 7 run incredibly slow on the DNA sequence test (taking approximately 15-20 minutes).
You can pause/resume the tests at any time, using the 'Pause' button (it won't affect the final numbers).
Viewing the Results
After the tests have finished running you'll be presented with a full breakdown of the tests (including sub-tests). Each individual sub-test will have its mean value presented in conjunction with an error spread of the results.
Additionally, you'll be given a URL for your results, saved on the server, which you can refer back to at any point, like the following:
- http://dromaeo.com/?id=223 (A single Firefox run.)
Once you've received a URL (and associated unique ID) for your test results you can then use it to compare against other result sets , like so:
- http://dromaeo.com/?id=222,223,219 (Compares Safari 3.1, Firefox 3.0b5, and Opera 9.5 nightly runs on OSX.)
- http://dromaeo.com/?id=232,226,229,237 (Compares Safari 3.1, Firefox 3.0b5, Opera 9.5 nightly, and IE 8b1 runs on Windows XP.)
- http://dromaeo.com/?id=250,246,251,256 (Compares Safari 3.1, Firefox 3.0b5, Opera 9.5 nightly, and IE 8b1 runs on Windows Vista.)
- http://dromaeo.com/?id=234,236,237 (Compares IE 6, IE 7, and IE 8 on Windows XP.)
Downloading the Suite
If you wish to run the suite offline you can download it from the following location:
The result should be identical to the suite running on dromaeo.com. You'll need to have PHP support if you wish to save results to the central server.
Adding New Tests
There are a number of techniques that the Dromaeo suite uses in order to achieve accurate results. Together they provide a solid foundation for allowing significant performance analysis to be completed.
All tests have an automatic version number, meaning that when the contents of a test change its results will no longer be used for comparison against other mis-matched results. This is taken care of, automatically, by the suite. For example, if you run the the test "3D Mesh Transformation v115" and then (at some point later, after changes) run "3D Mesh Transformation v116" and try to compare the results - no comparison will be allowed.
This is especially important as it allows for a tangible upgrade process to exist, for tests, considering that bugs or adjustments will inevitably arise causing some amount of conflict to occur. With this being completely baked into the process
All test results are automatically saved to the server and stored in a database for later retrieval and analysis. This is important for a couple reasons:
- Encoding test result data in the URL (such as via a query string) is not a scalable solution - and provides too little granularity.
- Server-side storage provides limitless information store and can even store additional information to be used later (such as min, max, and deviation).
- The simple URLs are easy to pass around are are quickly identifiable.
- Simple test IDs make test comparisons trivial for the end user.
This solution is much more scalable, and highly usable, providing a good long-term solution.
A number of checks are put in place to make sure that statistically significant results can be provided to the user. Some of the techniques used are:
- All tests are run, at minimum, 5 times. If a significant level of error cannot be reached then more runs are completed (up to 10).
- All results are fit on to a T-Distrubtion, set to a 95% confidence interval (values from this table). 95% confident +/- error intervals are provided with all results.
- All comparisons between results take into account the error intervals - offering a tie for result sets that are statistically tied within the associated error ranges.
The sum result is a system that's able to provide consistent results and present them in a fair, and meaningful, way.
Script Speed vs. Rendering Speed
To combat this issue all tests are pre-loaded by the browser and are run on-demand. This allows for no extra overhead and much more accurate results.
The vast majority of the tests are brought in from the Computer Language Shootout and adapted to run within the suite (often with greater levels of complexity).