Platform/Games/Testing/32v64bit

From MozillaWiki
< Platform‎ | Games‎ | Testing
Jump to: navigation, search

Summary

The following documents A-B testing comparing performance of 32-bit Firefox builds to 64-bit Firefox builds, utilizing the Massive benchmark.

Methodology

  1. Test using the same system, make note of the system profile
  2. Install the latest Firefox Nightly 32-bit build and start with a new profile
  3. Load http://kripken.github.io/Massive/ and click "Run the benchmark now!"
  4. Let the benchmark run
  5. Record the results below when the benchmark completes
  6. Include a screenshot of the result
  7. Repeat the benchmark 5 times on the same system and record the result separately
  8. Install the latest Firefox Nightly 64-bit and repeat the tests with a new profile

System Profile

Operating System Windows 7 64-bit
Firefox (user agent string with build ID) Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0, 20140604030202

Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:32.0) Gecko/20100101 Firefox/32.0, 20140604030202

Processor (make, model, speed) AMD FX(tm)-8320 Eight-Core Processor 3.50 Ghz
Memory (type, speed, amount) 8 GB DDR3 1333 Mhz
GPU (make, model, speed, memory, driver version) AMD Radeon R7 200 Series, Core Clock 780 Mhz, 2GB DDR3 900 Mhz, Catalyst version 13.12

Results

Testrun #1

Benchmark Firefox 32-bit Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold 0.452 0.348
main-thread-poppler-warm 0.428 0.372
main-thread-sqlite-cold 0.053 (±26%!) 0.062 (±25%!)
main-thread-sqlite-warm 0.033 (±24%!) 0.048 (±29%!)
Throughput
box2d-throughput 6.838 4.882
box2d-throughput-f32 6.316 4.567
lua-binarytrees 8.067 7.447
lua-scimark 10.545 11.730
poppler-throughput 7.008 6.445
sqlite-throughput 10.891 8.773
Preparation
poppler-cold-preparation 0.939 0.926
poppler-warm-preparation 0.185 0.185
sqlite-cold-preparation 1.084 1.347
sqlite-warm-preparation 0.098 0.123
Variance
box2d-variance 1.466 1.261
poppler-variance 4.843 4.395
Overall Score 3,967 4,120

Testrun #2

Benchmark Firefox 32-bit Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold 0.461 0.367
main-thread-poppler-warm 0.427 0.361
main-thread-sqlite-cold 0.065 0.064
main-thread-sqlite-warm 0.040 (±30%!) 0.043 (±28%!)
Throughput
box2d-throughput 6.987 4.666
box2d-throughput-f32 6.317 4.277
lua-binarytrees 7.899 7.182
lua-scimark 10.555 12.055
poppler-throughput 7.023 5.959
sqlite-throughput 10.733 8.485
Preparation
poppler-cold-preparation 0.953 0.846
poppler-warm-preparation 0.186 0.180
sqlite-cold-preparation 1.103 1.209
sqlite-warm-preparation 0.109 0.115
Variance
box2d-variance 1.758 1.290
poppler-variance 4.830 4.036
Overall Score 3,839 4,287

Testrun #3

Benchmark Firefox 32-bit Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold 0.456 0.338
main-thread-poppler-warm 0.445 0.356
main-thread-sqlite-cold 0.050 0.063
main-thread-sqlite-warm 0.034 (±24%!) 0.042 (±28%!)
Throughput
box2d-throughput 6.806 4.643
box2d-throughput-f32 6.316 4.284
lua-binarytrees 8.002 7.239
lua-scimark 10.250 12.185
poppler-throughput 7.034 6.003
sqlite-throughput 10.759 8.366
Preparation
poppler-cold-preparation 0.937 0.842
poppler-warm-preparation 0.183 0.173
sqlite-cold-preparation 1.071 1.203
sqlite-warm-preparation 0.110 0.121
Variance
box2d-variance 1.467 1.210
poppler-variance 4.836 4.083
Overall Score 3,942 4,323

Testrun #4

Benchmark Firefox 32-bit Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold 0.448 0.359
main-thread-poppler-warm 0.451 0.362
main-thread-sqlite-cold 0.056 (±22%!) 0.071 (±31%!)
main-thread-sqlite-warm 0.033 (±22%!) 0.042 (±26%!)
Throughput
box2d-throughput 6.834 4.674
box2d-throughput-f32 6.317 4.304
lua-binarytrees 8.219 7.226
lua-scimark 9.925 12.005
poppler-throughput 7.021 5.951
sqlite-throughput 10.768 8.414
Preparation
poppler-cold-preparation 0.953 0.840
poppler-warm-preparation 0.184 0.173
sqlite-cold-preparation 1.091 1.212
sqlite-warm-preparation 0.120 0.140
Variance
box2d-variance 1.546 2.258
poppler-variance 4.832 4.067
Overall Score 3,869 4,225

Testrun #5

Benchmark Firefox 32-bit Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold 0.453 0.360
main-thread-poppler-warm 0.453 0.351
main-thread-sqlite-cold 0.047 (±21%!) 0.060 (±27%!)
main-thread-sqlite-warm 0.038 (±26%!) 0.052 (±40%!)
Throughput
box2d-throughput 7.161 4.666
box2d-throughput-f32 6.589 4.292
lua-binarytrees 8.364 7.230
lua-scimark 9.820 12.195
poppler-throughput 7.596 5.989
sqlite-throughput 11.282 8.583
Preparation
poppler-cold-preparation 1.035 0.844
poppler-warm-preparation 0.199 0.177
sqlite-cold-preparation 1.232 1.236
sqlite-warm-preparation 0.112 0.137
Variance
box2d-variance 1.967 1.297
poppler-variance 5.149 4.086
Overall Score 3,762 4,211

Overall

Build Lowest Score Highest Score Median Score Average Score
Firefox 32.0a1 32-bit 3,762 3,967 3,869 3875.8
Firefox 32.0a1 64-bit 4,120 4,323 4,225 4233.2

Best score is highlighted

Notes

The benchmark freezes very often at the "sqlite-warm-preparation" test. I had to right click on the page, or to select a word, open/close a new tab, or stuff like that to make it "refresh" and continue

Conclusions

  • Overall, 64-bit builds seem to perform about 9% better than 32-bit builds
  • 32-bit builds perform about 7% to 21% better on sqlite benchmarks