Platform/Games/GameFocusedBenchmarking

Benchmarks

Benchmarks require some porting effort before they can run in the automated framework. A list of benchmarks and their status is available here.

Benchmarking Framework

Test results are averaged over 20 iterations. The browser is restarted between each iteration. For test runs on mobile devices, the device is rebooted between iterations.

Tests for Windows and Linux are done on Dell Vostro 470 machines (Intel i5-3450, 4GB memory, Geforce GT620), running Windows 7 and Ubuntu 12.04.

All Android and FirefoxOS tests are done on Nexus 4 phones. This is to ensure a common baseline and have a device with enough memory to run all the benchmarks, many of which would not function on more resource-constrained devices.

Current Results

View [1] and click to drill down to details.

Scope

• Phase I - establish weekly measurements of the minimum viable product:
  • Tests we want to run.
  • Approved tests that are compatible with the framework.
  • List of phones already on hand.
  • List of phones/OS/browser combinations we need to add support to.
• Phase II:
  • Full automation
  • Remaining platforms (OSX)
  • Benchmark porting guide

People

• Martin Best - Sr. Engineering Program Manager - mbest@mozilla.com
• Robert Clary - Automation Developer - bclary@mozilla.com
• Alan Kligman - Platform Engineer - akligman@mozilla.com
• Joel Maher - Automation Developer - jmaher@mozilla.com
• Deb Richardson - Product Manager - deb@mozilla.com
• Milan Sreckovic - Engineering Manager, GFx - msreckovic@mozilla.com
• Clint Talbert - Automation & Tools Engineering Lead - ctalbert@mozilla.com
• Kannan Vijayan - Developer - kvijayan@mozilla.com
• Vladimir Vukicevic - Engineering Director - vladimir@mozilla.com
• Robert Wood - Automation Developer - rwood@mozilla.com

Communication

Project Team Meeting	Thursdays at 1:30 PT for 30 mins Vidyo Room: Vladimir Vukicevic's Vidyo Room Physical Meeting Room: TOR-5E Finch Invitation: Contact mmucci@mozilla.com to get added to the meeting invite list. Meeting Notes: Meeting Notes Etherpad
IRC	Server: irc.mozilla.org Channel: #games