Platform priority

The following platform priority list is from a recent Channel meeting, but has been extended to include existing platforms and new platforms that are already in-flight:

• OS X 10.8 signing of 10.7 builds
• unified Windows builders (single 64bit image that can create Win8, win64, win32-on-win64 builds to prevent linker issues in the future)
• Windows 8 build hardware
• Windows 8 test hardware
• b2g
• unified Linux builders (single 64bit image+mock)
• Rapid Betas
• mobile test hardware (pandaboards)
• builds for armv6
• builds for android-on-intel
• Mac 10.8 test hardware
• Mac 10.8 build hardware?
• new Windows test hardware
• new Linux test hardware

Major goals

• tackle many targets in parallel rather than serially as in the past
  • requires investing more than one human resource
• virtualize as much hardware as is possible
  • caveats:
    • only works for building and unittests(correctness), not talos (performance)
    • build times must be competitive with hardware:
      • perf: 25% slower may be acceptable, order-of-magnitude is not
      • cost: costs of bulk AWS VMs? costs of VMs (incl ESX lic.), netapps, hosts?
    • should VMs be able to run on both AWS and Mozilla hardware?
• Queue collapsing
  • currently using queue collapsing to mitigate lack of capacity.
  • should have enough capacity that queue collapsing is again considered an error condition; setup nagios alerts when queue collapsing triggered.

Plan of attack

1. acquire signing key from Apple
   • releng: joduinn, catlee
2. setup 10.8 signing machine
   • https://bugzilla.mozilla.org/show_bug.cgi?id=744789
   • releng: bhearsum
3. segregate 5 Win64 build machines to be exclusively used as Win8 builders until a unified image is available
   • unblocks developers immediately
   • tools can be installed by hand until GPO is ready
   • releng: armenzg; IT: MaRu
   • https://bugzilla.mozilla.org/show_bug.cgi?id=754008
   • https://bugzilla.mozilla.org/show_bug.cgi?id=737994
4. create single linux64 VM that can build all current linux-based builds: linux32, linux64, android*, b2g
   • mock envs for each build type
     • catlee already has an image on AWS that does almost all of this AIUI. Still many services issues to be worked out there though.
   • based on something more recent than CentOS 5 (CentOS 6?)
   • virtualizing b2g builders frees up DL120 hardware for use as Win8 testers
   • virtualizing all linux builders frees up ix machines for use as Windows builders
   • releng: catlee, bear; IT: arr
5. create single Win64 hardware image that can build all current Windows-based builds: Win32, Win64, Win8
   • image all ix machines with new Win64 image
   • releng: catlee, armenzg; IT: MaRu
   • https://bugzilla.mozilla.org/show_bug.cgi?id=753132
6. setup DL120s as Win8 testers
   • buy graphics cards for existing DL120s (have 3, need 50)
   • determine how many more systems we need to purchase
7. rackmount enclosure for pandaboards
   • still blocked on initial automation work by A-Team
8. create mock env for armv6
9. create mock env for android-on-intel

Open questions

• switching (back!) to VMs for unittests
  • switching unittests from VMs to hardware reduce intermittent oranges.
  • is this practical?
• where should VM capacity come from (AWS vs Mozilla IT vs ???)
• capacity for existing platforms. Do we need more:
  • tegras?
  • rev5 builders?
  • rev5 testers?
• new hardware for general Linux and Windows testing?
  • same hardware between platforms? (no requirement that it be the same from the dev side)
  • DL120s?
  • how many, and for which OSes?