Background

So, for B2G we absolutely must do testing on phones. We have to test making cellular calls, text messages etc. And we have to test on-the-phone performance. We have a similar situation on Android as well where we do some performance and correctness testing on a range of actual android phones, but supporting those is a second ask from this same hardware setup I'm proposing for B2G.

We have rigs for this in Mountain View because right now, while the phones have a 5% failure rate, that means that someone actually has to physically pull the battery out of one or another of these things 2-5 times a week depending on the number of jobs we run that week.

From what I've heard from product and QA, we will be scaling our on-phone automated testing to 3 lines of phones (each line represents a shipping branch of the b2g codebase) and each line will have about 10 phones. The one gotcha here is security fixes and ongoing support. No one at the moment seems to know or say how long we are supporting these builds with security fixes. If we do the standard ESR style support then we could be looking at these lines growing to something like 6-8 by this time next year. (We'd be supporting 3 current releases, plus 3-5 old releases).

So, if we do some worst case planning we need to find space for about 60-80 phones by this time next year.

Project Requirements

• Need an open wireless network for phones to join with extremely low latency and low contention (it's used for performance tests) in the datacenter
• Need a cellular provider for phones to use (they will also need sims, the ones currently running already have them)
• Need a means to power cycle the phone
• Need a means to flash the phone (currently, each phone is connected to one mac mini, and that mac mini flashes the phone using adb directly) - we want to move away from that but b2g has a bug where all phones have the same serial ID in ADB and therefore are not singularly addressable. We cannot as yet flash from the sutagent, but perhaps that is worth investigating as an alternative on these b2g phones. Or, we fix the b2g bug and use adb. (bug xxx)
• Phone needs to have the camera unblocked for camera automation
• Phone needs headphone jack connected (or simulated to be connected) for testing fm radio application
• Phone rack needs to be able to supply different voltages and amperage levels per device (these change from design to design) Right now range is: 3.7V - 4.2V and 1400mAh - 1540mAh
• Phone rack needs to adapt to different areas for the battery leads (these change from design to design)
• Phone rack needs to adapt to different locations for power button (changes from design to design)
• Phone rack should also not block front-facing cameras (I'm sure we'll have those at some point)
• Phone racks should support phones of different physical sizes (we will likely want to do this with our collection of Android phones as well since they are also going to need to leave the mountain view office).
• Phone needs to be easy to extract from the setup (or perhaps the chassis connected to the phone is detachable from the larger rackmounted unit
• The phones should fit in a rack mountable system. - you probably have specific requirements for the U's they take up
• Phone chassis and rack mount for batch of phones should be designed so as to cause minimal interference with wi-fi signal
• Mozpool should keep track of these devices, even if it cannot flash them
• Flashing story needs to be sorted out - one mini per device is a non-starter.
• Ideally, mozpool should own flashing the device (whether that is through uboot or through mozpool directing a computer with phones connected via usb to flash them, I'm happy)

Progress

We will work on this over multiple quarters.

• 2013 Q2 Elements