Firefox OS/Performance/Power/Testing

Goals

The goals of the initial power consumption testing are as follows:

• Baseline version 1.1 power consumption performance for B2G on Hamatchi, Nexus S, and Nexus 4 hardware.
  • Detect power consumption regressions in 1.2 and later versions of B2G.
• Baseline Android ICS and JB on the Nexus S and Nexus 4 hardware.
  • Identify areas where B2G has non-trivially worse power consumption performance than Android.
• Gain the capability to triage, verify, and confirm leo+ bugs coming from hardware partners.

Methods

We are taking a black-box approach to power consumption testing. We are using a digital ammeter wired between the battery and the device to measure power consumption directly during testing runs. The problem of time correlation is to be solved initially by measuring the sampling delay in the commercial-off-the-shelf (COTS) digital ammeter that we currently are using to measure current. Once we know the mean time to first sample (MTFS) from the ammeter, then it is just a matter of coordinating the system clocks between the host PC and the handset device using NTP. The device will record timestamps for events in the test pass (e.g. simulated taps, app process start, etc) which will later be correlated with the samples coming from the ammeter which will be timestamped by the host PC.

We are building a custom ammeter and control harness for our in-house testing lab that will have firmware that supports time synchronization with the host PC, and reporting back of the time of first sample which will initialize the time correlation of the event data from the phone. Because the time stamp of the first sample will come from the time-coordinated ammeter, it will have higher correlation accuracy then the method used with the COTS ammeter.

Sampling Delay Calibration of COTS Ammeter

A simple USB powered/controlled digital I/O board will be used to estimate the MTFS of the COTS ammeter. The first step is to estimate the I/O propagation delay of the digital I/O board by connecting an output pin to an input pin and measuring the time between sending the command to assert the output pin and receiving the value from the input pin. By dividing that time in half, we get an accurate enough estimation of the time it takes to assert the output pin from the host PC. I call that time delay the I/O propagation delay.

The next step is to connect the digital ammeter between the I/O board's output pin and input pin. Then, measure the time between sending the command to assert the output pin and receiving a sample on the ammeter that shows current flowing. If we subtract the I/O propagation delay from the total time between sending the command and receiving a sample showing current, we get an estimate of the sampling delay in the COTS ammeter.

The COTS ammeter delay will then be use as the negative time delta from the time stamped samples coming from the ammeter when correlating current measurements with the event timestamps coming from the handset device. This should be good enough for our initial baselining and bug fixing activities. The design of our own custom ammeter and control harness seeks to make the time stamp correlation much more precise by using a clock coordination algorithm to synchronize the clocks on the device and the ammeter with that of the host PC.

Hardware

The COTS ammeter we use to measure the current draw is the Yocto-Amp from Yoctopuce Labs.

For the initial estimate of the MTFS for the ammeter, we will also use a GoodFET22 USB I/O device for the timing described in the above section.

For the second revision of our setup, the hardware will consist of a 3D printed battery harness that sits between the battery and the device, allowing us to measure the power draw without soldering directly to the battery and device. The PCB used in the revision two battery harness will be a custom built ammeter and relay board that will be used to actuate buttons as well as measure power usage.

Software

There are several pieces of software that need to be written for the power draw testing.

Calibration

• USB I/O board propagation delay timer.
• USB I/O board and ammeter measurement delay timer.
• Changes to logging level in both marionette and gecko to output I/O event timestamps into the logs.

Measurement

• Changes to the marionette driver scripts to run NTP over ADP UDP forward to synchronize the device clock with the host PC clock.
• Changes to the marionette driver scripts to drive the ammeter and collect data from it.