Firefox OS/Performance/UserStories

Perception of cause and effect (140ms)

• Time: 140ms
• Bugzilla whiteboard: c=effect
• Use cases
  • Touch States (i.e. Keyboard)
  • Transitions
  • Device Rotation
• Stories
  • As a user I expect a visual change within 140ms of launching an app.
  • As a user I expect buttons or list items to show their highlight state within 140ms of being touched.
  • As a user I expect screen transitions to begin to occur within 140ms of initiation.
  • As a user I expect an app to start redrawing as portrait/landscape within 140ms of rotating the device.

Perception of progress

• Time: 1s
• Bugzilla whiteboard: c=progress
• Use cases
  • App Launch
  • Initial paint
  • Load Above the fold
  • Load Fully
  • Long-running operations (eg: downloads, wifi connect)
  • Time to First Interaction
• Stories
  • As a user I expect an app's first view (above the fold) to complete rendering within 1 sec.
  • As a user I expect continuous visual updates of the progress of long running operations.
  • As a user I expect to be able to interact with an app within 1 sec.

Hand-eye coordination

• Time: 100ms
• Bugzilla whiteboard: c=handeye
• Use cases
  • Drag & Drop (Homescreen & dock icon moving)
  • Scrolling
  • Pinch/zoom
  • Swiping pages, opening drawers
• Stories
  • As a user I expect to a drag operation to visually respond within 100 ms
  • As a user I expect a visual response within 100 ms when I pinch/zoom where it's supported.
  • As a user I expect scroll operations to visually respond within 100ms of initiation
  • As a user I expect scrolling to remain in synch with touch events within 100ms
  • As a user I expect a visual response within 100 ms when initiating a swipe.

Power Management

• Bugzilla whiteboard: c=power
• Use cases
  • ...
• Stories
  • As a FirefoxOS product manager, I would like to make sure that system lifetime on a common workload does not regress between versions. (Sandip: Need to have an exact current drain reference per app on a given HW/SW. e.g. with 7227 chipset products launched on v1.0.1, we can list the per app requirements ad ensure that we do not regress those on 1.1. However this will be a challenge to state these numbers on 8x26chipset + 1.2 as we would need the "reference numbers" first from chipmaker on our sw.)
  • As a Gaia developer, I would like to make sure that my changes to an application does not regress battery life when someone is using my application.
  • As a Gaia developer, I would like to make sure that my changes to an application does not regress battery life when someone is using my application.
  • As a platform developer, I want to know *whether* overall system battery life has regressed or improved after I make a change.
  • As a platform developer, I would like to know *why* overall system battery life regressed or improved after I made a change.

Question to be resolved: Developers may not need exact information on how changes affect battery life, so long as the information we get is representative of what would happen on the device. I.e. it may be acceptable to run power management tests on desktop class hardware using emulated versions of Gaia and/or FirefoxOS itself (Gaia developers may only need the capability of testing the former?). In the case where we want actual numbers on a real device, may it be sufficient to get numbers for *one* device, and not all of them? On the other hand, it is likely (?) that a product manager would be very particular about what specific device(s) these tests were run on. (Sandip: As a product manager, I would like to see the exact HW/SW combination (reference device) for stating current drain limitations/requirements.)

Performance Try-Server

• Bugzilla whiteboard: c=automation
• Use cases
  • ...
• User stories
  • As a platform developer, I would like to make sure that my changes to Gecko do not negatively impact the load times of the main gaia applications on a shipping device (or conversely, whether my changes improve such times)
  • As a platform developer, I would like to know *why* the startup time of particular Gaia applications has decreased or increased after I made a change.
  • As a Gaia developer, I would like to make sure that my changes to an application do not negatively its load time on a shipping device (or conversely, whether my changes improve such times)
  • As a Gaia developer, I would like to know why the loading times of an application I'm working on has improved or regressed.
  • As a platform developer, I want to make sure that the responsiveness of the main gaia applications does not regress with my changes (or conversely, whether my changes improve such times)
  • As a platform developer, I would like to know why the responsiveness of the main Gaia applications has changed.
  • As a Gaia developer, I would like to make sure that my changes to an application do not negatively impact its responsiveness (or conversely, whether my changes improve such times)
  • As a Gaia developer, I would like to know why the responsiveness of an application I've been working on has changed.

(many of the power management user stories are likely to apply here too)

As with power management, likely what is needed here depends on the audience. Some types of tests probably need real hardware (and possibly eideticker's camera setup) to measure effectively. In other cases, you might be able to get representative data with desktop-class hardware.

Second Tier (not storied yet)

• Jank (Visual Glitches)
• Perception of successive images (eg animation stutter, 50ms)
• Perception of audio gaps
• Attention span to complete sub-task