MozillaWiki

Firefox OS/Performance/Automation/Launch Latency: Difference between revisions

Page Discussion

Firefox OS/Performance/Automation/Launch Latency (view source)

Revision as of 13:59, 1 February 2015

610 bytes added ,  1 February 2015
m
Lakrits moved page FirefoxOS/Performance/Automation/Launch Latency to Firefox OS/Performance/Automation/Launch Latency: The official spelling of "Firefox OS" leaves a space between the two parts of the name. It's easier to find a page if the sp...
m (Lakrits moved page FirefoxOS/Performance/Automation/Launch Latency to Firefox OS/Performance/Automation/Launch Latency: The official spelling of "Firefox OS" leaves a space between the two parts of the name. It's easier to find a page if the sp...)
 
(30 intermediate revisions by one other user not shown)
Line 1: Line 1:
= Launch Latency =
= Launch Latency =
== Summary ==
== Summary ==


Measures elapsed time from the first launch of an application to a series of defined [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines startup events].
Measures elapsed time from the first launch of an application to a series of defined [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines startup events].


== Precision ==
== Usage ==
 
TBD


== Scenarios ==
=== Test Cases ===


Launch each of the following apps from a cold state:
Launch each of the following apps from a cold state:


* Browser
* Browser
* Calendar
* Camera
* Camera
* Clock
* Contacts
* Contacts
* Dialer
* Dialer
* Email
* Email
* FM Radio
* Gallery
* Gallery
* Marketplace
* Marketplace
* Music
* Settings
* Settings
* SMS
* SMS
* Template
* Usage
* Video


For each app, report the time elapsed to the following events:
For each app, report the time elapsed to the following events:
Line 30: Line 36:
* moz-app-loaded
* moz-app-loaded


== Dashboards ==
=== Precision ===
 
TBD


=== 2.0 ===
=== How to Run On-Demand ===
 
TBD
 
=== Published Results ===
 
==== FxOS 2.0 ====


{| class="wikitable"
{| class="wikitable"
! Test
!
! Flame
! Flame
|-
|-
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
|}
|}


=== 2.1 ===
==== FxOS 2.1 ====


{| class="wikitable"
{| class="wikitable"
! Test
!
! Flame
! Flame
|-
|-
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
|}
|}


== Procedure ==
=== References ===
=== Setup ===
 
* [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines Firefox OS App Responsiveness Guidelines]
 
== Development ==
 
=== Design ===
 
==== Common Setup ====
# Set up workload
# Restart B2G process
# Restart B2G process


=== Test ===  
==== All Cases ====
# Invoke launch programmatically  
# Invoke launch programmatically  
# Inject the Performance Helper and save start time
# Inject the Performance Helper and save start time
# Instrumented throws timeline events at appropriate times
# App instrumentation throws timeline events at appropriate times
# Performance helper observes events
# Performance helper observes events
# Event emit times are compared with start time for time elapsed
# Event emit times are compared with start time for time elapsed


=== Execution ===
==== Results ====
# Test should be executed 30 times, take average and median
* Result is time elapsed to each event
 
* Execute test 30 times, for each event report mean, median, deviation
== References ==
 
* [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines Firefox OS App Responsiveness Guidelines]
 
== Roadmap ==


=== Overview ===
=== Project Roadmap ===


* '''Lead:''' Eli Perelman
* '''Lead:''' Eli Perelman
* '''Tracking Bug:''' TBD
* '''Tracking Bug:''' TBD
=== Milestone 1: Design ===
'''Tracking Bug:''' TBD


{| class="wikitable"
{| class="wikitable"
! width="120px" |  
! width="120px" |
! Target Release
! colspan="2" | Milestone 1: Design
! Test Design  
|-
!
! Test Design
! Validity Review
! Validity Review
|-
|-
| '''All''' || TBD || Bug || Bug
| '''All Cases''' || Bug || Bug
|-
|-
|}
|}
=== Milestone 2: Test ===
'''Tracking Bug:''' TBD


{| class="wikitable"
{| class="wikitable"
! width="120px" |
! width="120px" |
! Target Release
!
! colspan="4" | Milestone 2: Test
! colspan="2" | Milestone 3: Publication
|-
!
! Availability
! Instrumentation
! Instrumentation
! Workload
! Workload
! Single-run test
! On-Demand Test
! Results Review
! Results Review
! Available
! Published Results
|-
! Documentation
| '''Browser''' || TBD || {{bug|837668}} || Bug || Bug || Bug ||
|-
| '''Calendar''' || TBD || {{bug|837671}} || Bug || Bug || Bug ||
|-
| '''Camera''' || TBD || {{bug|1015385}} || N/A || Bug || Bug ||
|-
| '''Clock''' || TBD || {{bug|837668}} || N/A || Bug || Bug ||
|-
| '''Contacts''' || TBD || {{bug|1015388}} || Light || Bug || Bug ||
|-
| '''Dialer''' || TBD || {{bug|837668}} || Light || Bug || Bug ||
|-
| '''Email''' || TBD || {{bug|837677}} || Bug || Bug || Bug ||
|-
|-
| '''FM Radio''' || TBD || {{bug|1015412}} || Bug || Bug || Bug ||
| '''Browser''' || TBD || {{bug|837668}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Gallery''' || TBD || {{bug|837674}} || Light || Bug || Bug ||
| '''Calendar''' || TBD || {{bug|837671}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Marketplace''' || TBD || {{bug|837668}} || Bug || Bug || Bug ||
| '''Camera''' || TBD || {{bug|1015385}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Music''' || TBD || {{bug|837675}} || Bug || Bug || Bug ||
| '''Clock''' || TBD || {{bug|837668}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Settings''' || TBD || {{bug|1015405}} || N/A || Bug || Bug ||
| '''Contacts''' || TBD || {{bug|1015388}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''SMS''' || TBD || {{bug|1015390}} || Light || Bug || Bug ||
| '''Dialer''' || TBD || {{bug|837668}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
| '''Usage''' || TBD || {{bug|837678}} || Bug || Bug || Bug ||
|-
| '''Video''' || TBD || {{bug|837658}} || Bug || Bug || Bug ||
|}
 
=== Milestone 3: Publication ===
'''Tracking Bug:''' TBD
 
{| class="wikitable"
! width="120px" |
! Target Release
! Published Results
! Documentation
! Available
|-
|-
| '''Browser''' || TBD || Bug || Bug ||
| '''Email''' || TBD || {{bug|837677}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Calendar''' || TBD || Bug || Bug ||
| '''FM Radio''' || TBD || {{bug|1015412}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Camera''' || TBD || Bug || Bug ||
| '''Gallery''' || TBD || {{bug|837674}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Contacts''' || TBD || Bug || Bug ||
| '''Marketplace''' || TBD || {{bug|837668}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Dialer''' || TBD || Bug || Bug ||
| '''Music''' || TBD || {{bug|837675}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Email''' || TBD || Bug || Bug ||
| '''Settings''' || TBD || {{bug|1015405}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Gallery''' || TBD || Bug || Bug ||
| '''SMS''' || TBD || {{bug|1015390}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Marketplace''' || TBD || Bug || Bug ||
| '''Template''' || TBD || {{bug|1015391}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Settings''' || TBD || Bug || Bug ||
| '''Usage''' || TBD || {{bug|837678}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''SMS''' || TBD || Bug || Bug ||
| '''Video''' || TBD || {{bug|837658}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
|}
|}


<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Test_Template template]</small>
<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Metric_Template template]</small>
Lakrits
3,860

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/999397...1052495"