MozillaWiki

Firefox OS/Performance/Automation/Launch Latency: Difference between revisions

Page Discussion

Firefox OS/Performance/Automation/Launch Latency (view source)

Revision as of 13:59, 1 February 2015

848 bytes added ,  1 February 2015
m
Lakrits moved page FirefoxOS/Performance/Automation/Launch Latency to Firefox OS/Performance/Automation/Launch Latency: The official spelling of "Firefox OS" leaves a space between the two parts of the name. It's easier to find a page if the sp...
m (Lakrits moved page FirefoxOS/Performance/Automation/Launch Latency to Firefox OS/Performance/Automation/Launch Latency: The official spelling of "Firefox OS" leaves a space between the two parts of the name. It's easier to find a page if the sp...)
 
(53 intermediate revisions by one other user not shown)
Line 1: Line 1:
= Launch Latency =
= Launch Latency =
== Summary ==
== Summary ==


Measures elapsed time from the first launch of an application to a series of defined [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines startup events].
Measures elapsed time from the first launch of an application to a series of defined [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines startup events].


== Precision ==
== Usage ==
 
''This would be filled in with the estimated precision of the test based on result variability and scientific precision of the various measurements. This is different than accuracy, and represents how much "scatter" of the results around the true result we might expect.
 
We might estimate a precision of 10ms or we might estimate a precision of 100ms. The latter would imply very different margins before we could assume a regression.''


== Scenarios ==
=== Test Cases ===


Launch each of the following apps from a cold state:
Launch each of the following apps from a cold state:


* Browser
* Browser
* Calendar
* Camera
* Camera
* Clock
* Contacts
* Contacts
* Dialer
* Dialer
* Email
* Email
* FM Radio
* Gallery
* Gallery
* Marketplace
* Marketplace
* Music
* Settings
* Settings
* SMS
* SMS
* Template
* Usage
* Video


For each app, report the time elapsed to the following events:
For each app, report the time elapsed to the following events:
Line 32: Line 36:
* moz-app-loaded
* moz-app-loaded


== Dashboards ==
=== Precision ===
 
TBD
 
=== How to Run On-Demand ===
 
TBD


''This should be a series of links to dashboards per branch, device, and test. Unfortunately, Datazilla doesn't seem to allow direct linking without embedding revisions, specific implemented apps, and other time-specific artifacts in the URL. This is a real problem.''
=== Published Results ===


=== 2.0 ===
==== FxOS 2.0 ====


{| class="wikitable"
{| class="wikitable"
! Test
!
! Flame
! Flame
|-
|-
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
|}
|}


=== 2.1 ===
==== FxOS 2.1 ====


{| class="wikitable"
{| class="wikitable"
! Test
!
! Flame
! Flame
|-
|-
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
|}
|}


== Procedure ==
=== References ===
 
* [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines Firefox OS App Responsiveness Guidelines]


''This needs more detail.''
== Development ==


# Clear app from memory
=== Design ===
# Inject launch command
# Monitor app events for each of the five events under Scenarios
# Time elapsed is the time between that event and ''start time?''


== References ==
==== Common Setup ====
# Set up workload
# Restart B2G process


TBD
==== All Cases ====
# Invoke launch programmatically
# Inject the Performance Helper and save start time
# App instrumentation throws timeline events at appropriate times
# Performance helper observes events
# Event emit times are compared with start time for time elapsed


== Roadmap ==
==== Results ====
* Result is time elapsed to each event
* Execute test 30 times, for each event report mean, median, deviation


=== Overview ===
=== Project Roadmap ===


* '''Lead:''' Eli Perelman
* '''Lead:''' Eli Perelman
* '''Tracking Bug:''' TBD
* '''Tracking Bug:''' TBD
=== Milestone 1: Design ===
'''Tracking Bug:''' TBD


{| class="wikitable"
{| class="wikitable"
! width="120px" |  
! width="120px" |
! Target Release
! colspan="2" | Milestone 1: Design
! Test Design  
|-
!
! Test Design
! Validity Review
! Validity Review
|-
|-
| '''All''' || TBD || Bug || Bug
| '''All Cases''' || Bug || Bug
|-
|-
|}
|}
=== Milestone 2: Test ===
'''Tracking Bug:''' TBD


{| class="wikitable"
{| class="wikitable"
! width="120px" |
! width="120px" |
! Target Release
!
! colspan="4" | Milestone 2: Test
! colspan="2" | Milestone 3: Publication
|-
!
! Availability
! Instrumentation
! Instrumentation
! Workloads
! Workload
! Single-run test
! On-Demand Test
! Results Review
! Results Review
! Published Results
! Documentation
|-
|-
| '''Browser''' || TBD || Bug || Bug || Bug || Bug
| '''Browser''' || TBD || {{bug|837668}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Camera''' || TBD || Bug || Bug || Bug || Bug
| '''Calendar''' || TBD || {{bug|837671}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Contacts''' || TBD || Bug || Bug || Bug || Bug
| '''Camera''' || TBD || {{bug|1015385}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Dialer''' || TBD || Bug || Bug || Bug || Bug
| '''Clock''' || TBD || {{bug|837668}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Email''' || TBD || Bug || Bug || Bug || Bug
| '''Contacts''' || TBD || {{bug|1015388}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Gallery''' || TBD || Bug || Bug || Bug || Bug
| '''Dialer''' || TBD || {{bug|837668}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Marketplace''' || TBD || Bug || Bug || Bug || Bug
| '''Email''' || TBD || {{bug|837677}} || Bug || Bug || Bug || Bug || Bug
|-
| '''Settings''' || TBD || Bug || Bug || Bug || Bug
|-
| '''SMS''' || TBD || Bug || Bug || Bug || Bug
|-
|}
 
=== Milestone 3: Publication ===
'''Tracking Bug:''' TBD
 
{| class="wikitable"
! width="120px" |
! Target Release
! Published Results
! Documentation
|-
|-
| '''Browser''' || TBD || Bug || Bug
| '''FM Radio''' || TBD || {{bug|1015412}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Camera''' || TBD || Bug || Bug
| '''Gallery''' || TBD || {{bug|837674}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Contacts''' || TBD || Bug || Bug
| '''Marketplace''' || TBD || {{bug|837668}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Dialer''' || TBD || Bug || Bug
| '''Music''' || TBD || {{bug|837675}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Email''' || TBD || Bug || Bug
| '''Settings''' || TBD || {{bug|1015405}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Gallery''' || TBD || Bug || Bug
| '''SMS''' || TBD || {{bug|1015390}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Marketplace''' || TBD || Bug || Bug
| '''Template''' || TBD || {{bug|1015391}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Settings''' || TBD || Bug || Bug
| '''Usage''' || TBD || {{bug|837678}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''SMS''' || TBD || Bug || Bug
| '''Video''' || TBD || {{bug|837658}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
|}
|}


<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Test_Template template]</small>
<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Metric_Template template]</small>
Lakrits
3,860

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/998382...1052495"