Firefox OS/Performance/Automation/Launch Latency: Difference between revisions

m
Lakrits moved page FirefoxOS/Performance/Automation/Launch Latency to Firefox OS/Performance/Automation/Launch Latency: The official spelling of "Firefox OS" leaves a space between the two parts of the name. It's easier to find a page if the sp...
m (Lakrits moved page FirefoxOS/Performance/Automation/Launch Latency to Firefox OS/Performance/Automation/Launch Latency: The official spelling of "Firefox OS" leaves a space between the two parts of the name. It's easier to find a page if the sp...)
 
(76 intermediate revisions by one other user not shown)
Line 1: Line 1:
= Launch Latency =
= Launch Latency =
== Summary ==
== Summary ==


Measures elapsed time from the first launch of an application to a series of defined [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines startup events].
Measures elapsed time from the first launch of an application to a series of defined [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines startup events].


== Precision ==
== Usage ==
 
TBD


== Scenarios ==
=== Test Cases ===


Each scenario is to launch one of the following apps from a cold state:
Launch each of the following apps from a cold state:


* Browser
* Browser
* Calendar
* Camera
* Camera
* Clock
* Contacts
* Contacts
* Dialer
* Dialer
* Email
* Email
* FM Radio
* Gallery
* Gallery
* Marketplace
* Marketplace
* Music
* Settings
* Settings
* SMS
* SMS
* Template
* Usage
* Video


== Targets ==
For each app, report the time elapsed to the following events:


=== 2.0 ===
* moz-chrome-dom-loaded
* moz-chrome-interactive
* moz-app-visually-complete
* moz-content-interative
* moz-app-loaded


=== Precision ===


TBD


=== 2.1 ===
=== How to Run On-Demand ===


== Procedure ==
TBD
 
''This needs more detail.''
 
# Clear app from memory
# Inject launch event
# Monitor app events for:
#* moz-chrome-dom-loaded
#* moz-chrome-interactive
#* moz-app-visually-complete
#* moz-content-interative
#* moz-app-loaded
# Time elapsed is the time between that event and ''start time?''
 
== Roadmap ==
 
=== Overview ===


* '''Lead:''' Eli Perelman
=== Published Results ===
* '''Metabug:''' TBD


=== Milestone 1: Design ===
==== FxOS 2.0 ====


{| class="wikitable"
{| class="wikitable"
! width="120px" |  
!
! Target Release
! Flame
! Test Design
|-
! Validity Review
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''All''' || TBD || Bug || Bug
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
|}
|}


=== Milestone 2: Test ===
==== FxOS 2.1 ====


{| class="wikitable"
{| class="wikitable"
! width="120px" |
!
! Target Release
! Flame
! Instrumentation
! Workloads
! Single-run test
! Results Review
|-
|-
| '''Browser''' || TBD || Bug || Bug || Bug || Bug
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''Camera''' || TBD || Bug || Bug || Bug || Bug
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''Contacts''' || TBD || Bug || Bug || Bug || Bug
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''Dialer''' || TBD || Bug || Bug || Bug || Bug
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''Email''' || TBD || Bug || Bug || Bug || Bug
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''Gallery''' || TBD || Bug || Bug || Bug || Bug
|}
 
=== References ===
 
* [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines Firefox OS App Responsiveness Guidelines]
 
== Development ==
 
=== Design ===
 
==== Common Setup ====
# Set up workload
# Restart B2G process
 
==== All Cases ====
# Invoke launch programmatically
# Inject the Performance Helper and save start time
# App instrumentation throws timeline events at appropriate times
# Performance helper observes events
# Event emit times are compared with start time for time elapsed
 
==== Results ====
* Result is time elapsed to each event
* Execute test 30 times, for each event report mean, median, deviation
 
=== Project Roadmap ===
 
* '''Lead:''' Eli Perelman
* '''Tracking Bug:''' TBD
 
{| class="wikitable"
! width="120px" |
! colspan="2" | Milestone 1: Design
|-
|-
| '''Marketplace''' || TBD || Bug || Bug || Bug || Bug
!
|-
! Test Design
| '''Settings''' || TBD || Bug || Bug || Bug || Bug
! Validity Review
|-
|-
| '''SMS''' || TBD || Bug || Bug || Bug || Bug
| '''All Cases''' || Bug || Bug
|-
|-
|}
|}
=== Milestone 3: Publish ===


{| class="wikitable"
{| class="wikitable"
! width="120px" |
! width="120px" |
! Target Release
!
! colspan="4" | Milestone 2: Test
! colspan="2" | Milestone 3: Publication
|-
!
! Availability
! Instrumentation
! Workload
! On-Demand Test
! Results Review
! Published Results
! Published Results
! Documentation
! Documentation
|-
|-
| '''Browser''' || TBD || Bug || Bug
| '''Browser''' || TBD || {{bug|837668}} || Bug || Bug || Bug || Bug || Bug
|-
| '''Calendar''' || TBD || {{bug|837671}} || Bug || Bug || Bug || Bug || Bug
|-
| '''Camera''' || TBD || {{bug|1015385}} || N/A || Bug || Bug || Bug || Bug
|-
| '''Clock''' || TBD || {{bug|837668}} || N/A || Bug || Bug || Bug || Bug
|-
| '''Contacts''' || TBD || {{bug|1015388}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
| '''Dialer''' || TBD || {{bug|837668}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
| '''Email''' || TBD || {{bug|837677}} || Bug || Bug || Bug || Bug || Bug
|-
| '''FM Radio''' || TBD || {{bug|1015412}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Camera''' || TBD || Bug || Bug
| '''Gallery''' || TBD || {{bug|837674}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Contacts''' || TBD || Bug || Bug
| '''Marketplace''' || TBD || {{bug|837668}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Dialer''' || TBD || Bug || Bug
| '''Music''' || TBD || {{bug|837675}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Email''' || TBD || Bug || Bug
| '''Settings''' || TBD || {{bug|1015405}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Gallery''' || TBD || Bug || Bug
| '''SMS''' || TBD || {{bug|1015390}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Marketplace''' || TBD || Bug || Bug
| '''Template''' || TBD || {{bug|1015391}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Settings''' || TBD || Bug || Bug
| '''Usage''' || TBD || {{bug|837678}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''SMS''' || TBD || Bug || Bug
| '''Video''' || TBD || {{bug|837658}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
|}
|}


<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Test_Template template]</small>
<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Metric_Template template]</small>
3,860

edits