Firefox OS/Performance/Release Acceptance: Difference between revisions
< Firefox OS | Performance
Jump to navigation
Jump to search
(Updated terminology from Cold Launch to First Launch and added Relaunch) |
(Added 2.1 Criteria and updated 2.0 & 2.1 to link to 319MB flame results) |
||
| Line 4: | Line 4: | ||
; First Launch : Milliseconds elapsed from the user-initiated launch of a not-already-in-memory app to the stable display of that app and its visible content. These timings are captured to the '''''moz-app-visually-complete''''' event [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines documented on MDN]. | ; First Launch : Milliseconds elapsed from the user-initiated launch of a not-already-in-memory app to the stable display of that app and its visible content. These timings are captured to the '''''moz-app-visually-complete''''' event [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines documented on MDN]. | ||
; First Launch Memory : Maximum bytes of RAM used by an application after completing its First Launch. | |||
<!-- | |||
; Relaunch : Milliseconds elapsed from the user-initiated launch of a previously launched app to the stable display of that app and its visible content. | ; Relaunch : Milliseconds elapsed from the user-initiated launch of a previously launched app to the stable display of that app and its visible content. | ||
; Max Memory : Maximum bytes of RAM used by an application during some tbd usage scenario. | ; Max Memory : Maximum bytes of RAM used by an application during some tbd usage scenario. | ||
; Max Power : Maximum milliamps used by an application during some tbd usage scenario. | ; Max Power : Maximum milliamps used by an application during some tbd usage scenario. | ||
| Line 12: | Line 13: | ||
''Note: All measurements are the median results calculated from multiple iterations of relevant tests. For example, Datazilla's App Launch values are calculated from 30 such iterations.'' | ''Note: All measurements are the median results calculated from multiple iterations of relevant tests. For example, Datazilla's App Launch values are calculated from 30 such iterations.'' | ||
== 2.1 == | |||
* Reference Device: Flame 319MB | |||
* Reference Workload: Light | |||
{| class="wikitable" | |||
|- | |||
! Application !! [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,clock,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,sms,video&app=communications/dialer&plot=median First Launch] !! First Launch Memory <!-- Relaunch !! Max Memory !! Max Power --> | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar&app=calendar&plot=median Calendar] || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=camera&app=camera&plot=median Camera] || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=clock&app=clock&plot=median Clock] || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=communications/contacts&app=communications/contacts&plot=median Contacts] || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=costcontrol&app=costcontrol&plot=median Cost Control] || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=communications/dialer&app=communications/dialer&plot=median Dialer] || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=email%20FTU&app=email%20FTU&plot=median Email] || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=fm&app=fm&plot=median FM Radio] || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=gallery&app=gallery&plot=median Gallery] || 1000 ms | |||
|- | |||
| Homescreen || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=settings&app=settings&plot=median Settings] || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=sms&app=sms&plot=median SMS] || 1000 ms | |||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=video&app=video&plot=median Video] || 1000 ms | |||
|} | |||
''NOTE: These apps have been [https://bugzilla.mozilla.org/show_bug.cgi?id=996038#user_story_header instrumented] to emit events that support timing their [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines#wiki-document-head launch states].'' | |||
== 2.0 == | == 2.0 == | ||
* Reference Device: Flame | * Reference Device: Flame 319MB | ||
* Reference Workload: Light | * Reference Workload: Light | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
! Application !! [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range= | ! Application !! [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=bookmark,browser,calendar,camera,clock,communications/contacts,communications/dialer,communications/ftu,contacts,costcontrol,email%20FTU,emergency-call,findmydevice,fm,fm_radio,gallery,homescreen,marketplace,messages,music,operatorvariant,pdfjs,phone,pss,rss,settings,sms,system_pss,system_rss,system_uss,system_vsize,template,usage,uss,video,vsize&app=settings&plot=median First Launch] | ||
|- | |||
| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar&app=calendar&plot=median Calendar] || 1000 ms | |||
|- | |- | ||
| | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=camera&app=camera&plot=median Camera] || 1000 ms | ||
|- | |- | ||
| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range= | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=clock&app=clock&plot=median Clock] || 1000 ms | ||
|- | |- | ||
| | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=communications/contacts&app=communications/contacts&plot=median Contacts] || 1000 ms | ||
|- | |- | ||
| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range= | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=costcontrol&app=costcontrol&plot=median Cost Control] || 1000 ms | ||
|- | |- | ||
| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range= | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=communications/dialer&app=communications/dialer&plot=median Dialer] || 1000 ms | ||
|- | |- | ||
| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range= | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=email%20FTU&app=email%20FTU&plot=median Email] || 1000 ms | ||
|- | |- | ||
| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range= | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=fm&app=fm&plot=median FM Radio] || 1000 ms | ||
|- | |- | ||
| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range= | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=gallery&app=gallery&plot=median Gallery] || 1000 ms | ||
|- | |- | ||
| Homescreen || 1000 ms | | Homescreen || 1000 ms | ||
|- | |- | ||
| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range= | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=settings&app=settings&plot=median Settings] || 1000 ms | ||
|- | |- | ||
| SMS || 1000 ms | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=sms&app=sms&plot=median SMS] || 1000 ms | ||
|- | |- | ||
| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range= | | [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=video&app=video&plot=median Video] || 1000 ms | ||
|} | |} | ||
Revision as of 01:44, 5 August 2014
Criteria
Definitions
- First Launch
- Milliseconds elapsed from the user-initiated launch of a not-already-in-memory app to the stable display of that app and its visible content. These timings are captured to the moz-app-visually-complete event documented on MDN.
- First Launch Memory
- Maximum bytes of RAM used by an application after completing its First Launch.
- Reference Workload
- A standardized collection of application data preloaded before the test is run.
Note: All measurements are the median results calculated from multiple iterations of relevant tests. For example, Datazilla's App Launch values are calculated from 30 such iterations.
2.1
- Reference Device: Flame 319MB
- Reference Workload: Light
| Application | First Launch | First Launch Memory |
|---|---|---|
| Calendar | 1000 ms | |
| Camera | 1000 ms | |
| Clock | 1000 ms | |
| Contacts | 1000 ms | |
| Cost Control | 1000 ms | |
| Dialer | 1000 ms | |
| 1000 ms | ||
| FM Radio | 1000 ms | |
| Gallery | 1000 ms | |
| Homescreen | 1000 ms | |
| Settings | 1000 ms | |
| SMS | 1000 ms | |
| Video | 1000 ms |
NOTE: These apps have been instrumented to emit events that support timing their launch states.
2.0
- Reference Device: Flame 319MB
- Reference Workload: Light
| Application | First Launch |
|---|---|
| Calendar | 1000 ms |
| Camera | 1000 ms |
| Clock | 1000 ms |
| Contacts | 1000 ms |
| Cost Control | 1000 ms |
| Dialer | 1000 ms |
| 1000 ms | |
| FM Radio | 1000 ms |
| Gallery | 1000 ms |
| Homescreen | 1000 ms |
| Settings | 1000 ms |
| SMS | 1000 ms |
| Video | 1000 ms |
NOTE: These apps have been instrumented to emit events that support timing their launch states.
1.4
- Reference Device: Buri/Hamachi
- Reference Workload: Medium
| Application | First Launch |
|---|---|
| Browser | No regression since 1.3 |
| Camera | No regression since 1.3 |
| Contacts | No regression since 1.3 |
| Dialer | No regression since 1.3 |
| No regression since 1.3 | |
| Gallery | No regression since 1.3 |
| Marketplace | No regression since 1.3 |
| Settings | No regression since 1.3 |
| SMS | No regression since 1.3 |
1.3T
- Reference Device: Tarako
Acceptance for 1.3T will be determined by reviewing final performance results which will include cold launch time.