Confirmed users
243
edits
Firefox OS/Performance/Release Acceptance: Difference between revisions
Firefox OS/Performance/Release Acceptance (view source)
Revision as of 09:57, 12 August 2015
, 12 August 2015→Criteria: restructure
Bobbychien (talk | contribs) (→2.5: wording) |
Bobbychien (talk | contribs) (→Criteria: restructure) |
||
| Line 13: | Line 13: | ||
''Note: All measurements are the median results calculated from multiple iterations of relevant tests. For example, Datazilla's App Launch values are calculated from 30 such iterations.'' | ''Note: All measurements are the median results calculated from multiple iterations of relevant tests. For example, Datazilla's App Launch values are calculated from 30 such iterations.'' | ||
== Application cold launch == | |||
== 2.5 == | == 2.5 == | ||
| Line 86: | Line 89: | ||
''NOTE: These apps have been [https://bugzilla.mozilla.org/show_bug.cgi?id=996038#user_story_header instrumented] to emit events that support timing their [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines#wiki-document-head launch states].'' | ''NOTE: These apps have been [https://bugzilla.mozilla.org/show_bug.cgi?id=996038#user_story_header instrumented] to emit events that support timing their [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines#wiki-document-head launch states].'' | ||
== 2.2 == | === FxOS 2.0, 2.1, 2.2 === | ||
* Reference Device: | * Reference Device: Flame 319MB | ||
* Reference Workload: Light | * Reference Workload: Light | ||
* For Camera App, as discussed with the PM, the performance boundary will be based on the average of 3 performance runs done in Oct 2nd, 21st, and 30th. If the number regresses more than 100ms from the criteria, it will be considered as a regression. | * For Camera App, as discussed with the PM, the performance boundary will be based on the average of 3 performance runs done in Oct 2nd, 21st, and 30th. If the number regresses more than 100ms from the criteria, it will be considered as a regression. | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
! Application !! [https://datazilla.mozilla.org/b2g/?branch= | ! Application !! [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,clock,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,sms,video&app=communications/dialer&plot=median FxOS 2.2] !! [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,clock,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,sms,video&app=communications/dialer&plot=median FxOS 2.1] !! [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,clock,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,sms,video&app=communications/dialer&plot=median FxOS 2.0] | ||
|- | |- | ||
| Calendar || [https://datazilla.mozilla.org/b2g/?branch= | | Calendar | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar&app=calendar&plot=median 1150 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar&app=calendar&plot=median 1150 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar&app=calendar&plot=median 1000 ms] | |||
|- | |- | ||
| Camera || [https://datazilla.mozilla.org/b2g/?branch= | | Camera | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=camera&app=camera&plot=median 1550 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=camera&app=camera&plot=median 1550 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=camera&app=camera&plot=median 1000 ms] | |||
|- | |- | ||
| Clock || [https://datazilla.mozilla.org/b2g/?branch= | | Clock | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=clock&app=clock&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=clock&app=clock&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=clock&app=clock&plot=median 1000 ms] | |||
|- | |- | ||
| Contacts || [https://datazilla.mozilla.org/b2g/?branch= | | Contacts | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=communications/contacts&app=communications/contacts&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=communications/contacts&app=communications/contacts&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=communications/contacts&app=communications/contacts&plot=median 1000 ms] | |||
|- | |- | ||
| Cost Control || [https://datazilla.mozilla.org/b2g/?branch= | | Cost Control | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=costcontrol&app=costcontrol&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=costcontrol&app=costcontrol&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=costcontrol&app=costcontrol&plot=median 1000 ms] | |||
|- | |- | ||
| Dialer || [https://datazilla.mozilla.org/b2g/?branch= | | Dialer | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=communications/dialer&app=communications/dialer&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=communications/dialer&app=communications/dialer&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=communications/dialer&app=communications/dialer&plot=median 1000 ms] | |||
|- | |- | ||
| Email || [https://datazilla.mozilla.org/b2g/?branch= | | Email | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=email%20FTU&app=email%20FTU&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=email%20FTU&app=email%20FTU&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=email%20FTU&app=email%20FTU&plot=median 1000 ms] | |||
|- | |- | ||
| FM Radio || [https://datazilla.mozilla.org/b2g/?branch= | | FM Radio | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=fm&app=fm&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=fm&app=fm&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=fm&app=fm&plot=median 1000 ms] | |||
|- | |- | ||
| Gallery || [https://datazilla.mozilla.org/b2g/?branch= | | Gallery | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=gallery&app=gallery&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=gallery&app=gallery&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=gallery&app=gallery&plot=median 1000 ms] | |||
|- | |- | ||
| Homescreen || 1000 ms | | Homescreen | ||
|| 1000 ms | |||
|| 1000 ms | |||
|| | |||
|- | |- | ||
| Marketplace || 1000 ms | | Marketplace | ||
|| 1000 ms | |||
|| 1000 ms | |||
|| 1000 ms | |||
|- | |- | ||
| Music || [https://datazilla.mozilla.org/b2g/?branch= | | Music | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=music&app=music&gaia_rev=a53a3a1d5d4f8d75&gecko_rev=fdad48b0e5b5&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=music&app=music&gaia_rev=a53a3a1d5d4f8d75&gecko_rev=fdad48b0e5b5&plot=median 1000 ms] | |||
|| | |||
|- | |- | ||
| Settings || [https://datazilla.mozilla.org/b2g/?branch= | | Settings | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=settings&app=settings&plot=median 2600 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=settings&app=settings&plot=median 2600 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=settings&app=settings&plot=median 1000 ms] | |||
|- | |- | ||
| SMS || [https://datazilla.mozilla.org/b2g/?branch= | | SMS | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=sms&app=sms&plot=median 1200 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=sms&app=sms&plot=median 1200 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=sms&app=sms&plot=median 1000 ms] | |||
|- | |- | ||
| Video | | Video | ||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.2&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=video&app=video&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=video&app=video&plot=median 1000 ms] | |||
|| [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=video&app=video&plot=median 1000 ms] | |||
| | |||
|} | |} | ||
== 1.4 == | == 1.4 == | ||
| Line 209: | Line 209: | ||
Acceptance for 1.3T will be determined by reviewing final performance results which will include [https://datazilla.mozilla.org/b2g/?branch=v1.3t&device=tarako&range=90&test=cold_load_time&plot=median cold launch time]. | Acceptance for 1.3T will be determined by reviewing final performance results which will include [https://datazilla.mozilla.org/b2g/?branch=v1.3t&device=tarako&range=90&test=cold_load_time&plot=median cold launch time]. | ||
== Memory Usage == | |||
== Power Consumption == | |||
== MTBF (Mean time between failures) == | |||