Firefox OS/Performance/Release Acceptance: Difference between revisions
< Firefox OS | Performance
Jump to navigation
Jump to search
Bobbychien (talk | contribs) (→Criteria: restructure) |
Bobbychien (talk | contribs) (→Definitions: Add Reference Devices) |
||
| (6 intermediate revisions by the same user not shown) | |||
| Line 13: | Line 13: | ||
''Note: All measurements are the median results calculated from multiple iterations of relevant tests. For example, Datazilla's App Launch values are calculated from 30 such iterations.'' | ''Note: All measurements are the median results calculated from multiple iterations of relevant tests. For example, Datazilla's App Launch values are calculated from 30 such iterations.'' | ||
== Reference Devices == | |||
* [https://developer.mozilla.org/en-US/Firefox_OS/Phone_guide/Flame Flame] | |||
* [https://en.wikipedia.org/wiki/Sony_Xperia_Z3_Compact Sony Xperia Z3 Compact (Aries)] | |||
* [https://developer.mozilla.org/en-US/Firefox_OS/Phone_guide/Phone_specs Others phone ] | |||
== Application cold launch == | == Application cold launch == | ||
== 2.5 == | === FxOS 2.5 === | ||
* Reference Device: Flame-KK 319MB and Aries-KK | * Reference Device: Flame-KK 319MB and Aries-KK | ||
| Line 176: | Line 182: | ||
|} | |} | ||
== 1.4 == | === FxOS 1.4 === | ||
* Reference Device: Buri/Hamachi | * Reference Device: Buri/Hamachi | ||
| Line 204: | Line 210: | ||
|} | |} | ||
== 1.3T == | === FxOS 1.3T === | ||
* Reference Device: Tarako | * Reference Device: Tarako | ||
| Line 211: | Line 217: | ||
== Memory Usage == | == Memory Usage == | ||
== Power Consumption == | == [[Firefox OS/Performance/Power Consumption|Power Consumption]] == | ||
== MTBF (Mean time between failures) == | |||
== [[Firefox_OS/Performance/MTBF | MTBF (Mean time between failures)]] == | |||
Latest revision as of 09:02, 26 August 2015
Criteria
Definitions
- First Launch
- Milliseconds elapsed from the user-initiated launch of a not-already-in-memory app to the stable display of that app and its visible content. These timings are captured to the moz-app-visually-complete event documented on MDN.
- First Launch Memory
- Maximum bytes of RAM used by an application after completing its First Launch.
- Reference Workload
- A standardized collection of application data preloaded before the test is run.
Note: All measurements are the median results calculated from multiple iterations of relevant tests. For example, Datazilla's App Launch values are calculated from 30 such iterations.
Reference Devices
Application cold launch
FxOS 2.5
- Reference Device: Flame-KK 319MB and Aries-KK
- Reference Workload: Light
| Application | First Launch | Regression | Nominated |
|---|---|---|---|
| Calendar | 1150 ms | Bug 1181019 | |
| Camera | 1550 ms | ||
| Clock | 1000 ms | Bug 1181021 | |
| Contacts | 1000 ms | ||
| Dialer | 1000 ms | ||
| 1000 ms | |||
| FM Radio | 1000 ms | ||
| Gallery | 1000 ms | ||
| Music | 1000 ms | ||
| Settings | 2600 ms | Bug 1181023 | |
| SMS | 1200 ms | Bug 1181017 | |
| Video | 1000 ms |
NOTE: These apps have been instrumented to emit events that support timing their launch states.
FxOS 2.0, 2.1, 2.2
- Reference Device: Flame 319MB
- Reference Workload: Light
- For Camera App, as discussed with the PM, the performance boundary will be based on the average of 3 performance runs done in Oct 2nd, 21st, and 30th. If the number regresses more than 100ms from the criteria, it will be considered as a regression.
| Application | FxOS 2.2 | FxOS 2.1 | FxOS 2.0 |
|---|---|---|---|
| Calendar | 1150 ms | 1150 ms | 1000 ms |
| Camera | 1550 ms | 1550 ms | 1000 ms |
| Clock | 1000 ms | 1000 ms | 1000 ms |
| Contacts | 1000 ms | 1000 ms | 1000 ms |
| Cost Control | 1000 ms | 1000 ms | 1000 ms |
| Dialer | 1000 ms | 1000 ms | 1000 ms |
| 1000 ms | 1000 ms | 1000 ms | |
| FM Radio | 1000 ms | 1000 ms | 1000 ms |
| Gallery | 1000 ms | 1000 ms | 1000 ms |
| Homescreen | 1000 ms | 1000 ms | |
| Marketplace | 1000 ms | 1000 ms | 1000 ms |
| Music | 1000 ms | 1000 ms | |
| Settings | 2600 ms | 2600 ms | 1000 ms |
| SMS | 1200 ms | 1200 ms | 1000 ms |
| Video | 1000 ms | 1000 ms | 1000 ms |
FxOS 1.4
- Reference Device: Buri/Hamachi
- Reference Workload: Medium
| Application | First Launch |
|---|---|
| Browser | No regression since 1.3 |
| Camera | No regression since 1.3 |
| Contacts | No regression since 1.3 |
| Dialer | No regression since 1.3 |
| No regression since 1.3 | |
| Gallery | No regression since 1.3 |
| Marketplace | No regression since 1.3 |
| Settings | No regression since 1.3 |
| SMS | No regression since 1.3 |
FxOS 1.3T
- Reference Device: Tarako
Acceptance for 1.3T will be determined by reviewing final performance results which will include cold launch time.