Firefox OS/Performance/Memory acceptance criteria

This page intends to define the criteria by which we say "memory management is good enough to ship", and test plans we can use to check those criteria.

Terminology

Foreground. For an application X, part of X's user interface is visible onscreen.

Perceivable. For an application X, no part of X's user interface is visible onscreen, but a user can perceive effects of X. For example, a music player that runs while its user interface is not visible is perceivable.

Background. For an application X, no part of X's user interface is visible onscreen, and it is not perceivable.

System message. A notification generated by Gecko which is dispatched to a certified application X that has registered to listen to the message. If application X is not running when a system message is sent to it, X is launched. For example, an incoming telephone generates a system message that the Dialer application listens for.

Activity. A request generated by a user as part of an interaction with an application X. Activities target an application Y. If Y is not running when an activity targets it, Y is launched. Y may send back data to X as part of the activity. For example, choosing an avatar for a contact generates an activity which may be targeted at the Camera, Gallery, or many other applications.

Criteria

These criteria are only defined for "core Gaia applications" (TODO: enumerate these). Firefox OS developers have no direct control over third-party applications.

MANDATORY: Users' explicit launch of every application X (tap X's icon on homescreen) results in X being loaded and its user interface brought to the foreground.

MANDATORY: Every application X launched by a system message results in X being loaded and provides X the opportunity to display a user interface.

MANDATORY: For every activity initiated by a user in application X that

1. Launches application Y
2. Is "one-way", that is, application Y does not return data to X

application Y must be loaded and its user interface brought to the foreground.

MANDATORY: For every "activity" initiated by a user in application X that

1. Launches application Y
2. Is "two-way", that is, application Y must return data to X

application Y must be loaded and its user interface brought to the foreground. Application X must remain loaded while Y is in the foreground. When application Y returns data to X, X's user interface must be brought to the foreground.

MANDATORY: No foreground application is killed due to memory pressure while a perceivable or background application is alive.

MANDATORY: No perceivable application is killed due to memory pressure while a background application is alive.

QoS: The number of perceivable and background applications that stay alive over the workloads defined below does not regress, while meeting the mandatory criteria above. Informally, a best effort is made to maximize the number of background and perceivable applications that stay alive. Builds that regress the number of apps that stay alive under our workloads are rejected.

(TODO: define if or to what extent third-party code can affect these criteria

Constraints on user data

This section defines the extremes of user data that are considered within the scope of the conformance requirements in this document.

(TODO)

Gallery

• 1000 images? at what max size?
• max image size?

Workloads

This section defines the workloads by which the criteria above are evaluated. Along with each workload, the acceptance status is listed for the given commit IDs below.

Gecko Gaia

MW0: [PASS] Every app is successfully launched into the foreground

MW1: [PASS] Music stays alive

MW2: [PASS] Active call stays active

MW3: [PASS] 112MB: (QoS) Maximum amount of memory that web apps can directly allocate (not including per-app memory)

MW4: [PASS] Apps are successfully launched into the foreground under memory pressure

MW0: Every app is successfully launched into the foreground

1. Ensure wifi is enabled and a network has been successfully configured
2. Long-tap POWER button, choose "Restart"
3. Launch Dialer app, press HOME button
4. Launch SMS app, press HOME button
5. Launch Contacts app, press HOME button
6. Launch Browser app, press HOME button
7. Launch Feedback app, press HOME button
8. Launch Camera app, press HOME button
9. Launch Gallery app, press HOME button
10. Launch FM Radio app, press HOME button
11. Launch Settings app, press HOME button
12. Launch Marketplace app, press HOME button
13. Launch Maps app, press HOME button
14. Launch Calendar app, press HOME button
15. Launch Clock app, press HOME button
16. Launch Cost Control app, press HOME button
17. Launch E-Mail app, press HOME button
18. Launch Music app, press HOME button
19. Launch Video app, press HOME button
20. Launch Calculator app, press HOME button
21. Launch Dev Marketplace app, press HOME button
22. Launch Staging Marketplace app, press HOME button
23. Swipe to Everything.me screen, load the app UI
24. Swipe down system tray UI. Tap Cost Control widget to launch Cost Control app.

MW1: Music stays alive

1. Long-tap POWER button, choose "Restart"
2. Launch Music app
3. Start playing "Treasure Island"
4. Press HOME button
5. Run MW0 starting from step 3

MW2: Active call stays active

1. Long-tap POWER button, choose "Restart"
2. Pull down system tray, ensure Cost Control widget loads
3. Press HOME button
4. Launch CalculatorFeedbackMusic app, press HOME button
5. Launch Clock app, press HOME button
6. Launch FM Radio app, press HOME button
7. Launch Camera app, press HOME button
8. Launch SMS app, press HOME button
9. Launch Browser app
10. Load nytimes.com, wait for load to finish
11. Press HOME button
12. Launch Dialer app
13. Initiate a call to any number
14. Press HOME button
15. Launch Contacts app
16. Choose to add new contact
17. Tap First Name field
18. Enter name "John"
19. Tap "Add Picture"
20. Choose Gallery activity in popup
21. Select image of woman at lecturn
22. Do not crop
23. Tap checkmark button in top-right
24. Tap Done to add contact
25. Press HOME button
26. Launch Browser app
27. Ensure that nytimes.com loads
28. Tap green dialer UI strip at top of screen
29. Tap red button to disconnect call

MW3: (QoS) Maximum amount of memory that web apps can directly allocate

1. Long-tap POWER button, choose "Restart"
2. Open settings app, go to "Display" and set "Screen timeout" to "Never"
3. Go to "Device information", "More information", "Developer" and check "Remote debugging" and "Console enabled"
4. Open browser app, navigate to http://people.mozilla.com/~cjones/membuster.html
5. Tap the "Bust system memory" button
6. Wait for the browser tab to crash
7. Connect phone to USB debugging over adb and run the following command from the shell

adb logcat -d | grep membuster | tail -1 | awk '{ print $(NF) }'

MW4: Apps are successfully launched into the foreground under memory pressure

1. Long-tap POWER button, choose "Restart"
2. Open browser app, navigate to http://people.mozilla.com/~cjones/membuster.html
3. Tap the "Bust processes" button
4. Wait for example.com loads to finish in all tabs
5. Place a call to phone from any number

In-flight improvements

See bug 797189 for the latest list.