Firefox OS/MappedArrayBuffer

Memory-mapped array buffer for XHR response

The feature allows to read data in packaged app by XHR with array buffer type as memory-mapped. The memory-mapped array buffer is created as copy-on-write.

Use Case

The feature helps to save RAM memory usage, especially for big data files in a packaged app. For example, when enabled, the word suggestion(auto correction) or the IME database in the keyboard APP no longer consumes RAM memory. To use this feature, the data file in the packaged app must be uncompressed.

• To determine whether to use it, you may want to consider these factors:
  • RAM size
  • Flash size
  • File size
  • File compression ratio

Support status

• Supported platform
  • Linux
  • Firefox OS(except Windows simulator)

Windows support follow up at https://bugzilla.mozilla.org/show_bug.cgi?id=988813

• Parent process and child process
• Main thread and worker thread
• Preference
  • Use "dom.mapped_arraybuffer.enabled" to enable/disable this feature
  • Enabled for Firefox OS by default

Requirements

• Target file data offset in a packaged app must be
  • Uncompressed
  • Aligned to
    • 8 bytes minimum, and page size[1] recommended[2]
    • After [3] landed, page size aligment is mandatory
• If requirements not meet
  • Fall back to be normal array buffer

[1] Page size on 32-bits Linux is 4096 bytes.

[2] Page alignment of target file in the zip may increase the zip package size a little, but it saves you one page heap memory overhead.

[3] Bug 855669 - Figure out how emscripten &co can decommit memory

How to use it?

You need to make the target files uncompressed and aligned when building a packaged app. To do this, put a configuration file "metadata.json" in the gaia app directory to specify memory-mapped files.

Below is an example of "metadata.json" for keyboard app:

 {
   "external": false,
   "zip": {
     "mmap_files": [
       "js/imes/latin/dictionaries/en_us.dict",
       "js/imes/jszhuyin/data/database.data"
     ]  
   }
 }

After that, the files specified in "mmap_files" will be uncompressed and aligned to 4096 bytes in the zip package.

 $ unzip -v gaia/profile/webapps/keyboard.gaiamobile.org/application.zip | grep Stored
 1451390  Stored  1451390   0% 1970-01-01 08:00 3d22f787  js/imes/latin/dictionaries/en_us.dict
 4541832  Stored  4541832   0% 1970-01-01 08:00 426a7f8e  js/imes/jszhuyin/data/database.data

• For more information, please see:
  • https://bugzilla.mozilla.org/show_bug.cgi?id=957497
  • https://bugzilla.mozilla.org/show_bug.cgi?id=959047

Note: The current approach first makes specified files uncompressed, then re-organize the zip package to make uncompressed files to be 4096 bytes alignment. This could introduce some overhead on zip package size, because some uncompressed files may not want to be read as memory-mapped, which need not and should not be made aligned.

There is a follow up to align for individual files in the zip package https://bugzilla.mozilla.org/show_bug.cgi?id=961622.

How it is done in child process?

See https://bugzilla.mozilla.org/show_bug.cgi?id=988816 for detail.

Q & A

• How to know if the array buffer is memory-mapped in an XHR response?

The Content-Type header is "application/mem-mapped" in such response.

• How to check memory usage?

You can check the memory usage by the following command:

 $ tools/get_about_memory.py --no-gc-cc-log

Below is an example for English and JSZhuyin IME, which means they are not taking any RAM memory.

 │   ├──5.13 MB (27.03%) -- worker(./js/imes/jszhuyin/lib/worker.js, 0xb22dd400)
 │   │  ├──4.57 MB (24.10%) -- zone(0xb104c800)
 │   │  │  ├──4.45 MB (23.46%) -- compartment(web-worker)
 │   │  │  │  ├──4.36 MB (22.99%) -- objects
 │   │  │  │  │  ├──4.33 MB (22.83%) -- non-heap
 │   │  │  │  │  │  ├──4.33 MB (22.83%) ── elements/mapped
 │   │  │  │  │  │  └──0.00 MB (00.00%) ── code/asm.js
 │   │  │  │  │  └──0.03 MB (00.16%) ++ (2 tiny)
 │   │  │  │  └──0.09 MB (00.48%) ++ (2 tiny)
 │   │  │  └──0.12 MB (00.63%) ++ (4 tiny)
 │   │  ├──0.29 MB (01.54%) ++ runtime
 │   │  ├──0.23 MB (01.24%) -- gc-heap
 │   │  │  ├──0.21 MB (01.11%) ── unused-arenas
 │   │  │  └──0.02 MB (00.12%) ++ (2 tiny)
 │   │  └──0.03 MB (00.16%) ++ zone(0xb104bc00)
 │   └──2.02 MB (10.64%) -- worker(js/imes/latin/worker.js, 0xb0d02800)
 │      ├──1.68 MB (08.86%) -- zone(0xb0d0d000)
 │      │  ├──1.56 MB (08.21%) -- compartment(web-worker)
 │      │  │  ├──1.48 MB (07.82%) -- objects
 │      │  │  │  ├──1.38 MB (07.30%) -- non-heap
 │      │  │  │  │  ├──1.38 MB (07.30%) ── elements/mapped
 │      │  │  │  │  └──0.00 MB (00.00%) ── code/asm.js
 │      │  │  │  └──0.10 MB (00.53%) ++ (2 tiny)
 │      │  │  └──0.07 MB (00.39%) ++ (2 tiny)
 │      │  └──0.12 MB (00.65%) ++ (4 tiny)
 │      ├──0.29 MB (01.54%) ++ runtime
 │      └──0.05 MB (00.25%) ++ (2 tiny)

Memory-mapped array buffer is reported at non-heap/elements/mapped

Normal array buffer is reported at malloc-heap/elements/non-asm.js