Firefox OS/MappedArrayBuffer
Jump to navigation
Jump to search
Memory-mapped array buffer for XHR response
The feature allows to read data in packaged app by XHR with array buffer type as memory-mapped. The memory-mapped array buffer is created as copy-on-write.
Use Case
The feature helps to save RAM memory usage, especially for big data files in a packaged app. For example, when enabled, the word suggestion(auto correction) or the IME database in the keyboard APP no longer consumes RAM memory. To use this feature, the data file in the packaged app must be uncompressed.
- To determine whether to use it, you may want to consider these factors:
- RAM size
- Flash size
- File size
- File compression ratio
Support status
- Supported platform
- Linux
- Firefox OS(except Windows simulator)
- Windows support follow up at https://bugzilla.mozilla.org/show_bug.cgi?id=988813
- Parent process and child process
- Main thread and worker thread
- How to enable?
- Use preference "dom.mapped_arraybuffer.enabled" to enable/disable this feature
- Enabled for Firefox OS by default
Requirements
- Target file data offset in a packaged app must be
- Uncompressed
- Aligned to
- 8 bytes minimum, and page size[1] recommended[2]
- After [3] landed, page size aligment is mandatory
- If requirements not meet
- Fall back to be normal array buffer
- [1] Page size on 32-bits Linux is 4096 bytes.
- [2] Page alignment of target file in the zip may increase the zip package size a little, but it saves you one page heap memory overhead.
- [3] Bug 855669 - Figure out how emscripten &co can decommit memory
How to use it?
- You need to make the target files uncompressed and aligned when building a packaged app. To do this, put a configuration file "metadata.json" in the gaia app directory to specify memory-mapped files.
- Below is an example of "metadata.json" for keyboard app:
{
"external": false,
"zip": {
"mmap_files": [
"js/imes/latin/dictionaries/en_us.dict",
"js/imes/jszhuyin/data/database.data"
]
}
}
- After that, the files specified in "mmap_files" will be uncompressed and aligned to 4096 bytes in the zip package.
$ unzip -v gaia/profile/webapps/keyboard.gaiamobile.org/application.zip | grep Stored 1451390 Stored 1451390 0% 1970-01-01 08:00 3d22f787 js/imes/latin/dictionaries/en_us.dict 4541832 Stored 4541832 0% 1970-01-01 08:00 426a7f8e js/imes/jszhuyin/data/database.data
- For more information, please see:
Note: The current approach first makes specified files uncompressed, then re-organizes the zip package to make uncompressed files to aligned by padding in extended field of the header. This could introduce some overhead on zip package size, because not all uncompressed files are expected to be read as memory-mapped, which need not and should not be made aligned.
- There is a follow up to align for individual files in the zip package https://bugzilla.mozilla.org/show_bug.cgi?id=961622.
How it is done in child process?
- See https://bugzilla.mozilla.org/show_bug.cgi?id=988816 for detail.
Q & A
- How to know if the array buffer is memory-mapped in an XHR response?
- The Content-Type header is "application/mem-mapped" in such response.
- How to check memory usage?
- You can check the memory usage by the following command:
$ tools/get_about_memory.py --no-gc-cc-log
- Below is an example for English and JSZhuyin IME, which means they are not taking any RAM memory.
│ ├──5.13 MB (27.03%) -- worker(./js/imes/jszhuyin/lib/worker.js, 0xb22dd400) │ │ ├──4.57 MB (24.10%) -- zone(0xb104c800) │ │ │ ├──4.45 MB (23.46%) -- compartment(web-worker) │ │ │ │ ├──4.36 MB (22.99%) -- objects │ │ │ │ │ ├──4.33 MB (22.83%) -- non-heap │ │ │ │ │ │ ├──4.33 MB (22.83%) ── elements/mapped │ │ │ │ │ │ └──0.00 MB (00.00%) ── code/asm.js │ │ │ │ │ └──0.03 MB (00.16%) ++ (2 tiny) │ │ │ │ └──0.09 MB (00.48%) ++ (2 tiny) │ │ │ └──0.12 MB (00.63%) ++ (4 tiny) │ │ ├──0.29 MB (01.54%) ++ runtime │ │ ├──0.23 MB (01.24%) -- gc-heap │ │ │ ├──0.21 MB (01.11%) ── unused-arenas │ │ │ └──0.02 MB (00.12%) ++ (2 tiny) │ │ └──0.03 MB (00.16%) ++ zone(0xb104bc00) │ └──2.02 MB (10.64%) -- worker(js/imes/latin/worker.js, 0xb0d02800) │ ├──1.68 MB (08.86%) -- zone(0xb0d0d000) │ │ ├──1.56 MB (08.21%) -- compartment(web-worker) │ │ │ ├──1.48 MB (07.82%) -- objects │ │ │ │ ├──1.38 MB (07.30%) -- non-heap │ │ │ │ │ ├──1.38 MB (07.30%) ── elements/mapped │ │ │ │ │ └──0.00 MB (00.00%) ── code/asm.js │ │ │ │ └──0.10 MB (00.53%) ++ (2 tiny) │ │ │ └──0.07 MB (00.39%) ++ (2 tiny) │ │ └──0.12 MB (00.65%) ++ (4 tiny) │ ├──0.29 MB (01.54%) ++ runtime │ └──0.05 MB (00.25%) ++ (2 tiny)
- Memory-mapped array buffer is reported at non-heap/elements/mapped
- Normal array buffer is reported at malloc-heap/elements/non-asm.js
- Can we write to a memory-mapped array buffer?
- The array buffer is mapped as copy-on-write by private mapping. If you write to a memory-mapped array buffer, it's not visible to other processes mapping the same file, and the updates are not carried through to the underlying file. However, you should avoid that because it introduces heap memory allocation when writing to the address of each page in the virtual address space.
- Is it possible to apply memory-mapped array buffer on blobs coming from IndexedDB?
- Yes, there is a follow up at https://bugzilla.mozilla.org/show_bug.cgi?id=988815