NuwaTemplateProcess

Nuwa is a mechanism to fork content processes from a template process. The template process is called Nuwa process. The processes forked from Nuwa process are parasited processes. The basic idea of Nuwa is running Nuwa process like a normal content process, but freeze all its threads after running into a stable state. All threads of Nuwa process will be frozen at a freeze point. The freeze points can be poll, epoll, pthread_mutex_lock, pthread_cond_wait, ... etc, functions that block a thread and release CPU to other threads or processes. For most programs, include B2G, the threads are not always busy. They usually sleep at some point by calling one of above functions. Nuwa process is frozen at freeze points to keep itself at some stable state. Then, Nuwa process forks itself, and its children are used as content processes. All threads are rebuilt and restarted from their freeze points at parasited processes.

Freeze Points

Nuwa make wrappers for all functions being freeze points. All these wrappers are in BionicGlue.cpp. We use --wrap arguments to make GNU LD redirect all function calls of freeze points to respective wrappers. In most time, these wrapper will call respective back functions. For example, the wrapper of pthread_mutex_lock() will call pthread_mutex_lock() to do the task. These wrappers will be blocked on a lock, sThreadFreezeLock, for freezing the calling thread once Nuwa process run into a stable state. We say Nuwa process is frozen if all threads are blocked at these wrappers.

Threads

For Linux system, all threads are not surviving after fork. So, we need to create a new thread and running the frozen context for each thread in Nuwa process; parasite the frozen context on the new thread. To parasite the frozen context on the new thread, we reuse the stack of the frozen thread in the new thread. Each process forked from Nuwa process owns independent memory space, so it does not inference each other to reuse the stack of frozen stack.

 pthread_attr_setstack(attr, stack_ptr, stack_size)

is called to assign a stack for a thread created with the given attributes.

The context of a thread is comprised by registers and a stack. The registers are saved and restored by calling setjmp() and longjmp(). setjmp() is called at freeze point just before freezing the thread. longjmp() is called just at beginning of the new thread. Nuwa passes a special start routine for calling pthread_create(). The routine will call longjmp() ASAP to parasite the context of frozen thread on the new thread.

TLS

Thread local storage, aka TLS, is a part of a thread. TLS provides a separated storage for each thread, the data stored in TLS are accessed through a key. Nuwa process will remember all keys ever been used, and read and remember the associated data just before freeze point for every thread, then restore them at parasited threads. See

• SaveTLSInfo(),
• RestoreTLSInfo(),
• __wrap_pthread_key_create(), and
• __wrap_pthread_key_delete()

in BionicGlue.cpp.

Thread ID

The thread ID of parasited threads are different from frozen threads. So, Nuwa make a wrapper for pthread_self() to map thread ID of parasited threads to thread IDs of frozen threads. Some code use Linux depend gettid() syscall, I don't see any reason why people use gettid instead of pthread_self() for most situation. So, change the code using gettid() to call pthread_self() instead. Even more worse, some code call syscall() to invoke gettid. A wrapper of syscall() affects a lot of code, studying more is required before making a decision.

PIPE and Socketpair

For B2G, a lot of PIPEs and sockets are used for synchronization between threads for dispatching tasks among threads. For Linux, file descriptors will be shared after a fork without CLOEXEC. It is fine for normal files but not for communication channels. So, PIPEs and sockets should be recreated. Nuwa remember what pipes and socketpairs are created by Nuwa process and create new ones for them for each parasited process. Since file descriptors of new PIPEs and sockets are not the same as what in Nuwa process, dup2() is called for replacing the file description of old file descriptor by a new PIPE or socket.

Mutics and Conditions

For mutics, there at most one of threads will own the lock. Nuwa freeze just before pthread_mutex_lock(), it means still not owning the mutex. So, the wrapper needs only to call pthread_mutex_lock() at parasited thread. If the mutex is already owned by another thread, and the frozen thread is blocked by calling pthread_mutex_lock(), not by sThreadFreezeLock, the wrapper also calls pthread_mutex_lock() and be blocked again at the parasited thread. So, the wrapper of pthread_mutex_lock() always redo the lock in parasited threads.

For conditions of pthread, it is more complex since it can depend on a mutex. Once the pthread_cond_wait() is called, it means the given mutex, if have one, is released. The give mutex is acquired again when the call is returned. For parasited thread, we need to acquire the lock before calling pthread_cond_wait() to restore the context. So, it can be one of two situations, the mutex is hold by some one or free/released. pthread_mutex_trylock() can be used to test if a mutex have been acquired. For free/released situation, pthread_mutex_trylock() will acquire the mutex and pthreaad_cond_wait() will put the mutex back to the released state, so it is fine. For hold state, pthread_mutex_trylock() can not acquire the mutex, pthread_cond_wait() will put the mutex to the released state different from original hold state. So, we need to acquire the mutex to put is back to hold state after the parasited thread was blocked by pthread_cond_wait(). So, the wrapper will ask the main thread to acquire the mutex again, then the state of the mutex will be consistent with Nuwa process.

Dark Magic of Stack

Binder

Binder is a special feature of Android, it does not follow the paradigm of UN*X/Linux. ...

IPC

IPC of Gecko/B2G is comprised by various protocols. The peers of a protocol instance are represented by a pair of actors; For example PBlobParent and PBlobChild are base classes of PBlob actors of parent and child side respective. PBlob* are generated by the IPDL code generator. Nuwa does not do any special for child actors for parasite processes since a new memory space are created and COW. But, for parent actors in the parent process, Nuwa should clone them and replace their IPC channels with new ones. Nuwa process will create new socket pairs for actors as the IPC channel between the new parasite process and the parent process.

For running IPC between Nuwa process and the parent process, the IO thread and the main thread should not be frozen. For the main thread, no any special should be done, it just continue the control flow after fork. But, for the IO thread, it should create a parasite point just before the fork to tell Nuwa where to restart the thread. See

• ParasitePointThreadNuwa() in BionicGlue.cpp, and
• DoParasitePoint() in ContentChild.cpp.

Defense

Share Data Among Content Processes

With Nuwa, static data could be shared among content processes with just one copy and copy-on-write. If you think some deserve to be shared, you should load it at the Nuwa process before being frozen. PreloadSlowThings() is a good place to load/or initialize the data. Please check dom/ipc/preload.js of m-c.

Preloading

Preloading in B2G is a mechanism that loads assets and modules prior to opening an app to improve the app loading time. Preloading in Nuwa can leverage Nuwa's copy-on-write advantage to minimize memory space occupied by preloaded data shared accross process. However, there are 2 reasons that make preloading not be able to run in Nuwa.

• Preloading modules initiate asynchronous tasks, some of the tasks are even dispatched to chrome process through IPC. The asynchronous natural makes Nuwa hard to determine whether or not all of the tasks are done and to decide when to freeze. A callback invoked after Nuwa frozen can make Nuwa's main thread try to acquire a lock held by a frozen thread, which results in deadlock.
• Some modules setup ppmm/cpmm-based communication channels in there initialization processes. Since the channels are not based on IPDL, Nuwa won't duplicate the channels for the newly-forked process, thus the communication channels have to be built again after a process has been forked from Nuwa.

To address the first issue, a method of determining whether the task are all done or not has been proposed. The idea is that we monitor each thread in chrome process, and see if there's a moment that all of the threads are idle. Since we are not accessing network during boot, having all of the threads idle means all of the tasks previously generated are consumed, and Nuwa can freeze. The second issue is addressed by splitting preload.js into 2 parts. The first part is where to put modules that are good to be loaded in Nuwa, while the second part contains modules that are not suitable to be loaded in Nuwa and scripts that rebuild communication channels. See

• Bug 970307