NuwaTemplateProcess

Nuwa is a mechanism to fork content processes from a template process. The template process is called Nuwa process. The processes forked from Nuwa process are parasited processes. The basic idea of Nuwa is running Nuwa process like a normal content process, but freeze all its threads after running into a stable state. All threads of Nuwa process will be frozen at a freeze point. The freeze points can be poll, epoll, pthread_mutex_lock, pthread_cond_wait, ... etc, functions that block a thread and release CPU to other threads or processes. For most programs, include B2G, the threads are not always busy. They usually sleep at some point by calling one of above functions. Nuwa process is frozen at freeze points to keep itself at some stable state. Then, Nuwa process forks itself, and its children are used as content processes. All threads are rebuilt and restarted from their freeze points at parasited processes.

Freeze Points

Nuwa make wrappers for all functions being freeze points. All these wrappers are in BionicGlue.cpp. We use --wrap arguments to make GNU LD redirect all function calls of freeze points to respective wrappers. In most time, these wrapper will call respective back functions. For example, the wrapper of pthread_mutex_lock() will call pthread_mutex_lock() to do the task. These wrappers will be blocked on a lock, sThreadFreezeLock, for freezing the calling thread once Nuwa process run into a stable state. We say Nuwa process is frozen if all threads are blocked at these wrappers.

Threads

For Linux system, all threads are not surviving after fork. So, we need to create a new thread and running the frozen context for each thread in Nuwa process; parasite the frozen context on the new thread. To parasite the frozen context on the new thread, we reuse the stack of the frozen thread in the new thread. Each process forked from Nuwa process owns independent memory space, so it does not inference each other to reuse the stack of frozen stack.

 pthread_attr_setstack(attr, stack_ptr, stack_size)

is called to assign a stack for a thread created with the given attributes.

The context of a thread is comprised by registers and a stack. The registers are saved and restored by calling setjmp() and longjmp(). setjmp() is called at freeze point just before freezing the thread. longjmp() is called just at beginning of the new thread. Nuwa passes a special start routine for calling pthread_create(). The routine will call longjmp() ASAP to parasite the context of frozen thread on the new thread.

TLS

Thread local storage, aka TLS, is a part of a thread. TLS provides a separated storage for each thread, the data stored in TLS are accessed through a key. Nuwa process will remember all keys ever been used, and read and remember the associated data just before freeze point for every thread, then restore them at parasited threads. See

• SaveTLSInfo(),
• RestoreTLSInfo(),
• __wrap_pthread_key_create(), and
• __wrap_pthread_key_delete()

in BionicGlue.cpp.

PIPE and Socketpair

For B2G, a lot of PIPEs and sockets are used for synchronization between threads for dispatching tasks among threads. For Linux, file descriptors will be shared after a fork without CLOEXEC. It is fine for normal files but not for communication channels. So, PIPEs and sockets should be recreated. Nuwa remember what pipes and socketpairs are created by Nuwa process and create new ones for them for each parasited process. Since file descriptors of new PIPEs and sockets are not the same as what in Nuwa process, dup2() is called for replacing the file description of old file descriptor by a new PIPE or socket.

Mutics and Conditions

For mutics, there always one of threads will own the lock. Nuwa freeze just before pthread_mutex_lock(), it means still not owning the mutex. So, the wrapper needs only to call pthread_mutex_lock() at parasited thread. If the mutex is already owned by another thread, and this thread is blocked by calling pthread_mutex_lock(), not by sThreadFreezeLock, the wrapper also calls pthread_mutex_lock() and be blocked again at the parasited thread. So, the wrapper of pthread_mutex_lock() always redo the lock in parasited threads.

For conditions of pthread, it is more complex since it can depend on a mutex. Once the pthread_cond_wait() is called, it means the given mutex, if have one, is released. The give mutex is acquired again when the call is returned. For parasited thread, we need to acquire the lock before calling pthread_cond_wait() to restore the context. So, it can be one of two situations, the mutex is hold by some one or free/released. pthread_mutex_trylock() can be used to test if a mutex have been acquired. For free/released situation, pthread_mutex_trylock() will acquire the mutex and pthreaad_cond_wait() will put the mutex back to the released state, so it is fine. For hold state, pthread_mutex_trylock() can not acquire the mutex, pthread_cond_wait() will put the mutex to the released state different from original hold state. So, we need to acquire the mutex to put is back to hold state after the parasited thread was blocked by pthread_cond_wait(). So, the wrapper will ask the main thread to acquire the mutex again, then the state of the mutex will keep consistent with Nuwa process.

Binder

Binder is a special feature of Android, it does not follow the style of UN*X/Linux. ...