Gaia/Contacts/Data Refactor
Current situation
The current Contacts app has been growing with time around the MozContacts API, started like a simple application but has been adding more and more requirements.
It was not just the only FirefoxOS app using this API, so MozContacts end up trying to support several requirements from different applications, and at the end of the day what we have is a store for vCard objects, which limited search capabilities.
With the inclusion of external contacts sources like Facebook, Gmail, Hotmail, vCard from SDCard, bluetooth and so on, we have been extending the use of the API via hacks and dark techniques.
Also we added a great feature like contacts merging, everything working under the same MozContacts API.
Motivation
Contacts list performance
Due to the design of Contacts app, we should be able to jump to letters and scroll super fast with the letter shortcuts. It wasn't right to add application specific requirements to MozContacts API so we have been doing black magic to keep the performance at 60fps when loading huge amount of contacts (> 2000).
Search restrictions
Again, adding more and more indexes just cause specific app requirements wasn't a good idea so we had to implement our search based on DOM, which is extremely slow and affects as well to the contacts list rendering time.
Impossible to unmerge contacts
During the process of merging contacts we end up loosing the information of the original contacts as we only have a MozContacts API to store this information. So there is no way to unmerge contacts.
Dependency with Facebook integration
We want other apps different than the Contacts app to be able to read the contacts data imported from external sources like FB, Hotmail, Gmail, etc. However, we have legal restrictions that avoid us to share the data imported from FB with an app that is not owned by Gaia. This made us take this restricted data out of the MozContacts API because this API is accessible by privileged apps. Initially we had to create an independent IndexedDB with FB information that used to live in the Communications app, but once we were able to use the DataStore API we moved this data to a shared DataStore. So now consumers of contacts information that are interested on FB data need to query the MozContacts API and the FB datastore.
Hard to integrate with 3rd party contacts providers
We added extra support for Gmail and Hotmail contacts but this is embedded in the Contacts application itself, which is not scalable. We want to make any 3rd party app capable of providing contacts to have a seamless integration with the Contacts APP.
Architecture
Current context
MozContacts API
Right now is frozen, we know that and will be for good a nice way of storing contacts locally, but we won't be able to add extra requirements.
DataStore API
The Datastore API is a new API designed to make apps share information or making them collaborate to add data to a common store. It allows synchronization and being able to notify about changes. It's a pretty simple key/value store, where both key and value can be any kind of object, and it doesn't implement any kind of search, just direct access by key.
Contacts data consumers (in Gaia)
- Contacts app: Obviously. This is the central hub of contacts information in Gaia. We want the user to have access to the whole list of contacts whichever its source is. We want local contacts and contacts imported from external sources like FB, Gmail, Hotmail, etc. The user won't be able to modify the contact information from an external source. All changes done to contacts will be kept locally and won't affect its remote replication. Each contact provider is responsible for its own data. In this case, the Contacts app is only provider and so responsible of the MozContacts API data, although this data can also be modified by other apps with enough permissions.
- Messaging app: Whenever the user sends or receives an SMS or MMS, we need to get the information associated to the contact(s) subject of this message. We are also allowing users to do quick searches by contact name and phone number when composing a new message. We are currently querying the MozContacts API and the FB datastore.
- Dialer app: Same situation as above, but we are only doing searches by phone number. We are currently also querying both stores, MozContacts and FB.
- Emergency calls: We do a basic search by contact ID on the MozContacts store to show the ICE contacts bar if appropriate. No FB search so far, but we want users to be able to set FB contacts as ICE contacts, so we will need it in the future. No need for quick incremental searches by phone number or any other field.
- Email app: The email app queries the MozContacts API by ID and email. No need for incremental searches so far, but we might need it at some point. No queries to the FB datastore.
- Search app: Currently disabled. So far there is a search by name implemented. It should be able to search any contact (include FB data) by any field at some point (?).
Proposal
High level overview
Principles
Contact Providers
We want a generic mechanism for apps to share contact information and so became Contact Providers. An app that wants to share each contact information should:
- Be the owner of their own information (CRUD). Contact Providers are the only responsible for adding, updating and removing contacts on their own stores. The Global Contact Data Store won't have write access to other provider's store.
- Provide a readonly datastore named contacts with the information it shares.
- That datastore will have a unique index per contact as key and a *MozContact* object as value.
Global Contacts DataStore (GCDS)
This is the global hub of information about Contact Providers and the data they share. We will have a certified app with no UI that will do the following:
- Provide a datastore, Global Contacts Datastore (GCDS), that gives a vision of all contacts in the different datastores owned by the different Contact Providers of the system. This global datastore will contain an entry for each and a pointer to the original contact source.
- Merge passively contacts that are detected to be the same ones. Active merge (with UI) is responsability of each Contact Provider.
- Provide a helper library for users to have the view of a single contact coming from it's different (or unique) sources.
- Listen to changes in other datastores to keep the whole view unified.
- Able to unmerge contacts references on demand.
Local contacts
- A new datastore owned by the system will be provided and will contain the same contact related information stored in the MozContacts API indexed DB.
- The content of this datastore will be modifiable via the existing MozContacts API only and the GCDS will treat this datastore as it treats any other datastore owned by any other Contact Provider.
- Because the MozContacts API can be accessed by privileged apps, certified only data won't be added to this datastore. This is the case of FB data.
Contacts Data Consumer
- The Contacts app is considered a Contacts Consumer, just like the rest of the apps listed on Contacts Data Consumers in Gaia
- Each consumer of the GCDS will need to build their own indexedDB from the data obtained through the GCDS and the datastore owned by the pointed Contact Providers. This indexedDB should contain the specific indexes required for the specific needs of the app like incremental searches, jump to letter, better cursor walking, etc.
Data flows
Contact consumer initial fetch
Local contact addition
Local contact change
External contact addition
External contact change
Contact consumer search
Requirements
In order to do this we sill have some requirements that needs to be implemented in the platform to allow us do the following:
- Expose mozContacts API content as a datastore:
- bug 1016838 Bug 1016838 - Provide a system-level datastore for local contacts
- Better notification when data changes in Datastores:
- In order to make the flow happen when changes are performed in the different datastores, we will need to be able to receive system messages if a DS changes in our app: bug 1014023 [Datastore] Notify apps of changes in datastore without being opened
FAQ
- What happen if a new app is installed and offers contacts?
We have filled bug 968278 against platform to get notified of datastores of one kind (contacts in this case) being installed or removed to act accordinly.
- Where lives the data?
In each application. This works in both sides, providers, and consumers as well.
- What happen if I create a new contact?
That will be a local contact, accessible from mozContacts API and it's datastore. A reference in the GCDS pointing to the original source will be created as well (unless merged with an existing one).
- What happen if I modify a contact from a external source?
TBD, we need to talk with the sync team, but we expect that each source of contacts could decide it via datastore permissions (TBD).
- Will this GCD have access to the contacts of an specific app?
We want the user to decide that, which external datastores will be able to being used by the GCDS.
Known issues
Implementation
Bugs associated
14 Total; 14 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Experiments
- Below you can find a list of experiments we did to check how our ideas perform, feasibility and performance:
- bug 1031315 - [Contacts][Data Refactor]Check performance of building contact details from different DS
- Here we tried to check how much it cost to build a contact detail from different sources each time, compared with storing the result in indexeddb and retrieving it.
- With 500 contacts, obviously is much faster the indexedDB option, but we wanted to see how much is the difference between bot solutions having 30-40ms to get the data from indexedDb and ~150ms while composing from different sources.
- To be sure that we don't consume much space on disk, we decided to cache the detail of those contacts that we visit.
- bug 1031315 - [Contacts][Data Refactor]Check performance of building contact details from different DS
- bug 1031318 - [Contacts][Data Refactor] Better search in contacts
- All in memory: 1MB with 2000 contacts
- Indexeddb: Problems with traversing, we need 8 seconds to do this.
- Finally we have decided to go with indexeddb since the memory consumption is too high, also we need to check concepts like suffix array to see if we can apply them to indexeddb.
- bug 1031318 - [Contacts][Data Refactor] Better search in contacts
- bug 1031327 - [Contacts][Data Refactor] Measure time to finish cursor when information stored is a full mozContact or just the bare minimun information (for 2000 contacts)
- Time to first contact is more or less the same ~ 80-100 ms
- Time to last contact with mozContacts is around 6.6~6.8 seconds
- Time to last contact coocked with indexeddb is around 2.8~3 seconds
- Storing in indexeddb a set of minimun information without having to perform operations to calculate what to display helps incredibly.
- bug 1031327 - [Contacts][Data Refactor] Measure time to finish cursor when information stored is a full mozContact or just the bare minimun information (for 2000 contacts)
- Other findings:
- Thumbnails are expensive in space, perhaps we could offer a thumbnail DS for certified apps.
- Other findings: