Gaia/Contacts/Data Refactor

Current situation

Current contacts app have been growing with time around the mozContact API, started like a simple application but has been adding more and more requirements.

It was not just the only FirefoxOS app using this api, so mozContacts end up trying to support several requirements from different applications, and at the end of the day what we have is a store for vcard objects, which limited search capabilities.

With the inclusion of external contacts sources like Facebook, Gmail, Hotmail, vcard from sdcard, bluethoot and so on, we have been extending the use of the API via hacks and dark techniques.

Also we added a great feature like contacts merging, everything working under the same mozContacts API.

Problems

Some problems that we are facing now are the following:

• Contacts list performance:
  • Due to the design of contacts app, we should be able to jump to letters and scroll super fast with the letter shor cuts. It wasn't right to add application specific requirements to mozContacts API so we have been doing black magic to keep the performance at 60fps when loading huge amount of contacts ( > 2000).

• Search restrictions:
  • Again, adding more and more indexes just cause specific app requirements wasn't a good idea so we had to implement our search based on DOM, which is extremly slow and affects as well to the contacts list rendering time.

• Impossible to unmerge contacts:
  • During the process of merging contacts we end up losing the information as we just have a mozContacts API, we lose data when we merge.

• Dependency with Facebook integration:
  • When we started to create the FB integration we needed to somehow make the contacts live in mozContacts but as well apply some legal restrictions to how data was shared. We didn't have the same technology that we have right now for doing that and this made our code being dependant of one partner.

• Hard to integrate with 3rd party contacts providers:
  • We added extra support for Gmail and Hotmail contacts but this is embeded in the Contacts application itself, which is not scalable. We want to make any app capable of provide contacts to have a seamless integration with the Contacts APP.

Architecture

Current context

• mozContacts API
  • Right now is frozen, we know that and will be for good a nice way of storing contacts locally, but we won't be able to add extra requirements.
• DataStore API
  • The Datastore API is a new API designed to make apps share information or making them collaborate to add data to a common store. It allows synchronization and being able to notify about changes. It's a pretty simple key/value store, where both key and value can be any kind of objectm, and it doesn't implement any kind of search, just direct access by key.

Proposal

Data flow

Principles

• There are many apps capable of provide contacts, each app should:
  • Be the owner of their own information (CRUD).
  • Provide a datastore name contacts with the information it shares.
  • That datastore will have a unique index per contact as key and a *mozContact* object as value.

• There will be an app without UI that will:
  • Provide a datastore, Global Contacts Datastore (GCDS), that gives a vision of all contacts in the different datastores (applications) of the system.
  • Merge passively contacts that are detected to be the same ones.
  • Will offer pointers to the data in the original datastores.
  • Provide a helper library for users to have the view of a single contact coming from it's different (or unique) sources.
  • Listen to changes in other datastores to keep the whole view unified.
  • Able to unmerge contacts references on demand.

• Local contacts
  • Will continue using mozContacts API.
  • A new system Datastore will be provided with the local contacts contained in mozContacts API.

• Contacts APP
  • As any other client app of the GCDS will need to build their own indexedDB to be able to list and search information.
  • Create specific indexes to make current requirements being searchable (name, given, family, org, tel, email, phonetic, address ...)
  • Use directly indexedDB to provide a better peforming contacts list (jump to letter, better cursor walking ...).

Requirements

In order to do this we sill have some requirements that needs to be implemented in the platform to allow us do the following:

• Expose mozContacts API content as a datastore:
  • bug 1016838 Bug 1016838 - Provide a system-level datastore for local contacts
• Better notification when data changes in Datastores:
  • In order to make the flow happen when changes are performed in the different datastores, we will need to be able to receive system messages if a DS changes in our app: bug 1014023 [Datastore] Notify apps of changes in datastore without being opened

Questions

• What happen if a new app is installed and offers contacts?

We have filled bug 968278 against platform to get notified of datastores of one kind (contacts in this case) being installed or removed to act accordinly.

• Where lives the data?

In each application. This works in both sides, providers, and consumers as well.

• What happen if I create a new contact?

That will be a local contact, accessible from mozContacts API and it's datastore. A reference in the GCDS pointing to the original source will be created as well (unless merged with an existing one).

• What happen if I modify a contact from a external source?

TBD, we need to talk with the sync team, but we expect that each source of contacts could decide it via datastore permissions (TBD).

• Will this GCD have access to the contacts of an specific app?

We want the user to decide that, which external datastores will be able to being used by the GCDS.

Bugs associated

Bugzilla link

Full Query

ID	Assigned to	Summary	Resolution	Assigned to	Depends on	Whiteboard	Blocking b2g	Feature b2g	Target milestone
968099		Separate Gmail importer in an independent app loaded from contacts / ftu	WONTFIX			[priority]	---	---	---
968278		Contacts should be able to find dynamically datastores providing contacts information	INVALID			[priority]	---	---	---
982152		[Contacts] Transition search adapters to data objects	WONTFIX		982081	[c= p=5 s= u=] [priority]	---	---	---
989923	Francisco Jordano [:arcturus] [:francisco]	[Contacts][Refactor] New app to host Contacts DataStore in test apps	FIXED	Francisco Jordano [:arcturus] [:francisco]			---	---	1.4 S6 (25apr)
989925	Sergi Mansilla [:sergi] (Telenor) [PTO]	[Contacts][Refactor] Implement the corresponding mechanisms to fetch/update contacts in Global Contacts Datastore	WONTFIX	Sergi Mansilla [:sergi] (Telenor) [PTO]	1014023, 1046667	[priority][p=4]	---	---	2.2 S2 (19dec)
989927	Jose Manuel Cantera	[Contacts][Refactor] Create a library that reads the Global Contacts DataStore and provide a single contact	FIXED	Jose Manuel Cantera		[priority][p=2]	---	---	2.0 S4 (20june)
989928		[Contacts][Refactor] Implement/Check different access levels based on app manifest	WONTFIX			[priority]	---	---	---
989929	Jose Manuel Cantera	[Contacts][Refactor] Create Contacts indexed DB library to support synchronization from the Global Contacts Datastore	FIXED	Jose Manuel Cantera		[priority] [p=2]	---	---	2.0 S5 (4july)
989932		[Contacts][Refactor] Implement a sync mechanism from Global Contacts DataStore and Contacts app own store	WONTFIX		989929, 1081143	[priority][p=10]	---	---	---
1013940	Francisco Jordano [:arcturus] [:francisco]	[Contacts][Data Refactor] Add examples applications to play with extra contact datastores	FIXED	Francisco Jordano [:arcturus] [:francisco]			---	---	2.0 S2 (23may)
1026473	Jose Manuel Cantera	[Contacts][Refactor] Implement the Global Contacts Datastore data management module	FIXED	Jose Manuel Cantera		[p=2]	---	---	2.0 S6 (18july)
1041494		[Contacts][Data Refactor] Expose a DS for thumbnails	WONTFIX				---	---	---
1041496		[Contacts][Data Refactor] Hide matching index information from the GCDS	WONTFIX				---	---	---
1081082	Jose Manuel Cantera	[Contacts] [Data Refactor] Provide a convenient way for providers to get access to their 'contacts' datasore	FIXED	Jose Manuel Cantera		[p=3]	---	---	2.1 S7 (24Oct)

14 Total; 14 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Experiments

• Below you can find a list of experiments we did to check how our ideas perform, feasibility and performance:
  • bug 1031315 - [Contacts][Data Refactor]Check performance of building contact details from different DS
    • Here we tried to check how much it cost to build a contact detail from different sources each time, compared with storing the result in indexeddb and retrieving it.
    • With 500 contacts, obviously is much faster the indexdDB option, but we wanted to see how much is the difference between bot solutions having 30-40ms to get the data from indexedDb and ~150ms while composing from different sources.
    • To be sure that we don't consume much space on disk, we decided to cache the detail of those contacts that we visit.

• • bug 1031318 - [Contacts][Data Refactor] Better search in contacts
    • All in memory: 1MB with 2000 contacts
    • Indexeddb: Problems with traversing, we need 8 seconds to do this.
    • Finally we have decided to go with indexeddb since the memory consumption is too high, also we need to check concepts like suffix array to see if we can apply them to indexeddb.

• • bug 1031327 - [Contacts][Data Refactor] Measure time to finish cursor when information stored is a full mozContact or just the bare minimun information (for 2000 contacts)
    • Time to first contact is more or less the same ~ 80-100 ms
    • Time to last contact with mozContacts is around 6.6~6.8 seconds
    • Time to last contact coocked with indexeddb is around 2.8~3 seconds
    • Storing in indexeddb a set of minimun information without having to perform operations to calculate what to display helps incredibly.

• • Other findings:
    • Thumbnails are expensive in space, perhaps we could offer a thumbnail DS for certified apps.