Lightning Offline Sync

This page is for tracking the Lightning Offline Sync Feature (bug 380060) to be completed as part of the Google Summer of Code 2011 programme.

The Idea

Essentially the idea is to have the cached calendar function as a full-blown offline counterpart as well without having the need to implement any additional providers or managers. The information about offline user actions is retained in the cache and replayed when going online.

The state transition table below gives the states which the offline item will transition to so as to preserve minimum number of calls to the network providers.

TODOs

Item Name	Priority	Owner	Deadline	Progress	Comments
Adapt schema initialization and versioning code (calCalendarManager)		mohit		done	Added versioning code in calStorageUpgradeHelper.jsm file. cal_events and cal_todos have an extra "offline_journal" column. and DB SCHEMA Version is 20.
Add search flags "ITEM_FILTER_OFFLINE_DELETED", "ITEM_FILTER_OFFLINE_CREATED" and "ITEM_FILTER_OFFLINE_UPDATED"		wolfgang		done
Check whether "ITEM_FILTER_ALL_ITEMS" is used in the views, if so, we need to create "ITEM_FILTER_ALL_ONLINE_ITEMS" ("ITEM_FILTER_ALL_ITEMS" & ~ITEM_FILTER_OFFLINE_DELETED)		wolfgang		cancelled	Not needed since ALL_ITEMS = 0xffff and our flags are = [29..31]
Adapt getItems in calStorageCalendar with the above and the GUI code if required		mohit		done
Adapt mSelectNonRecurringEventsByRange and this.mSelectNonRecurringTodosByRange queries		mohit		done
Write offline addItem method (calStorageCalendar::addOfflineItem)		mohit		done	To be discussed
Write offline delete method (calStorageCalendar::deleteOfflineItem)		mohit		done
Write offline modify method (calStorageCalendar::modifyOfflineItem)		mohit		done	*Note* cachedCalendar::modifyItem deletes the old item before putting in the new item with the changes hence the offline_flag is lost. Solution: make offline_journal flag part of the interface..
Test the above		wolfgang/mohit		done	Testing steps should be discussed
Write replay method		mohit		done
Write "switch code" in calCachedCalendar		wolfgang		done	switch code = offline to online mode and back
test the above		wolfgang/mohit		done	Testing steps shud be discussed
Design a workflow for handling errors pertaining to unavailability of server + replaying the changes when back online		mohit		done		Needs discussion with Wolfgang afterwards
Implement the above workflow after discussion		mohit		done
Adapt code from Lightning 1.0b2 to HEAD		mohit		done		Should be done at the very end of the above

---

Workflows

Offline operations

• addItem:

1. (entry) calCachedCalendar::addItem
2. calStorageCalendar::addOfflineItem
   1. setup calOperationListener
   2. execute calStorageCalendar::addItem with above listener
   3. listener: update flag with 'c' value (see table below)
   4. listener: invoke original listener with result code

• modifyItem

1. (entry) calCachedCalendar::modifyItem
2. calStorageCalendar::modifyOfflineItem
   1. setup calOperationListener
   2. execute calStorageCalendar::modifyItem with above listener
   3. listener: update flag with 'm' value (see table below)
   4. listener: invoke original listener with result code

• deleteItem

1. (entry) calCachedCalendar::deleteItem
2. calStorageCalendar::deleteOfflineItem
   1. update flag with 'd' value (see table below)
   2. invoke original listener with result code

Note: we cannot delete the record yet as we will need to pass the complete calItem back to the online/uncached calendar during reconcilliation

• getItem/getItems

1. code forwarded immediately to the calStorageCalendar

Note: getItems excludes results with offline column set to 'd', unless the ITEM_FILTER_OFFLINE_DELETED flag is passed via the item filter

Transition between online and offline modes

Nothing seems to be required at this stage.

Transition between offline and online modes

New code must be added to calCachedCalendar::onOfflineStatusChanged:

• manage deletions

1. setup calOperationListener "listener1"
2. invoke calStorageCalendar::getItems with ITEM_FILTER_OFFLINE_DELETE in item_filter and the above listener
3. listener1 execution:
   1. onGetResult:
      1. setup empty calOperationListener "listener2"
      2. invoke calCachedCalendar::deleteItem (this) with each resulting events and listener2
   2. onOperationComplete:
      1. perform next step ("manage additions")

• manage additions

1. setup calOperationListener "listener1"
2. invoke calStorageCalendar::getItems with ITEM_FILTER_OFFLINE_CREATED in item_filter and the above listener
3. listener1 execution:
   1. onGetResult:
      1. setup empty calOperationListener "listener2"
      2. invoke calCachedCalendar::addItem (this) with each resulting events and listener2
   2. onOperationComplete:
      1. perform next step ("manage modifications")

• manage additions

1. setup calOperationListener "listener1"
2. invoke calStorageCalendar::getItems with ITEM_FILTER_OFFLINE_MODIFIED in item_filter and the above listener
3. listener1 execution:
   1. onGetResult:
      1. setup empty calOperationListener "listener2"
      2. invoke calCachedCalendar::modifyItem (this) with each resulting events and listener2
   2. onOperationComplete:
      1. perform calCachedCalendar::refresh (this)

Note: reconciliation must occur before calIChangeLog method because "uncached calendars" (at least calDavCalendar) contain code that manages conflicts during data upload. Otherwise, the local changes would immediately be overwritten by server-side changes.

Efficient model for state updates during offline operations

From/To	Created	Modified	Deleted
(without state)	Mark as "created"	Mark as "modified"	Mark as "deleted"
Created	(none)	Keep as "created"	Delete record
Modified	(none)	Keep as "modified"	Mark as deleted
Deleted	(none)	(none)	(none)

Additional issues

1. CalDAV provider does not handle offline mode at all due to the fact that it performs the same DAV checks every time it's instantiated
   1. (done) Calendar properties (organizerCN, organizerCN, imip.identity + CalDAV capabilities + items metadata) should be stored in the cache for offline use + most of those attributes should never be queried anymore
   2. (postponed) certain checks should be only performed in a separate method that should be executed when the calendar is created from the calendar creation dialog
   3. (done) the calDavCalendar does not know its storage counterpart when calICalendar operations are first invoked, this should preferably be done right after instantiation, especially since "cache mode" will become the sole available mode, sooner or later
2. (cancelled: since those interfaces are not used by the same components, it makes sense to keep them distinct) we should probably merge calIOfflineStorage with calISyncWriteCalendar into a single interface as those are actually part of the same functionality (and choose the former name since it's more significant)
3. any calendars that are not available due to network errors should automatically be put in offline mode and enable further operations
4. more or less off-topic: the caldav conflict dialog does not show the differences between events