Device Sync

We are working on the ability to synchronize calendar with handhelds (pda's). This should work with a wide range of devices, and on the main platforms (windows, linux, mac). It should work with all types of calendars that we offer (ics, caldav, sqlite)

• On linux, communicating with the handheld can either be done directly (libvisor) or by third party software, such as multisync or opensync. OpenSync is a general synchronization framework that currently runs on linux and mac. A windows port is also planned. The plugins are ported from the multisync project, so everything available there, will also be available in opensync (palm support, irmc, syncml, gpe, opie, wince devices). You can also have a look at the short whitepaper describing opensync. Since opensync is just a library it would be possible to embed it directly in the application.
• On windows, there is activesync and hotsync (iirc). But I don't know enough of them to decide what to use. What are its capabilities? Does it have sync logic build in?
• AIUI, HotSync is a Palm-specific (closed) sync client, while ActiveSync is (closed) WinCE-specific. You can write conduits for HotSync; I think there are dev kits available for this. I don't know about ActiveSync. Syncing to mobile devices also includes mobiles IMHO, which will probably use a mix of sync clients (e.g. Siemens S55 uses XTNDConnectPC, Motorola use StarFish's TrueSync) which could all probably have "conduit" equivalents written for them. I don't know if this is the approach you want - it's potentially a lot of different interfaces. -- BenS
• Concerning ActiveSync, there is a possibility to write conduits. This however involves serious headaches, as IIRC the API is a true Microsoft spec, if you know what I mean. Therefore many 3rd party sync utilities rely on some sort of client-pull mechanism on the PocketPC side. (see Funambol)
• An addition to BenS comment above: Late-model Sony Ericsson phones also use XTNDConnectPC to sync with Windows and also directly with Microsoft Outlook.
• On mac, i don't know the situation at all.
• There's something called ".Mac" sync on OS X. To integrate with this, you can use the DotMac SDK [[1]] -- BenS
• While iSync from Mac OS 10.3 is not syncML compatible, iSync & .Mac sync (included with Mac OS 10.4) both are.

Calendar already needs to have support for syncing two calICalendars (see Calendar:Networking Sharing Syncing). The logic to sync two calendars could be reused, either by implementing a calICalendar for the handheld, or by making the logic a little bit more general.

Another approach is to make the third party app export the handheld data into a file calendar can read. This calendar can then be synced, and after that the third party app can send the data back to the handheld.

The third way is to create some ipc between calendar and the thirdparty app, and let that app do the logic. multisync has the logic build in, but what about windows and mac? And do we want to depend on multisync?

The downside with those approaches is that calendar needs to run while syncing. This is mainly because we also want syncing to work with non-ics calendars, like caldav.

[SaintDanBert wrote:] One might expect that calendar sync happens automatically. Any design that does not plan for mostly hands off sync seems flawed from the start. Once one requests a PDA sync with the workstation, I do not want any manual steps between my PDA and the final destination if they can be avoided.

A 3rd party app might sit between Sunbird/Lightning calendar and any PalmOS aware calendar. The app can sniff the PalmOS calendar on some schedule, grab the data of interest, and make it available to Sunbird/Lightning. It could also deliver a smart data-export using a TEE in the logic: write export file AND post to Mozilla calendar parts. --Saint.

SyncML

SyncML is a standard, but can we use it safely without any worries?

• AIUI (from reading other projects' mailing lists and web pages), it is implemented slightly differently across different clients. There is certification for interoperability of SyncML products; the list is long and at [[2]]. NB it can use HTTP as a transport, and a fair few of the clients on that list seem to support HTTP rather than OBEX; this should be relatively easy to integrate into Mozilla :-) - BenS

• All of the IPR patents for SyncML are presently only at application stage according to the Open Mobile Alliance's IPR Declaration page [[3]]. If they make it through, I'm guessing either the Moz Foundation or users would have to pay licensing on it. What's the policy on this kind of thing? - BenS

• SyncML seems a good idea to me in general. Perhaps a generic SyncML support would be great, so conduit sync could be left to the end user. (Using, for example, the Funambol project mentioned above.

Which implementations, libs or middleware could be used by Mozilla?

• There's a new project called TSync that claims to work via SyncML with various publich SyncML servers, with Thunderbird 1.x & 2.x. Inactive project, last release 2006. But might be possible to get the source code...? http://www.topologilinux.com/syncml/

• We (the people from the opensync project) are currently implementing a new syncml plugin. Since we didnt find a usable syncml library we started to write our own with support for general syncml, DS, Devinfo and later maybe DM. The supported transport mechanisms are http client, http server and obex client. It will be ready for testing at around middle of june. The license will be GPL.

• Looking at your (opensync) web-page, it seems that a possible approach would be to use opensync as the sync engine, taking it as a whole library - forget just SyncML, it would make more sense to simply use opensync as a whole. Are there any major obstacles to that?

Which other software supports it?

• The current version of Apples iSync (2.2 (shipped with Tiger)) is the first version of the software that is syncML compatible.

Which devices support it?

• Most Symbian OS Series 60 and above, late-model Sony Ericsson, and (AFIACT) all >=2004 Motorola phones have SyncML support in one form or another. Some only via OTA (Over The Air, ie: HTTP), and some like the new Motorola Linux-based phones (a768i, a780) support it via OTA, USB and infrared. One benefit of SyncML support, as BenS noted, would be a large base of device support.

Help Wanted

To make a plan that will work, we need to know what the situation is on Windows and on MacOSX. What apis are available to talk to handhelds? What logic is available?

If you know more about those platforms from a developers perspective, please note it.

For ActiveSync-API you can show at Microsoft�s Compact Framework

http://msdn.microsoft.com/netframework/programming/netcf/cffaq/default.aspx#ActiveSync

Sync with PalmOS

I don't know anything about MacOS and PocketPCs, but I can tell a bit about Windows and Palm-pda's. There is a sync-software called HotSync (as written above). You can write conduits for this software. Don't know why they are called conduits, they are basically plugins. They are loaded and run when you 'hotsync' a pda with your pc. This is done by pressing the hotsync-button on the pda's base-station. These conduits can be used to syncronize information between the pda and software on a pc. The sync-process is not startet by pc-software (i.e. Sunbird), but instead by the pda's owner pressing a button. I guess it shouldn't be a problem to write a conduit that syncronizes events and tasks between a pda and Sunbird, if there is an api to the calendar-database. So that you can access events and tasks without actually starting Sunbird.

Update I managed to access the Palm Todo database during a hotsync-operation using VisualBasic 6 (thats the only language I have). I will now try to figure out, how to access Sunbirds tasks. It would be great, if someone can give me some hints here.

SebastianKleine 01:24, 13 Jan 2006 (PST)

- I put a rough outline on your User Talk page Jminta

- Created a sourceforge project for developing a conduit to sync Palm Todo-list and tasks with Sunbird: PalmSunbirdSync. There is still some planning to make, before programming can begin. If anybody want's to join this project, feel free to do so. SebastianKleine 14:46, 17 Jan 2006 (PST)

- Does anybody know how to access a sunbird calendar from outside of Sunbird. SebboKleine

- As of Sunbird 0.3 - I believe - everything is stored in an SQLite database. Currently (0.7) it is an SQLite format 3 database; You will find it (on Windows) in %AppData%\Mozilla\Sunbird\Profiles\<profile> by default, and it is named storage.sdb. Keilaron 19:15, 8 January 2008 (PST)

I've made a few short notes about how a sync would work using a simple socket connection... this isn't exactly elegant, but at least it won't break when SQLite gets updated:

• • How to detect the presence and/or location of Thunderbird+Lightning?
    • Is there a way to find a running instance of TB?
    • Is there a way to determine if and where Thunderbird is installed?
      • Windows Registry HKLM/Software/Mozilla/ProductName
      • Some sort of system event???
    • Is there a way to do it without knowing the product name
      • Windows Registry is name-based, i.e. trunk builds are only found when querying for "Shredder"
  • How to launch Thunderbird correctly
    • Normal app launch using registry values?
    • Some sort of system event???
  • How to get Thunderbird to serve any data
    • Socked Server
    • XML Files
      • Would need a temporary file that can
      • Get quite large
      • May survive a crash and expose data
  • How to lock the communication to TB/conduit
    • Encryption
    • Lock to localhost
  • If a key is needed for safe communication, how to communicate the key?
    • Manually (copy/paste)
    • Direct injection into prefs file
    • Temporary insecure communication
  • How should TB serve data
    • As Server
    • As Client
  • How should a TB client know when to connect to the server?
  • How should a TB server know when to start?
  • How should data be communicated?
    • XML

So a possible implementation could work like this

• • User initiates Hotsync
  • Hotsync loads sync conduit
  • Sync conduit locates a running Instance of TB, somehow signaling the start of synchronization.
  • If no running instance is detected, SC launches TB, specifying a sync startup mode parameter
  • SC launches socket server
  • SC waits for TB-SyncAddon to request a synchronization.
  • SC pushes out a XML list of events/addresses/...
  • TB-SA replies with a list of events that either differed or were not part of the request
  • SC pushes the changes into the Palm Address Book/Calendar/...

Questions remaining

• • Is there a last-modified date on TB data? Because otherwise, we'll have a hard time resolving conflicts.
  • Is there something like a deleted item in TB data that shows were data existed, but was deleted by the user? Otherwise deleted items in TB would always be recreated by SC

Possible Windows/PocketPC workaround

OK, it isn't pretty, but as a temporary fix until a proper sync solution can be developed would it be possible to write something to sync between thunderbird/sunbird and [the desktop version of] Outlook (which is supplied with every PocketPC to my knowledge) - then ActiveSync will work normally in syncing between Outlook and PocketPC

Sync with PPC through ActiveSync is possible

As a Firefox and Thunderbird user who also has a Pocket PC, this project is very interesting to me. After some research with making programs that can sync using ActiveSync, I have discovered that it is possible. (Un)Fortunately to make this work, a version of Thunderbird would have to be created to run on the PPC. There may be some sort of hack that can be done just by writing DLLs for Thunderbird and possibly having to rewrite the ones for Outlook on the PPC. This page should give you the information needed to accomplish this. As a programmer I would be interesting in helping but am not sure how much time I would have to devote to this project. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnce21/html/ActiveSync.asp?frame=true

It might not be so complex - That MSDN article suggests you need something on the PPC and something on the PC that can talk to each other via ActiveSync - so all you'd need is a Thunderbird plugin on the PC that can access the user's TB contacts/calendar, and something on the PPC that can access the PPC's contacts/calendar, then you should be a able to sync one with the other --benjymous

As far as I know it is not so complex at all. The PPC's data is usually stored in a database volume (e.g. pim.vol) and PPC uses PocketOutlook to access these databeses (Calendar, Contacts, etc.). ActiveSync provides a COM library called "outstore.dll" on PC side (usually calles Desktop Provider Service) which has all functions to read "Appointment, Contacts, Inbox, Task" databases. Device Providers (that means ActiveSync plug-ins in PPC) are only needed, if you want to sync yor own PPC application with an application on your PC (e.g. Thunderbird on PC with PocketThunderbird on PPC). [4] I would suggest to write an ActivSync plug-in that syncs automatically with Thunderbird when the user's device is connected, like ActiveSync allready does with Outlook. On Thunderbird side one could write a plug-in which adds some additional buttons into Thunderbird, so that the user can manually start the syncing activity. frane

Sync with Mac OS X

Mac OS X has a built-in synchronization framework called the .Mac SDK. The framework is available for download from the Apple Developer Connection (ADC) at http://developer.apple.com, but requires free registration. The latest released version of the SDK is 1.2 (10-Oct-2005). A "developer preview" of the SDK v 2.0 is also available and was released on 17-Nov-2005. The Apple .Mac service is not free however, which cuts against the grain of free software like SunBird or Mozilla in general.

On the other hand, the iSync framework is the only native synchronization framework on the Mac. Furthermore, it supports the synchronization of Calendar, Task, Notes and Contacts with any iSync aware device or program. Most popular cell phones allow synchronization with iSync (I have personally used it successfully in combination with Nokia N95-2 and Samsung D900i). There are also other programs that are also iSync compatible such as Microsoft Entourage (thus a migration to Thunderbird+Calendar would actually be feasible) and Yahoo! Messenger. Using Yahoo! Messenger you can have a free address-book synchronization feature portable on multiple computers. MobileMe/.Mac is a feature of iSync but not the most important one, iSync's strengths lie in its wide industry acceptance on the Macintosh platform.

The SDK is made up of the service and a private framework. The private framework would need to be built into the project and is not open source. As such, how the Mozilla license and the license covering the .Mac SDK would interact is not clear. It may be necessary to wrap the SDK with a plug-in so that it is not part of the application.

The next version of Mac OS X Server (v10.5) will include a CalDAV server, which will allow any calendar application that supports the iCalendar format to access and share calendars. A brief blurb about iCal Server is available from Apple at http://www.apple.com/server/macosx/leopard/icalserver.html. You can find additional information about CalDAV at http://www.caldav.org/.