User:Anaaktgeboren/SearchEngineSync

Goals

• • Sync search engine prefs on Desktop
  • Format to be interoperable with firefox os, mobile firefox
  • Nongoals:
    • implementation on firefox os
    • implementation on mobile firefox

Requirements Gathering (deadline for input: july 26)

• Stakeholders
  • Desktop: Gavin
  • Desktop UX: madhava, boriss, ibarlow on mobile, poked twice
  • Mobile: bnicholson, mfinkle
  • B@G: N/A, Ben Francis, b2g has no plugins and few preferences at this time
  • Privacy: Tom, Sid
  • Security: David Chan
  • Product: Asa
  • Services Integration: Gps
  • Community: dev-planning, dev-apps-firefox lists
  • Legal: bug 778823,

• Current Requirements
  • does not sync hijacked queries
    • origin: Asa, browser cannot currently detect this, requires a new browser feature
  • syncs code & not a server call, mitigate risk
    • origin sstamm, reducing the addon sync threat
  • search order preserved
  • 3rd party plugins appear

Current Suggestions/Notes

• input
  • want to avoid syncing app-shipped search plugins, origin: gavin
  • dont sync XML files, sync the JSON cache that's the result of parse origin: gavin
  • "Search service changes will be required for this, almost certainly. Exposing the engine description as JSON and adding the ability to distinguish app-shipped plugins via the nsIBrowserSearchService API seem like good first steps to me" - gavin
  • "toolkit/components/search/nsSearchService.js. - gps
  • See also the XML files in the 'searchplugins' directory of the app/profile" - gps
  • mobile uses same nsISearchService as desktop -mfinkle
  • should it be applicationID based, the way addons?
  • should we limit to a blessed type of plugin to address hijacking?
  • what data represents a search plugin? and where are the keywords assoicated with it?

Design & Design Review

• Data Model, core of sync
  • ordering of entries
  • carrying code non-defaulted plugins
• Desktop design
  • Major dependencies
  • Product page can be made at this point
• Reviews? privacy security
• UI changes?

Implementation

• • not started

Release

• • targeted to land on Nightly during Q3 (not a promise, subject to scope change)

Reference

• this implementation bug: 444284
• addon sync implemetnation bug: 534956
• https://developer.mozilla.org/en/Adding_search_engines_from_web_pages
• http://www.opensearch.org/Specifications/OpenSearch/1.1#OpenSearch_description_document
• https://developer.mozilla.org/en/Creating_OpenSearch_plugins_for_Firefox
• http://docs.services.mozilla.com/sync/index.html

Draft Plan

• currently full of holes

Spec

• The OpenSearch plugins, ordering of search engines, keywords for search engines (both custom & default), and hidden flags (default engines only afaik) will be sync'ed across platforms (between mobile & desktop)
  • default engine OpenSearch objects will not be sync'ed, as these are hardcoded and already present on the machines. However their entries in the search db will synced as this contains the user generated ordering & keywords.
  • on the fence about whether to sync Sherlock objects in 1.0, or iterate on that later if anyone screams
  • on the fence about whether to sync extension shipped plugins, especially given Asa's concern about hijacking & the avg funny business, though searchservice will consider an extension shipped plugin as a default in _isDefault()
  • the currently selected engine will not be synced, as that is a local setting and probably does not make sense to sync
  • read-only engines cannot be serialized, and therefore probably should not be sync'ed (I believe this is just the default ones)
• When a user changes or removes a keyword or custom engine, the removal will be synced
• When a user removes a default engine, the removal will be synced (but note that the browser does not actually remove those, they are just hidden)
• The results of "Restore Defaults" will be synced (restores all deleted defaults, wipes db to default single entry (ketwords, ordering). This will not be treated as a special operation by sync
• Until the browser implements detecting hijacked searches, this engine will be preffed off by default for existing and new users
• Reconciliation: Merge by Default
  • When users choose model from settings, overrides this default?
• The custom search plugins will be synced as json
• Data Model
  • each client will provide one record containing
  • CustomSearch array containing the json blobs of all custom search engines
  • areHidden array containing names of all the default engines that are currently hidden
  • SearchRows array containing all the rows of search db [engineid, name, value]
    • note that value may be an intger or a string and that engineid is not unique. there may be several columns with hte same engineid

Implementation schedule

• largely made up, as the repositories re-write is a huge wild card
• feedback due by 26-jul
• spec finalized by 3-Aug
• Implement store, record for json + tests by 10-aug
• Implement store, record for db(ordering, keywords) + tests by 15-aug
• Implement tracker + xpc tracker tests by 24-aug
• Implement engine + xpc tests by 7-sep
  • most likely to get snagged here
• Implement tps tests (if possible) and any remaining xpc tests by sep-14
• Debugging (unknown)
  • 'cause there's going to a landmine somewhere

Test Plan

TPS Tests

• Important Notes
  • tps does not support the native mobile at this time. timeframe unknown for support
  • dependency - adding a new engine will require work on tps. timeframe unknown, but likely to be reasonable
    • tests cannot be written ahead of those changes to tps though
• Tests
  • only reordering syncs between profile 1 & profile 2
  • only keyword addition, change, and removal sync between profile 1 & 2
  • adding and removing a search engine plugin syncs between profile 1 & 2
  • add plugin, add keyword, reorder, sync between two desktops, remove & sync again
  • add plugin, add keyword, reorder, sync between mobile (if mobile client exists) & desktop, remove & sync again

XPC Shell Tests

• make sure there is heavy testing about reordering with hidden & default engines, as some of the calls may have unexpected side effects (and some functions consider hidden in counts and others do not, same deal with what counts as default)
• impacted by repositories re-write?

Dependencies

• TPS supporting desktop to mobile operations (jgriffen?)
  • iirc, engine additions require changes to tps each time
• Desktop client supporting hijacked search detection (gavin?)
• Someone implementing the corresponding mobile engine (?)
• repositories rewrite (gps)
• Do we know how many custom search engines the average user has? (asa?)
• What happens when one of the default engines are removed? I do not see it reflected in the db nor json

Scratch and Notes

• Insights
  • user created keywords are called 'alias'es in code, stored in search-metadata.json
  • /profile/search.json
    • "directories" will have two entries, the default one ("/Applications/Firefox.app/Contents/MacOS/searchplugins") & "/Users/mozilla/Documents/inbox-desktop/test_profiles/searchplugins" in my case) The first has no usefully syncable data, though it does have a lastModifiedTime (this is the lastModifiedTime of the directory, used for cache invalidation --Gavin)
  • /profile/search.sqlite(DO NOT SYNC) (this is no longer used after bug 699856, replaced by search-metadata.json --Gavin)
    • engineid column identifies engine xml file, and notably where it came fromt[app] or [profile]
      • which would map to the directories list in the search.json?
    • value column identifies order of appearance in dropdown or is a string if a keyword has been added for that engine
    • name column is either "order" or "used"
    • the same 'logical' entry may have more than one entry in the db with subtly different values
  • pref browser.search.selectedEngine represents the currently selected entry, status "user set" is useful
  • pref browser.search.useDBForOrder i bet that represents ordering (this gets set to true once the user has customized the search engine order, to indicate that we should use the positioning information in search-metadata.json rather than the one in default prefs (browser.search.order.*) --Gavin)
  • pref browser.search.order.1-3 are hard coded defaults. do not sync
  • JAR loading code is for mobile, for their default plugin handling

• Poking Pointers
  • changes frequently dont show up immediate (but get written on close)
  • getting a 3rd party search thingy
    • right click on a website search bar (ie youtube) and /profile/searchplugins will be created with an OpenSearch .xml
    • searchplugins doesn't exist until you create a non-default one
  • SQLite table: install addon, go to tools -> SQLiteManager -> select db, go
  • default engines have a hidden value, which afaic, is never applied to custom engines?

• current problems
  • how do i gracefully slurp the json content?
  • can i dump the json content back into the data store *without* the xml?
  • design pondering
    • Should this be like tabs and a one-record /client
    • or breakdown the table's multiple entry, grab the json and make it one record per 3rd party (but what about ordering of defaults?)
    • treat the table & the json as entirely separate things, essentially two record types? tis not really the sync way and I do not even know if it is feasible
    • do i extract the values of the json blob and rebuild it on the client, or send the thing over the wire wholesale?