User:Anaaktgeboren/SearchEngineSync

Goals

Sync search engine prefs on Desktop
- addition of an engine
- removal of an engine
- reordering of engines
- keywords/aliases for a particular engine
- restore defaults (potentially not a special case for implementation, but could be considered a feature by users)
Format to be interoperable with firefox os, mobile firefox, or other future clients

Nongoals:
- implementation on firefox os
- implementation on mobile firefox

Design

Requirements Gathering

(deadline for input: july 26)
Stakeholders
- Desktop: Gavin
- Desktop UX: madhava
- Mobile: bnicholson, mfinkle
- B@G: N/A, Ben Francis, b2g has no plugins and few preferences at this time
- Privacy: Tom, Sid
- Security: David Chan
- Product: Asa
- Services Integration: Gps
- Community: dev-planning, dev-apps-firefox lists
- Legal: bug 778823

Current Requirements

does not sync hijacked queries
- origin: Asa, browser cannot currently detect this, requires a new browser feature
syncs code & not a server call, mitigate risk
- origin sstamm, reducing the addon sync threat
syncs keywords
search order preserved
3rd party plugins appear
honors enable/disable
- desktop does not match mobile
does not assume matching default lists on different clients (which is to say, can't assume google et al are installed everywhere) and syncing of default engine code is likely
syncs which engines support the search suggest api

Current Suggestions/Notes

input
- want to avoid syncing app-shipped search plugins, origin: gavin
- dont sync XML files, sync the JSON cache that's the result of parse origin: gavin
- mobile uses same nsISearchService as desktop -mfinkle
- should it be applicationID based, the way addons?
- should we limit to a blessed type of plugin to address hijacking?
- impact of Australis?

Design Reviews if needed

privacy security?
UI changes?

Implementation

not started
desktop will need code changes to address missing enable/disable state for custom search engines
desktop will need nsSearchService changes to handle a consolidated data format

Release

- targeted to land on Nightly during Q3 (not a promise, subject to scope change)

Reference

this implementation bug: 444284
addon sync implemetnation bug: 534956
https://developer.mozilla.org/en/Adding_search_engines_from_web_pages
http://www.opensearch.org/Specifications/OpenSearch/1.1#OpenSearch_description_document
https://developer.mozilla.org/en/Creating_OpenSearch_plugins_for_Firefox
http://docs.services.mozilla.com/sync/index.html

Draft Plan

currently full of holes

Spec

proposed data model for a plugin record (now out of date as I try to work in the wrinkles exposed by mobile/ux)

{
  id // string id
  name // similiar but different from the description
  alias // optional, if user has set keywords
  order // optional if user has reordered engines
  hidden // optional, if user has 'deleted' a default engine, or an extensions had hidden a custom one.
  description // short description string
  searchForm // exists
  iconurl // the icon. found on defaults as well. 
  urls  //a nice fat object for included ones, less so for custom probably
  url // optional, not found on default
  installLocation // optional. if not present, this plugin is default [app]
  type //optional
  queryCharset //optional
  readOnly // optional in the event that we sync readonly plugins in the future
  locale // currently all records should have the same value, but it is reasonable for in the future 
    // for plugins to have different settings for this 

}

Not Syncing
- directory - machine specific
- lastModifiedTime - machine & directory specific. used to invalidate cache
- filepath - path to the xml, which wont exist since we arent syncing the xml and would be specific to a machine (where fx is installed, where profile is installed, etc)

The OpenSearch plugins, ordering of search engines, keywords for search engines (both custom & default), and hidden flags (default engines only afaik) will be sync'ed across platforms (though the mobile client(s) will be written later)
- parts of default engines will be synced (such as ordering & keywords)
- the currently selected engine will not be synced, as that is a local setting and probably does not make sense to sync
- read-only engines cannot be serialized, and therefore probably should not be sync'ed (I believe this is just the default ones)
When a user changes or removes a keyword or custom engine, the removal will be synced
When a user removes a default engine, the removal will be synced (but note that the browser does not actually remove those, it just hides it)
The results of "Restore Defaults" will be synced (restores all deleted defaults, wipes db to default single entry (ketwords, ordering). This will not be treated as a special operation by sync
Until the browser implements detecting hijacked searches, this engine will be preffed off by default for existing and new users
Reconciliation: Merge by Default
- When users choose model from settings, overrides this default?
- what if the default ordering is different on two clients?
Defaults on other Systems
- when synced to mobile, default engines that are not default on mobile (there are 4 to desktop's 7+) will be disabled (visible but greyed out). Users will have the option to enable those for their system.
  - implies that keywords wont work for disabled engines, but that would be a client decision
- when synced to mobile, user generated/added engines will be automatically enabled

Implementation schedule

largely made up, as the repositories re-write is a huge wild card
feedback due by 26-jul
spec finalized by 3-Aug
Implement store, record for json + tests by 10-aug
Implement store, record for db(ordering, keywords) + tests by 15-aug
Implement tracker + xpc tracker tests by 24-aug
Implement engine + xpc tests by 7-sep
- most likely to get snagged here
Implement tps tests (if possible, see dependencies) and any remaining xpc tests by sep-14
Debugging (unknown)
- 'cause there's going to a landmine somewhere

Dependencies

TPS supporting desktop to mobile operations (jgriffin's team)
- TPS new data tupe addition (jgriffin's team)
Desktop client supporting hijacked search detection (gavin?)
- so engine can be enabled by default for new or existing users
Someone implementing the corresponding mobile engine (?)
repositories rewrite (gps)

Test Plan

TPS Tests

Important Notes
- tps does not support the native mobile at this time. timeframe unknown for support
- dependency - adding a new engine will require work on tps. timeframe unknown, but likely to be reasonable
  - tests cannot really be written ahead of those changes to tps
Tests
- only reordering syncs between profile 1 & profile 2
- only keyword addition, change, and removal sync between profile 1 & 2
- adding and removing a search engine plugin syncs between profile 1 & 2
- add plugin, add keyword, reorder, sync between two desktops, remove & sync again
- add plugin, add keyword, reorder, sync between mobile (if mobile client exists) & desktop, remove & sync again

XPC Shell Tests

make sure there is heavy testing about reordering with hidden & default engines, as some of the calls may have unexpected side effects (and some functions consider hidden in counts and others do not, same deal with what counts as default)
impacted by repositories re-write?

Scratch and Notes

Insights

what should happen when default lists dont match
- defaults should be synced because they may not be present on other systems and users can expect to see them (though the mobile client may grey them out)
user created keywords are called 'alias'es in code, stored in search-metadata.json
search-metadata.json
- may not exist if nothigns been changed
- contains per plugin: order placement, hidden, and keywords
/profile/search.json
- "directories" will have two entries, the default one ("/Applications/Firefox.app/Contents/MacOS/searchplugins") & "/Users/mozilla/Documents/inbox-desktop/test_profiles/searchplugins" in my case) The first has no usefully syncable data, though it does have a lastModifiedTime (this is the lastModifiedTime of the directory, used for cache invalidation --Gavin)
JAR loading code is for mobile, for their default plugin handling
Australis is not making the search engines or their data go away
"Search service changes will be required for this, almost certainly. Exposing the engine description as JSON and adding the ability to distinguish app-shipped plugins via the nsIBrowserSearchService API seem like good first steps to me" - gavin
- Mobile will support search keywords & search suggest in the nearish future.
/profile/search.sqlite(DO NOT SYNC) (this is no longer used after bug 699856, replaced by search-metadata.json --Gavin)
pref browser.search.selectedEngine represents the currently selected entry, status "user set" is useful
pref browser.search.useDBForOrder i bet that represents ordering (this gets set to true once the user has customized the search engine order, to indicate that we should use the positioning information in search-metadata.json rather than the one in default prefs (browser.search.order.*) --Gavin)
pref browser.search.order.1-3 are hard coded defaults. do not sync

current problems

search suggest has global preferences associated with it. to sync or not to sync?
- and if so, should it go in the preferences engine, or in this one? I'm inclined to think it matters even if engines are not synced

mobile has state that desktop does not (enable/disable)
mobile will want more state about usage than desktop does
how do i gracefully slurp the json content?
design pondering

What do these data formats look like anyway?

search-metadata.json. reordering + keywords. Note this file can be empty

{
    "[app]/google.xml":
    {
        "hidden":true,
        "alias":null,
        "order":1
    },
    "[profile]/youtube-video-search.xml":
    {
        "order":2,
        "alias":"anaaktge_test_keyword"
    },
    "[app]/bing.xml":
    {
        "order":3
    },
    "[app]/amazondotcom.xml":
    {
        "order":4,
        "alias":"anaaktge_test_keyword_default"
    },
    "[app]/eBay.xml":
    {
        "order":5
    },
    "[app]/twitter.xml":
    {
        "order":6
    },
    "[app]/wikipedia.xml":
    {
        "order":7
    },
    "[app]/yahoo.xml":
    {
        "order":8
    }
}

search.json, one default plugin, one custom (youtube) plugin

{
    "version":7,
    "buildID":"20120730133507",
    "locale":"en-US",
    "directories":
    {
        "/Users/mozilla/moz/hg/services-central/./obj-x86_64-apple-darwin11.4.0/dist/NightlyDebug.app/Contents/MacOS/searchplugins":
        {
            "lastModifiedTime":1343683762000,
            "engines":
            [
                {
                    "_id":"[app]/amazondotcom.xml",
                    "_name":"Amazon.com",
                    "_hidden":false,
                    "description":"Amazon.com Search",
                    "__searchForm":"http://www.amazon.com/",
                    "_iconURL":"data:image/x-icon;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAAK/INwWK6QAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAHgSURBVHjalFM9TNtQEP4cB7PwM1RITUXIgsRaYEEVEyKZwhiyZAQyd0BhpFOlIjoBqhjSqVQMoVMLLAjEwECCQJkSkBqJYDOAFOMKFSf28d7DTUxiUDnp/Pzeu/vuu7t3ICKF6SLTMv2/lB0fRWKfjwDm4JJisYh0Oo3fpZLYT0SjSCQS8JAFMADNDZ3NZsnf1taiqVTKi4nGASruk5lkkmTmMB6JUKFQqO+DfX1eABWeQoVR6f7HSdM0obqu48Yw8G1tDT82NsRd1TSbU9BbGPCog8PDj+jLzurFoAVgMh4XxoNDQ6SqKi0tL9eBvAB8zZwymYxYY7EYAoEA8vm82BNTg6XUIs0MeGTZoR1mhXSnwNl4pmAbjU7mcjkKhkL1ynMnntZ4OEw3VyrV8utk7s5TdW++0QXz+1i3P7IK36t+PCfVn1OQOoOA0gXr5DPak+cPXbBK+/T3S69AtY3LJ98vZ1or/iLr+pTuvr59/A6s003UdqZFJF/PCKQ3o5CUznoBST2AfbEF/9iqYEDaIfwj73VJPEfgNTe0tWNYR0uwy9uOW0OkrgHI7z5ADo2C7v48nLV3XHKAT+x/1m1sX58xsBxg8rZJrDYD8DHHp4aJj/MK09sXjPOt46PcCzAACXY8/u34wN0AAAAASUVORK5CYII=",
                    "_urls":
                    [
                        {
                        "template":"http://www.amazon.com/exec/obidos/external-search/",
                        "rels":[],
                        "params":
                        [
                            {
                                "name":"field-keywords",
                                "value":"{searchTerms}"
                            },
                            {
                                "name":"mode",
                                "value":"blended"
                            },
                            {
                                "name":"tag",
                                "value":"mozilla-20"
                            },
                            {
                                "name":"sourceid",
                                "value":"Mozilla-search"
                            }
                        ]
                        }
                    ],
                    "filePath":"/Users/mozilla/moz/hg/services-central/./obj-x86_64-apple-darwin11.4.0/dist/NightlyDebug.app/Contents/MacOS/searchplugins/amazondotcom.xml"
                }
            ]
        }
        "/Users/mozilla/Documents/inbox-desktop/test_profiles/search_engine_sync_0/searchplugins":
        {
            "lastModifiedTime":1343685412000,
            "engines":
            [
                {
                    "_id":"[profile]/youtube-video-search.xml",
                    "_name":"YouTube Video Search",
                    "_hidden":false,
                    "description":"Search for videos on YouTube",
                    "__searchForm":"http://www.youtube.com",
                    "_iconURL":"data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAQAQAABMLAAATCwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgNkQkIDZGiCA2RzAgNkcwIDZH/CA2R/wgNkf8IDZH/CA2R/wgNkf8IDZH/CA2R2AgNkcwIDZHMCA2RhAgNkQYIDpWHCA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpWHCQ6ZzAkOmf8JDpn/CQ6Z/wkOmf8JDpb/BQhc/wgMgf8JDpn/CQ6Z/wkOmf8JDpn/CQ6Z/wkOmf8JDpn/CQ6ZzAkOnuoJDp7/CQ6e/wkOnv8JDp7/Exed/8jIy/9RU4j/Bwp0/wkOm/8JDp7/CQ6e/wkOnv8JDp7/CQ6e/wkOnuoJD6T8CQ+k/wkPpP8JD6T/CQ+k/xUbo//V1dX/1dXV/4yNrP8QFG//CA6Y/wkPpP8JD6T/CQ+k/wkPpP8JD6T8CQ+q/wkPqv8JD6r/CQ+q/wkPqv8WG6n/3d3d/93d3f/d3d3/v7/M/y0wjv8JD6r/CQ+q/wkPqv8JD6r/CQ+q/woQr/8KEK//ChCv/woQr/8KEK//Fx2v/+fn5//n5+f/5+fn/+jo6P+YmtP/ChCv/woQr/8KEK//ChCv/woQr/8KELX8ChC1/woQtf8KELX/ChC1/xgdtf/x8fH/8fHx//Ly8v+bndv/Ehi3/woQtf8KELX/ChC1/woQtf8KELX8ChG76goRu/8KEbv/ChG7/woRu/8YH77/+fn5/+/v9/9fY9H/ChG7/woRu/8KEbv/ChG7/woRu/8KEbv/ChG76goRwMwKEcD/ChHA/woRwP8KEcD/EBfB/6Ol5/8tM8n/ChHA/woRwP8KEcD/ChHA/woRwP8KEcD/ChHA/woRwMwLEcSHCxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcSHCxLICQsSyKULEsjMCxLI+QsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI0gsSyMwLEsiiAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP//AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAD//wAA//8AAA==",
                    "_urls":
                    [
                        {
                            "template":"http://www.youtube.com/results?search_query={searchTerms}&page={startPage?}&utm_source=opensearch",
                            "rels":[],
                            "params":[]
                        }
                    ],
                    "filePath":"/Users/mozilla/Documents/inbox-desktop/test_profiles/search_engine_sync_0/searchplugins/youtube-video-search.xml",
                    "_url":"http://www.youtube.com/opensearch?locale=en_US",
                    "_installLocation":2,
                    "type":3,
                    "queryCharset":"UTF-8",
                    "_readOnly":false
                }
            ]
        }
    }
}

profile/searchplugins/ .xml

<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/" xmlns:os="http://a9.com/-/spec/opensearch/1.1/">
<os:ShortName>YouTube Video Search</os:ShortName>
<os:Description>Search for videos on YouTube</os:Description>
<os:InputEncoding>UTF-8</os:InputEncoding>
<os:Image width="16" height="16">data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAQAQAABMLAAATCwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgNkQkIDZGiCA2RzAgNkcwIDZH/CA2R/wgNkf8IDZH/CA2R/wgNkf8IDZH/CA2R2AgNkcwIDZHMCA2RhAgNkQYIDpWHCA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpWHCQ6ZzAkOmf8JDpn/CQ6Z/wkOmf8JDpb/BQhc/wgMgf8JDpn/CQ6Z/wkOmf8JDpn/CQ6Z/wkOmf8JDpn/CQ6ZzAkOnuoJDp7/CQ6e/wkOnv8JDp7/Exed/8jIy/9RU4j/Bwp0/wkOm/8JDp7/CQ6e/wkOnv8JDp7/CQ6e/wkOnuoJD6T8CQ+k/wkPpP8JD6T/CQ+k/xUbo//V1dX/1dXV/4yNrP8QFG//CA6Y/wkPpP8JD6T/CQ+k/wkPpP8JD6T8CQ+q/wkPqv8JD6r/CQ+q/wkPqv8WG6n/3d3d/93d3f/d3d3/v7/M/y0wjv8JD6r/CQ+q/wkPqv8JD6r/CQ+q/woQr/8KEK//ChCv/woQr/8KEK//Fx2v/+fn5//n5+f/5+fn/+jo6P+YmtP/ChCv/woQr/8KEK//ChCv/woQr/8KELX8ChC1/woQtf8KELX/ChC1/xgdtf/x8fH/8fHx//Ly8v+bndv/Ehi3/woQtf8KELX/ChC1/woQtf8KELX8ChG76goRu/8KEbv/ChG7/woRu/8YH77/+fn5/+/v9/9fY9H/ChG7/woRu/8KEbv/ChG7/woRu/8KEbv/ChG76goRwMwKEcD/ChHA/woRwP8KEcD/EBfB/6Ol5/8tM8n/ChHA/woRwP8KEcD/ChHA/woRwP8KEcD/ChHA/woRwMwLEcSHCxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcSHCxLICQsSyKULEsjMCxLI+QsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI0gsSyMwLEsiiAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP//AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAD//wAA//8AAA==</os:Image>
<os:Url type="text/html" method="GET" template="http://www.youtube.com/results?search_query={searchTerms}&page={startPage?}&utm_source=opensearch">
</os:Url>
</SearchPlugin>

google search xml

<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/">
<ShortName>Google</ShortName>
<Description>Google Search</Description>
<InputEncoding>UTF-8</InputEncoding>
<Image width="16" height="16">data:image/png;base64,AAABAAEAEBAAAAEAGABoAwAAFgAAACgAAAAQAAAAIAAAAAEAGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADs9Pt8xetPtu9FsfFNtu%2BTzvb2%2B%2Fne4dFJeBw0egA%2FfAJAfAA8ewBBegAAAAD%2B%2FPtft98Mp%2BwWsfAVsvEbs%2FQeqvF8xO7%2F%2F%2F63yqkxdgM7gwE%2FggM%2BfQA%2BegBDeQDe7PIbotgQufcMufEPtfIPsvAbs%2FQvq%2Bfz%2Bf%2F%2B%2B%2FZKhR05hgBBhQI8hgBAgAI9ewD0%2B%2Fg3pswAtO8Cxf4Kw%2FsJvvYAqupKsNv%2B%2Fv7%2F%2FP5VkSU0iQA7jQA9hgBDgQU%2BfQH%2F%2Ff%2FQ6fM4sM4KsN8AteMCruIqqdbZ7PH8%2Fv%2Fg6Nc%2Fhg05kAA8jAM9iQI%2BhQA%2BgQDQu6b97uv%2F%2F%2F7V8Pqw3eiWz97q8%2Ff%2F%2F%2F%2F7%2FPptpkkqjQE4kwA7kAA5iwI8iAA8hQCOSSKdXjiyflbAkG7u2s%2F%2B%2F%2F39%2F%2F7r8utrqEYtjQE8lgA7kwA7kwA9jwA9igA9hACiWSekVRyeSgiYSBHx6N%2F%2B%2Fv7k7OFRmiYtlAA5lwI7lwI4lAA7kgI9jwE9iwI4iQCoVhWcTxCmb0K%2BooT8%2Fv%2F7%2F%2F%2FJ2r8fdwI1mwA3mQA3mgA8lAE8lAE4jwA9iwE%2BhwGfXifWvqz%2B%2Ff%2F58u%2Fev6Dt4tr%2B%2F%2F2ZuIUsggA7mgM6mAM3lgA5lgA6kQE%2FkwBChwHt4dv%2F%2F%2F728ei1bCi7VAC5XQ7kz7n%2F%2F%2F6bsZkgcB03lQA9lgM7kwA2iQktZToPK4r9%2F%2F%2F9%2F%2F%2FSqYK5UwDKZAS9WALIkFn%2B%2F%2F3%2F%2BP8oKccGGcIRJrERILYFEMwAAuEAAdX%2F%2Ff7%2F%2FP%2B%2BfDvGXQLIZgLEWgLOjlf7%2F%2F%2F%2F%2F%2F9QU90EAPQAAf8DAP0AAfMAAOUDAtr%2F%2F%2F%2F7%2B%2Fu2bCTIYwDPZgDBWQDSr4P%2F%2Fv%2F%2F%2FP5GRuABAPkAA%2FwBAfkDAPAAAesAAN%2F%2F%2B%2Fz%2F%2F%2F64g1C5VwDMYwK8Yg7y5tz8%2Fv%2FV1PYKDOcAAP0DAf4AAf0AAfYEAOwAAuAAAAD%2F%2FPvi28ymXyChTATRrIb8%2F%2F3v8fk6P8MAAdUCAvoAAP0CAP0AAfYAAO4AAACAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAQAA</Image>
<Url type="application/x-suggestions+json" method="GET" template="https://www.google.com/complete/search?client=firefox&q={searchTerms}"/>
<Url type="text/html" method="GET" template="https://www.google.com/search">
  <Param name="q" value="{searchTerms}"/><Param name="ie" value="utf-8"/><Param name="oe" value="utf-8"/><Param name="aq" value="t"/><Param name="rls" value="{moz:distributionID}:{moz:locale}:{moz:official}"/>
  <MozParam name="client" condition="defaultEngine" trueValue="firefox-a" falseValue="firefox"/>
</Url>
<!-- Keyword search URL is the same as the default, but with an additional parameter -->
<Url type="application/x-moz-keywordsearch" method="GET" template="https://www.google.com/search">
  <Param name="q" value="{searchTerms}"/><Param name="ie" value="utf-8"/><Param name="oe" value="utf-8"/><Param name="aq" value="t"/><Param name="rls" value="{moz:distributionID}:{moz:locale}:{moz:official}"/>
  <MozParam name="client" condition="defaultEngine" trueValue="firefox-a" falseValue="firefox"/>
  <Param name="channel" value="fflb"/>
</Url>
<!-- Context/Right-click search URL is the same as the default, but with an additional parameter -->
<Url type="application/x-moz-contextsearch" method="GET" template="https://www.google.com/search">
  <Param name="q" value="{searchTerms}"/><Param name="ie" value="utf-8"/><Param name="oe" value="utf-8"/><Param name="aq" value="t"/><Param name="rls" value="{moz:distributionID}:{moz:locale}:{moz:official}"/>
  <MozParam name="client" condition="defaultEngine" trueValue="firefox-a" falseValue="firefox"/>
  <Param name="channel" value="rcs"/>
</Url>
<SearchForm>https://www.google.com/</SearchForm>
</SearchPlugin>

amazon xml

<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/">
<ShortName>Amazon.com</ShortName>
<InputEncoding>ISO-8859-1</InputEncoding>
<Image width="16" height="16">data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAA2ZJREFUeNrEV01IFGEYfndXNzq0QdalJOoUbQiBUuopD2k/GFSG/Vxyq0OHbCUjwn7AEooUzUNRkdkhS1o9JBVaUDctKqhoRQi0WjtlxG6XVtfpfb6Z2caZ2ZnZXZdeeJxvx+973+f9+775XJIkkUaWMcoYG5TxfMpvxlvGMGM8+RYEFOxn/JJyLzOMiwwv7LqUCNQwHjpxIxKJ0Gg4LABZUVhIfr+f1jLSlNuMwyCwEHoZS6xmvxoZoc6Oq+JpJoVMpOncWdpcWZkOiRIQ2MKDp1az+kIhOtV40pHG6zdvpEOixc1/VtqFvKX5gmOX0pkL2yDgsZpxt+sORaPROe8Q6ncf3tPAk8eG3Ks14lA8brsZYZ2yukBAwOfzCeMHA3WGNXrCVpLnJKeqRyhAfX5RfNmILQF4urG0VIzxBBkU5aQI9agIeU4JqNLZ0UH9ob6sDWZE4MDefSn7P1txO/FcbxypQG18nhinew/u5zYC3dyG+qLL1qjjCKDg9C21q2a3oe9zRsCsn2PR2JzfKEy9PB96Nj8E0IJm54IaGZwPZsWJOU4jY1kD2OlAQhsJjKu3bSe7yPUzifpgMPsuOBY8brtR1evmyFt0IL0IzH4fJtcCH7kK1hn2/hh71G1yKKEdz/DBBIOTkUkRemzVl1uvCGKzE4OMIaFbiv1LSX51L7mXl4kxvgeO8vMaJk0PHiHvjl4DCTWs2lMOX0L6cwD/Bxlp6hNNv2gUT9MjsLiB8koaMOxJRgCMgPijWsqvaCP3qqqUZ4JVzUASYyFyL/WTqyggPxWH4qGtBlJzagCGIYjEzJv2zHe38vOUt6mNPGv20OyPMMV7yuVwL5IjBlLmRej1UX7VLdmLt+2CMVKTiSDn0wO1NPPyBOtdLL+MyzWkTbGhC5AGsBdKkEtWAjglgjUwCq/FGjhV0ZosdOhXI5FyH0DoIIK9slB0CS8UCgr8AlpvpamwqHpttcO4WtSqA57ioKENE05IqIYSY46uD4Is0qmGG2s9RYeS7adI3Paz3K6lTMmzIdFmXp/d1Gb1YvLF7i4IzxMfu1ITYWOe1VWix7U5tlKpXkzwYycONqfVLcW+cU7lQ0jePf360DqRS4zT/+Ny+ofRxPBoL6fa6zmu5uvtbkwZyE/lev6a8VV9+VeAAQADg36zc4GRNAAAAABJRU5ErkJggg==</Image>
<Url type="text/html" method="GET" template="http://www.amazon.com/exec/obidos/external-search/">
  <Param name="field-keywords" value="{searchTerms}"/>
  <Param name="mode" value="blended"/>
  <Param name="tag" value="mozilla-20"/>
  <Param name="sourceid" value="Mozilla-search"/>
</Url>
<SearchForm>http://www.amazon.com/</SearchForm>
</SearchPlugin>

Now let's turn those xml documents into JSON (records must contain JSONable code) (google, amz, custom(youtube))

{
  "SearchPlugin": {
    "-xmlns": "http://www.mozilla.org/2006/browser/search/",
    "ShortName": "Google",
    "Description": "Google Search",
    "InputEncoding": "UTF-8",
    "Image": {
      "-width": "16",
      "-height": "16",
      "#text": "data:image/png;base64,AAABAAEAEBAAAAEAGABoAwAAFgAAACgAAAAQAAAAIAAAAAEAGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADs9Pt8xetPtu9FsfFNtu%2BTzvb2%2B%2Fne4dFJeBw0egA%2FfAJAfAA8ewBBegAAAAD%2B%2FPtft98Mp%2BwWsfAVsvEbs%2FQeqvF8xO7%2F%2F%2F63yqkxdgM7gwE%2FggM%2BfQA%2BegBDeQDe7PIbotgQufcMufEPtfIPsvAbs%2FQvq%2Bfz%2Bf%2F%2B%2B%2FZKhR05hgBBhQI8hgBAgAI9ewD0%2B%2Fg3pswAtO8Cxf4Kw%2FsJvvYAqupKsNv%2B%2Fv7%2F%2FP5VkSU0iQA7jQA9hgBDgQU%2BfQH%2F%2Ff%2FQ6fM4sM4KsN8AteMCruIqqdbZ7PH8%2Fv%2Fg6Nc%2Fhg05kAA8jAM9iQI%2BhQA%2BgQDQu6b97uv%2F%2F%2F7V8Pqw3eiWz97q8%2Ff%2F%2F%2F%2F7%2FPptpkkqjQE4kwA7kAA5iwI8iAA8hQCOSSKdXjiyflbAkG7u2s%2F%2B%2F%2F39%2F%2F7r8utrqEYtjQE8lgA7kwA7kwA9jwA9igA9hACiWSekVRyeSgiYSBHx6N%2F%2B%2Fv7k7OFRmiYtlAA5lwI7lwI4lAA7kgI9jwE9iwI4iQCoVhWcTxCmb0K%2BooT8%2Fv%2F7%2F%2F%2FJ2r8fdwI1mwA3mQA3mgA8lAE8lAE4jwA9iwE%2BhwGfXifWvqz%2B%2Ff%2F58u%2Fev6Dt4tr%2B%2F%2F2ZuIUsggA7mgM6mAM3lgA5lgA6kQE%2FkwBChwHt4dv%2F%2F%2F728ei1bCi7VAC5XQ7kz7n%2F%2F%2F6bsZkgcB03lQA9lgM7kwA2iQktZToPK4r9%2F%2F%2F9%2F%2F%2FSqYK5UwDKZAS9WALIkFn%2B%2F%2F3%2F%2BP8oKccGGcIRJrERILYFEMwAAuEAAdX%2F%2Ff7%2F%2FP%2B%2BfDvGXQLIZgLEWgLOjlf7%2F%2F%2F%2F%2F%2F9QU90EAPQAAf8DAP0AAfMAAOUDAtr%2F%2F%2F%2F7%2B%2Fu2bCTIYwDPZgDBWQDSr4P%2F%2Fv%2F%2F%2FP5GRuABAPkAA%2FwBAfkDAPAAAesAAN%2F%2F%2B%2Fz%2F%2F%2F64g1C5VwDMYwK8Yg7y5tz8%2Fv%2FV1PYKDOcAAP0DAf4AAf0AAfYEAOwAAuAAAAD%2F%2FPvi28ymXyChTATRrIb8%2F%2F3v8fk6P8MAAdUCAvoAAP0CAP0AAfYAAO4AAACAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAQAA"
    },
    "Url": [
      {
        "-type": "application/x-suggestions+json",
        "-method": "GET",
        "-template": "https://www.google.com/complete/search?client=firefox&q={searchTerms}"
      },
      {
        "-type": "text/html",
        "-method": "GET",
        "-template": "https://www.google.com/search",
        "Param": [
          {
            "-name": "q",
            "-value": "{searchTerms}"
          },
          {
            "-name": "ie",
            "-value": "utf-8"
          },
          {
            "-name": "oe",
            "-value": "utf-8"
          },
          {
            "-name": "aq",
            "-value": "t"
          },
          {
            "-name": "rls",
            "-value": "{moz:distributionID}:{moz:locale}:{moz:official}"
          }
        ],
        "MozParam": {
          "-name": "client",
          "-condition": "defaultEngine",
          "-trueValue": "firefox-a",
          "-falseValue": "firefox"
        }
      },
      {
        "-type": "application/x-moz-keywordsearch",
        "-method": "GET",
        "-template": "https://www.google.com/search",
        "Param": [
          {
            "-name": "q",
            "-value": "{searchTerms}"
          },
          {
            "-name": "ie",
            "-value": "utf-8"
          },
          {
            "-name": "oe",
            "-value": "utf-8"
          },
          {
            "-name": "aq",
            "-value": "t"
          },
          {
            "-name": "rls",
            "-value": "{moz:distributionID}:{moz:locale}:{moz:official}"
          },
          {
            "-name": "channel",
            "-value": "fflb"
          }
        ],
        "MozParam": {
          "-name": "client",
          "-condition": "defaultEngine",
          "-trueValue": "firefox-a",
          "-falseValue": "firefox"
        }
      },
      {
        "-type": "application/x-moz-contextsearch",
        "-method": "GET",
        "-template": "https://www.google.com/search",
        "Param": [
          {
            "-name": "q",
            "-value": "{searchTerms}"
          },
          {
            "-name": "ie",
            "-value": "utf-8"
          },
          {
            "-name": "oe",
            "-value": "utf-8"
          },
          {
            "-name": "aq",
            "-value": "t"
          },
          {
            "-name": "rls",
            "-value": "{moz:distributionID}:{moz:locale}:{moz:official}"
          },
          {
            "-name": "channel",
            "-value": "rcs"
          }
        ],
        "MozParam": {
          "-name": "client",
          "-condition": "defaultEngine",
          "-trueValue": "firefox-a",
          "-falseValue": "firefox"
        }
      }
    ],
    "SearchForm": "https://www.google.com/"
  }
}

{
  "SearchPlugin": {
    "-xmlns": "http://www.mozilla.org/2006/browser/search/",
    "ShortName": "Amazon.com",
    "InputEncoding": "ISO-8859-1",
    "Image": {
      "-width": "16",
      "-height": "16",
      "#text": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAA2ZJREFUeNrEV01IFGEYfndXNzq0QdalJOoUbQiBUuopD2k/GFSG/Vxyq0OHbCUjwn7AEooUzUNRkdkhS1o9JBVaUDctKqhoRQi0WjtlxG6XVtfpfb6Z2caZ2ZnZXZdeeJxvx+973+f9+775XJIkkUaWMcoYG5TxfMpvxlvGMGM8+RYEFOxn/JJyLzOMiwwv7LqUCNQwHjpxIxKJ0Gg4LABZUVhIfr+f1jLSlNuMwyCwEHoZS6xmvxoZoc6Oq+JpJoVMpOncWdpcWZkOiRIQ2MKDp1az+kIhOtV40pHG6zdvpEOixc1/VtqFvKX5gmOX0pkL2yDgsZpxt+sORaPROe8Q6ncf3tPAk8eG3Ks14lA8brsZYZ2yukBAwOfzCeMHA3WGNXrCVpLnJKeqRyhAfX5RfNmILQF4urG0VIzxBBkU5aQI9agIeU4JqNLZ0UH9ob6sDWZE4MDefSn7P1txO/FcbxypQG18nhinew/u5zYC3dyG+qLL1qjjCKDg9C21q2a3oe9zRsCsn2PR2JzfKEy9PB96Nj8E0IJm54IaGZwPZsWJOU4jY1kD2OlAQhsJjKu3bSe7yPUzifpgMPsuOBY8brtR1evmyFt0IL0IzH4fJtcCH7kK1hn2/hh71G1yKKEdz/DBBIOTkUkRemzVl1uvCGKzE4OMIaFbiv1LSX51L7mXl4kxvgeO8vMaJk0PHiHvjl4DCTWs2lMOX0L6cwD/Bxlp6hNNv2gUT9MjsLiB8koaMOxJRgCMgPijWsqvaCP3qqqUZ4JVzUASYyFyL/WTqyggPxWH4qGtBlJzagCGIYjEzJv2zHe38vOUt6mNPGv20OyPMMV7yuVwL5IjBlLmRej1UX7VLdmLt+2CMVKTiSDn0wO1NPPyBOtdLL+MyzWkTbGhC5AGsBdKkEtWAjglgjUwCq/FGjhV0ZosdOhXI5FyH0DoIIK9slB0CS8UCgr8AlpvpamwqHpttcO4WtSqA57ioKENE05IqIYSY46uD4Is0qmGG2s9RYeS7adI3Paz3K6lTMmzIdFmXp/d1Gb1YvLF7i4IzxMfu1ITYWOe1VWix7U5tlKpXkzwYycONqfVLcW+cU7lQ0jePf360DqRS4zT/+Ny+ofRxPBoL6fa6zmu5uvtbkwZyE/lev6a8VV9+VeAAQADg36zc4GRNAAAAABJRU5ErkJggg=="
    },
    "Url": {
      "-type": "text/html",
      "-method": "GET",
      "-template": "http://www.amazon.com/exec/obidos/external-search/",
      "Param": [
        {
          "-name": "field-keywords",
          "-value": "{searchTerms}"
        },
        {
          "-name": "mode",
          "-value": "blended"
        },
        {
          "-name": "tag",
          "-value": "mozilla-20"
        },
        {
          "-name": "sourceid",
          "-value": "Mozilla-search"
        }
      ]
    },
    "SearchForm": "http://www.amazon.com/"
  }
}

{
  "SearchPlugin": {
    "-xmlns": "http://www.mozilla.org/2006/browser/search/",
    "-xmlns:os": "http://a9.com/-/spec/opensearch/1.1/",
    "os:ShortName": "YouTube Video Search",
    "os:Description": "Search for videos on YouTube",
    "os:InputEncoding": "UTF-8",
    "os:Image": {
      "-width": "16",
      "-height": "16",
      "#text": "data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAQAQAABMLAAATCwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgNkQkIDZGiCA2RzAgNkcwIDZH/CA2R/wgNkf8IDZH/CA2R/wgNkf8IDZH/CA2R2AgNkcwIDZHMCA2RhAgNkQYIDpWHCA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpWHCQ6ZzAkOmf8JDpn/CQ6Z/wkOmf8JDpb/BQhc/wgMgf8JDpn/CQ6Z/wkOmf8JDpn/CQ6Z/wkOmf8JDpn/CQ6ZzAkOnuoJDp7/CQ6e/wkOnv8JDp7/Exed/8jIy/9RU4j/Bwp0/wkOm/8JDp7/CQ6e/wkOnv8JDp7/CQ6e/wkOnuoJD6T8CQ+k/wkPpP8JD6T/CQ+k/xUbo//V1dX/1dXV/4yNrP8QFG//CA6Y/wkPpP8JD6T/CQ+k/wkPpP8JD6T8CQ+q/wkPqv8JD6r/CQ+q/wkPqv8WG6n/3d3d/93d3f/d3d3/v7/M/y0wjv8JD6r/CQ+q/wkPqv8JD6r/CQ+q/woQr/8KEK//ChCv/woQr/8KEK//Fx2v/+fn5//n5+f/5+fn/+jo6P+YmtP/ChCv/woQr/8KEK//ChCv/woQr/8KELX8ChC1/woQtf8KELX/ChC1/xgdtf/x8fH/8fHx//Ly8v+bndv/Ehi3/woQtf8KELX/ChC1/woQtf8KELX8ChG76goRu/8KEbv/ChG7/woRu/8YH77/+fn5/+/v9/9fY9H/ChG7/woRu/8KEbv/ChG7/woRu/8KEbv/ChG76goRwMwKEcD/ChHA/woRwP8KEcD/EBfB/6Ol5/8tM8n/ChHA/woRwP8KEcD/ChHA/woRwP8KEcD/ChHA/woRwMwLEcSHCxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcSHCxLICQsSyKULEsjMCxLI+QsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI0gsSyMwLEsiiAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP//AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAD//wAA//8AAA=="
    },
    "os:Url": {
      "-type": "text/html",
      "-method": "GET",
      "-template": "http://www.youtube.com/results?search_query={searchTerms}&page={startPage?}&utm_source=opensearch"
    }
  }
}

Digging around in the code Pointers

- http://jsontoxml.utilities-online.info/ convert between the two data formats
- changes frequently dont show up immediate (but get written on close)
- ordering in data store includes hidden engines. the number may not match the observed number
  - note in the example below all engines are off by 1 because google is order:1 and hidden, so youtube appears first in the list but has order:2
- getting a 3rd party search thingy
  - right click on a website search bar (ie youtube) and /profile/searchplugins will be created with an OpenSearch .xml
  - searchplugins doesn't exist until you create a non-default one
- SQLite table: install addon, go to tools -> SQLiteManager -> select db, go
- default engines have a hidden value, which afaic, is never applied to custom engines?

User:Anaaktgeboren/SearchEngineSync

Goals

Design

Requirements Gathering

Current Requirements

Current Suggestions/Notes

Design Reviews if needed

Implementation

Release

Reference

Draft Plan

Spec

Implementation schedule

Dependencies

Test Plan

TPS Tests

XPC Shell Tests

Scratch and Notes

Insights

current problems

What do these data formats look like anyway?

Digging around in the code Pointers

Navigation menu

Search