User:Anaaktgeboren/SearchEngineSync

Goals

• Sync search engine prefs on Desktop
  • addition of an engine
  • removal of an engine
  • reordering of engines
  • keywords/aliases for a particular engine
  • restore defaults (potentially not a special case for implementation, but could be considered a feature by users)
• Format to be interoperable with firefox os, mobile firefox, or other future clients

• Nongoals:
  • implementation on firefox os
  • implementation on mobile firefox

Design

Requirements Gathering

• (deadline for input: july 26)
• Stakeholders
  • Desktop: Gavin
  • Desktop UX: madhava
  • Mobile: bnicholson, mfinkle
  • B@G: N/A, Ben Francis, b2g has no plugins and few preferences at this time
  • Privacy: Tom, Sid
  • Security: David Chan
  • Product: Asa
  • Services Integration: Gps
  • Community: dev-planning, dev-apps-firefox lists
  • Legal: bug 778823

Current Requirements

• does not sync hijacked queries
  • origin: Asa, browser cannot currently detect this, requires a new browser feature
• syncs code & not a server call, mitigate risk
  • origin sstamm, reducing the addon sync threat
• syncs keywords
• search order preserved
• 3rd party plugins appear
• honors enable/disable
  • desktop does not match mobile
• does not assume matching default lists on different clients (which is to say, can't assume google et al are installed everywhere) and syncing of default engine code is likely
• syncs which engines support the search suggest api

Current Suggestions/Notes

• input
  • want to avoid syncing app-shipped search plugins, origin: gavin
  • dont sync XML files, sync the JSON cache that's the result of parse origin: gavin
  • mobile uses same nsISearchService as desktop -mfinkle
  • should it be applicationID based, the way addons?
  • should we limit to a blessed type of plugin to address hijacking?
  • impact of Australis?

Design Reviews if needed

• privacy security?
• UI changes?

Implementation

• not started
• desktop will need code changes to address missing enable/disable state for custom search engines

Release

• • targeted to land on Nightly during Q3 (not a promise, subject to scope change)

Reference

• this implementation bug: 444284
• addon sync implemetnation bug: 534956
• https://developer.mozilla.org/en/Adding_search_engines_from_web_pages
• http://www.opensearch.org/Specifications/OpenSearch/1.1#OpenSearch_description_document
• https://developer.mozilla.org/en/Creating_OpenSearch_plugins_for_Firefox
• http://docs.services.mozilla.com/sync/index.html

Draft Plan

• currently full of holes

Spec

• proposed data model for a plugin record

{
  id // string id
  name // similiar but different from the description
  alias // optional, if user has set keywords
  order // optional if user has reordered engines
  hidden // optional, if user has 'deleted' a default engine, or an extensions had hidden a custom one.
  description // short description string
  searchForm // exists
  iconurl // the icon. found on defaults as well. 
  urls  //a nice fat object for included ones, less so for custom probably
  url // optional, not found on default
  installLocation // optional. if not present, this plugin is default [app]
  type //optional
  queryCharset //optional
  readOnly // optional in the event that we sync readonly plugins in the future
  locale // currently all records should have the same value, but it is reasonable for in the future 
    // for plugins to have different settings for this 

}

• Not Syncing
  • directory - machine specific
  • lastModifiedTime - machine & directory specific. used to invalidate cache
  • filepath - path to the xml, which wont exist since we arent syncing the xml and would be specific to a machine (where fx is installed, where profile is installed, etc)

• The OpenSearch plugins, ordering of search engines, keywords for search engines (both custom & default), and hidden flags (default engines only afaik) will be sync'ed across platforms (though the mobile client(s) will be written later)
  • parts of default engines will be synced (such as ordering & keywords)
  • the currently selected engine will not be synced, as that is a local setting and probably does not make sense to sync
  • read-only engines cannot be serialized, and therefore probably should not be sync'ed (I believe this is just the default ones)
• When a user changes or removes a keyword or custom engine, the removal will be synced
• When a user removes a default engine, the removal will be synced (but note that the browser does not actually remove those, it just hides it)
• The results of "Restore Defaults" will be synced (restores all deleted defaults, wipes db to default single entry (ketwords, ordering). This will not be treated as a special operation by sync
• Until the browser implements detecting hijacked searches, this engine will be preffed off by default for existing and new users
• Reconciliation: Merge by Default
  • When users choose model from settings, overrides this default?
  • what if the default ordering is different on two clients?
• Defaults on other Systems
  • when synced to mobile, default engines that are not default on mobile (there are 4 to desktop's 7+) will be disabled (visible but greyed out). Users will have the option to enable those for their system.
    • implies that keywords wont work for disabled engines, but that would be a client decision
  • when synced to mobile, user generated/added engines will be automatically enabled

Implementation schedule

• largely made up, as the repositories re-write is a huge wild card
• feedback due by 26-jul
• spec finalized by 3-Aug
• Implement store, record for json + tests by 10-aug
• Implement store, record for db(ordering, keywords) + tests by 15-aug
• Implement tracker + xpc tracker tests by 24-aug
• Implement engine + xpc tests by 7-sep
  • most likely to get snagged here
• Implement tps tests (if possible, see dependencies) and any remaining xpc tests by sep-14
• Debugging (unknown)
  • 'cause there's going to a landmine somewhere

Dependencies

• TPS supporting desktop to mobile operations (jgriffin's team)
  • TPS new data tupe addition (jgriffin's team)
• Desktop client supporting hijacked search detection (gavin?)
  • so engine can be enabled by default for new or existing users
• Someone implementing the corresponding mobile engine (?)
• repositories rewrite (gps)

Test Plan

TPS Tests

• Important Notes
  • tps does not support the native mobile at this time. timeframe unknown for support
  • dependency - adding a new engine will require work on tps. timeframe unknown, but likely to be reasonable
    • tests cannot really be written ahead of those changes to tps
• Tests
  • only reordering syncs between profile 1 & profile 2
  • only keyword addition, change, and removal sync between profile 1 & 2
  • adding and removing a search engine plugin syncs between profile 1 & 2
  • add plugin, add keyword, reorder, sync between two desktops, remove & sync again
  • add plugin, add keyword, reorder, sync between mobile (if mobile client exists) & desktop, remove & sync again

XPC Shell Tests

• make sure there is heavy testing about reordering with hidden & default engines, as some of the calls may have unexpected side effects (and some functions consider hidden in counts and others do not, same deal with what counts as default)
• impacted by repositories re-write?

Scratch and Notes

Insights

• what should happen when default lists dont match
  • defaults should be synced because they may not be present on other systems and users can expect to see them (though the mobile client may grey them out)
• user created keywords are called 'alias'es in code, stored in search-metadata.json
• search-metadata.json
  • may not exist if nothigns been changed
  • contains per plugin: order placement, hidden, and keywords
• /profile/search.json
  • "directories" will have two entries, the default one ("/Applications/Firefox.app/Contents/MacOS/searchplugins") & "/Users/mozilla/Documents/inbox-desktop/test_profiles/searchplugins" in my case) The first has no usefully syncable data, though it does have a lastModifiedTime (this is the lastModifiedTime of the directory, used for cache invalidation --Gavin)
• JAR loading code is for mobile, for their default plugin handling
• Australis is not making the search engines or their data go away
• "Search service changes will be required for this, almost certainly. Exposing the engine description as JSON and adding the ability to distinguish app-shipped plugins via the nsIBrowserSearchService API seem like good first steps to me" - gavin
  • Mobile will support search keywords & search suggest in the nearish future.
• /profile/search.sqlite(DO NOT SYNC) (this is no longer used after bug 699856, replaced by search-metadata.json --Gavin)
• pref browser.search.selectedEngine represents the currently selected entry, status "user set" is useful
• pref browser.search.useDBForOrder i bet that represents ordering (this gets set to true once the user has customized the search engine order, to indicate that we should use the positioning information in search-metadata.json rather than the one in default prefs (browser.search.order.*) --Gavin)
• pref browser.search.order.1-3 are hard coded defaults. do not sync

current problems

• search suggest has global preferences associated with it. to sync or not to sync?
  • and if so, should it go in the preferences engine, or in this one? I'm inclined to think it matters even if engines are not synced

• mobile has state that desktop does not (enable/disable)
• mobile will want more state about usage than desktop does
• how do i gracefully slurp the json content?
• design pondering

What do these data formats look like anyway?

• search-metadata.json. reordering + keywords. Note this file can be empty

{
    "[app]/google.xml":
    {
        "hidden":true,
        "alias":null,
        "order":1
    },
    "[profile]/youtube-video-search.xml":
    {
        "order":2,
        "alias":"anaaktge_test_keyword"
    },
    "[app]/bing.xml":
    {
        "order":3
    },
    "[app]/amazondotcom.xml":
    {
        "order":4,
        "alias":"anaaktge_test_keyword_default"
    },
    "[app]/eBay.xml":
    {
        "order":5
    },
    "[app]/twitter.xml":
    {
        "order":6
    },
    "[app]/wikipedia.xml":
    {
        "order":7
    },
    "[app]/yahoo.xml":
    {
        "order":8
    }
}

• search.json, one default plugin, one custom (youtube) plugin

{
    "version":7,
    "buildID":"20120730133507",
    "locale":"en-US",
    "directories":
    {
        "/Users/mozilla/moz/hg/services-central/./obj-x86_64-apple-darwin11.4.0/dist/NightlyDebug.app/Contents/MacOS/searchplugins":
        {
            "lastModifiedTime":1343683762000,
            "engines":
            [
                {
                    "_id":"[app]/amazondotcom.xml",
                    "_name":"Amazon.com",
                    "_hidden":false,
                    "description":"Amazon.com Search",
                    "__searchForm":"http://www.amazon.com/",
                    "_iconURL":"data:image/x-icon;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAAK/INwWK6QAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAHgSURBVHjalFM9TNtQEP4cB7PwM1RITUXIgsRaYEEVEyKZwhiyZAQyd0BhpFOlIjoBqhjSqVQMoVMLLAjEwECCQJkSkBqJYDOAFOMKFSf28d7DTUxiUDnp/Pzeu/vuu7t3ICKF6SLTMv2/lB0fRWKfjwDm4JJisYh0Oo3fpZLYT0SjSCQS8JAFMADNDZ3NZsnf1taiqVTKi4nGASruk5lkkmTmMB6JUKFQqO+DfX1eABWeQoVR6f7HSdM0obqu48Yw8G1tDT82NsRd1TSbU9BbGPCog8PDj+jLzurFoAVgMh4XxoNDQ6SqKi0tL9eBvAB8zZwymYxYY7EYAoEA8vm82BNTg6XUIs0MeGTZoR1mhXSnwNl4pmAbjU7mcjkKhkL1ynMnntZ4OEw3VyrV8utk7s5TdW++0QXz+1i3P7IK36t+PCfVn1OQOoOA0gXr5DPak+cPXbBK+/T3S69AtY3LJ98vZ1or/iLr+pTuvr59/A6s003UdqZFJF/PCKQ3o5CUznoBST2AfbEF/9iqYEDaIfwj73VJPEfgNTe0tWNYR0uwy9uOW0OkrgHI7z5ADo2C7v48nLV3XHKAT+x/1m1sX58xsBxg8rZJrDYD8DHHp4aJj/MK09sXjPOt46PcCzAACXY8/u34wN0AAAAASUVORK5CYII=",
                    "_urls":
                    [
                        {
                        "template":"http://www.amazon.com/exec/obidos/external-search/",
                        "rels":[],
                        "params":
                        [
                            {
                                "name":"field-keywords",
                                "value":"{searchTerms}"
                            },
                            {
                                "name":"mode",
                                "value":"blended"
                            },
                            {
                                "name":"tag",
                                "value":"mozilla-20"
                            },
                            {
                                "name":"sourceid",
                                "value":"Mozilla-search"
                            }
                        ]
                        }
                    ],
                    "filePath":"/Users/mozilla/moz/hg/services-central/./obj-x86_64-apple-darwin11.4.0/dist/NightlyDebug.app/Contents/MacOS/searchplugins/amazondotcom.xml"
                }
            ]
        }
        "/Users/mozilla/Documents/inbox-desktop/test_profiles/search_engine_sync_0/searchplugins":
        {
            "lastModifiedTime":1343685412000,
            "engines":
            [
                {
                    "_id":"[profile]/youtube-video-search.xml",
                    "_name":"YouTube Video Search",
                    "_hidden":false,
                    "description":"Search for videos on YouTube",
                    "__searchForm":"http://www.youtube.com",
                    "_iconURL":"data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAQAQAABMLAAATCwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgNkQkIDZGiCA2RzAgNkcwIDZH/CA2R/wgNkf8IDZH/CA2R/wgNkf8IDZH/CA2R2AgNkcwIDZHMCA2RhAgNkQYIDpWHCA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpWHCQ6ZzAkOmf8JDpn/CQ6Z/wkOmf8JDpb/BQhc/wgMgf8JDpn/CQ6Z/wkOmf8JDpn/CQ6Z/wkOmf8JDpn/CQ6ZzAkOnuoJDp7/CQ6e/wkOnv8JDp7/Exed/8jIy/9RU4j/Bwp0/wkOm/8JDp7/CQ6e/wkOnv8JDp7/CQ6e/wkOnuoJD6T8CQ+k/wkPpP8JD6T/CQ+k/xUbo//V1dX/1dXV/4yNrP8QFG//CA6Y/wkPpP8JD6T/CQ+k/wkPpP8JD6T8CQ+q/wkPqv8JD6r/CQ+q/wkPqv8WG6n/3d3d/93d3f/d3d3/v7/M/y0wjv8JD6r/CQ+q/wkPqv8JD6r/CQ+q/woQr/8KEK//ChCv/woQr/8KEK//Fx2v/+fn5//n5+f/5+fn/+jo6P+YmtP/ChCv/woQr/8KEK//ChCv/woQr/8KELX8ChC1/woQtf8KELX/ChC1/xgdtf/x8fH/8fHx//Ly8v+bndv/Ehi3/woQtf8KELX/ChC1/woQtf8KELX8ChG76goRu/8KEbv/ChG7/woRu/8YH77/+fn5/+/v9/9fY9H/ChG7/woRu/8KEbv/ChG7/woRu/8KEbv/ChG76goRwMwKEcD/ChHA/woRwP8KEcD/EBfB/6Ol5/8tM8n/ChHA/woRwP8KEcD/ChHA/woRwP8KEcD/ChHA/woRwMwLEcSHCxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcSHCxLICQsSyKULEsjMCxLI+QsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI0gsSyMwLEsiiAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP//AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAD//wAA//8AAA==",
                    "_urls":
                    [
                        {
                            "template":"http://www.youtube.com/results?search_query={searchTerms}&page={startPage?}&utm_source=opensearch",
                            "rels":[],
                            "params":[]
                        }
                    ],
                    "filePath":"/Users/mozilla/Documents/inbox-desktop/test_profiles/search_engine_sync_0/searchplugins/youtube-video-search.xml",
                    "_url":"http://www.youtube.com/opensearch?locale=en_US",
                    "_installLocation":2,
                    "type":3,
                    "queryCharset":"UTF-8",
                    "_readOnly":false
                }
            ]
        }
    }
}

• profile/searchplugins/ .xml

<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/" xmlns:os="http://a9.com/-/spec/opensearch/1.1/">
<os:ShortName>YouTube Video Search</os:ShortName>
<os:Description>Search for videos on YouTube</os:Description>
<os:InputEncoding>UTF-8</os:InputEncoding>
<os:Image width="16" height="16">data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAQAQAABMLAAATCwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgNkQkIDZGiCA2RzAgNkcwIDZH/CA2R/wgNkf8IDZH/CA2R/wgNkf8IDZH/CA2R2AgNkcwIDZHMCA2RhAgNkQYIDpWHCA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpWHCQ6ZzAkOmf8JDpn/CQ6Z/wkOmf8JDpb/BQhc/wgMgf8JDpn/CQ6Z/wkOmf8JDpn/CQ6Z/wkOmf8JDpn/CQ6ZzAkOnuoJDp7/CQ6e/wkOnv8JDp7/Exed/8jIy/9RU4j/Bwp0/wkOm/8JDp7/CQ6e/wkOnv8JDp7/CQ6e/wkOnuoJD6T8CQ+k/wkPpP8JD6T/CQ+k/xUbo//V1dX/1dXV/4yNrP8QFG//CA6Y/wkPpP8JD6T/CQ+k/wkPpP8JD6T8CQ+q/wkPqv8JD6r/CQ+q/wkPqv8WG6n/3d3d/93d3f/d3d3/v7/M/y0wjv8JD6r/CQ+q/wkPqv8JD6r/CQ+q/woQr/8KEK//ChCv/woQr/8KEK//Fx2v/+fn5//n5+f/5+fn/+jo6P+YmtP/ChCv/woQr/8KEK//ChCv/woQr/8KELX8ChC1/woQtf8KELX/ChC1/xgdtf/x8fH/8fHx//Ly8v+bndv/Ehi3/woQtf8KELX/ChC1/woQtf8KELX8ChG76goRu/8KEbv/ChG7/woRu/8YH77/+fn5/+/v9/9fY9H/ChG7/woRu/8KEbv/ChG7/woRu/8KEbv/ChG76goRwMwKEcD/ChHA/woRwP8KEcD/EBfB/6Ol5/8tM8n/ChHA/woRwP8KEcD/ChHA/woRwP8KEcD/ChHA/woRwMwLEcSHCxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcSHCxLICQsSyKULEsjMCxLI+QsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI0gsSyMwLEsiiAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP//AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAD//wAA//8AAA==</os:Image>
<os:Url type="text/html" method="GET" template="http://www.youtube.com/results?search_query={searchTerms}&page={startPage?}&utm_source=opensearch">
</os:Url>
</SearchPlugin>

• google search xml

<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/">
<ShortName>Google</ShortName>
<Description>Google Search</Description>
<InputEncoding>UTF-8</InputEncoding>
<Image width="16" height="16">data:image/png;base64,AAABAAEAEBAAAAEAGABoAwAAFgAAACgAAAAQAAAAIAAAAAEAGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADs9Pt8xetPtu9FsfFNtu%2BTzvb2%2B%2Fne4dFJeBw0egA%2FfAJAfAA8ewBBegAAAAD%2B%2FPtft98Mp%2BwWsfAVsvEbs%2FQeqvF8xO7%2F%2F%2F63yqkxdgM7gwE%2FggM%2BfQA%2BegBDeQDe7PIbotgQufcMufEPtfIPsvAbs%2FQvq%2Bfz%2Bf%2F%2B%2B%2FZKhR05hgBBhQI8hgBAgAI9ewD0%2B%2Fg3pswAtO8Cxf4Kw%2FsJvvYAqupKsNv%2B%2Fv7%2F%2FP5VkSU0iQA7jQA9hgBDgQU%2BfQH%2F%2Ff%2FQ6fM4sM4KsN8AteMCruIqqdbZ7PH8%2Fv%2Fg6Nc%2Fhg05kAA8jAM9iQI%2BhQA%2BgQDQu6b97uv%2F%2F%2F7V8Pqw3eiWz97q8%2Ff%2F%2F%2F%2F7%2FPptpkkqjQE4kwA7kAA5iwI8iAA8hQCOSSKdXjiyflbAkG7u2s%2F%2B%2F%2F39%2F%2F7r8utrqEYtjQE8lgA7kwA7kwA9jwA9igA9hACiWSekVRyeSgiYSBHx6N%2F%2B%2Fv7k7OFRmiYtlAA5lwI7lwI4lAA7kgI9jwE9iwI4iQCoVhWcTxCmb0K%2BooT8%2Fv%2F7%2F%2F%2FJ2r8fdwI1mwA3mQA3mgA8lAE8lAE4jwA9iwE%2BhwGfXifWvqz%2B%2Ff%2F58u%2Fev6Dt4tr%2B%2F%2F2ZuIUsggA7mgM6mAM3lgA5lgA6kQE%2FkwBChwHt4dv%2F%2F%2F728ei1bCi7VAC5XQ7kz7n%2F%2F%2F6bsZkgcB03lQA9lgM7kwA2iQktZToPK4r9%2F%2F%2F9%2F%2F%2FSqYK5UwDKZAS9WALIkFn%2B%2F%2F3%2F%2BP8oKccGGcIRJrERILYFEMwAAuEAAdX%2F%2Ff7%2F%2FP%2B%2BfDvGXQLIZgLEWgLOjlf7%2F%2F%2F%2F%2F%2F9QU90EAPQAAf8DAP0AAfMAAOUDAtr%2F%2F%2F%2F7%2B%2Fu2bCTIYwDPZgDBWQDSr4P%2F%2Fv%2F%2F%2FP5GRuABAPkAA%2FwBAfkDAPAAAesAAN%2F%2F%2B%2Fz%2F%2F%2F64g1C5VwDMYwK8Yg7y5tz8%2Fv%2FV1PYKDOcAAP0DAf4AAf0AAfYEAOwAAuAAAAD%2F%2FPvi28ymXyChTATRrIb8%2F%2F3v8fk6P8MAAdUCAvoAAP0CAP0AAfYAAO4AAACAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAQAA</Image>
<Url type="application/x-suggestions+json" method="GET" template="https://www.google.com/complete/search?client=firefox&q={searchTerms}"/>
<Url type="text/html" method="GET" template="https://www.google.com/search">
  <Param name="q" value="{searchTerms}"/><Param name="ie" value="utf-8"/><Param name="oe" value="utf-8"/><Param name="aq" value="t"/><Param name="rls" value="{moz:distributionID}:{moz:locale}:{moz:official}"/>
  <MozParam name="client" condition="defaultEngine" trueValue="firefox-a" falseValue="firefox"/>
</Url>
<!-- Keyword search URL is the same as the default, but with an additional parameter -->
<Url type="application/x-moz-keywordsearch" method="GET" template="https://www.google.com/search">
  <Param name="q" value="{searchTerms}"/><Param name="ie" value="utf-8"/><Param name="oe" value="utf-8"/><Param name="aq" value="t"/><Param name="rls" value="{moz:distributionID}:{moz:locale}:{moz:official}"/>
  <MozParam name="client" condition="defaultEngine" trueValue="firefox-a" falseValue="firefox"/>
  <Param name="channel" value="fflb"/>
</Url>
<!-- Context/Right-click search URL is the same as the default, but with an additional parameter -->
<Url type="application/x-moz-contextsearch" method="GET" template="https://www.google.com/search">
  <Param name="q" value="{searchTerms}"/><Param name="ie" value="utf-8"/><Param name="oe" value="utf-8"/><Param name="aq" value="t"/><Param name="rls" value="{moz:distributionID}:{moz:locale}:{moz:official}"/>
  <MozParam name="client" condition="defaultEngine" trueValue="firefox-a" falseValue="firefox"/>
  <Param name="channel" value="rcs"/>
</Url>
<SearchForm>https://www.google.com/</SearchForm>
</SearchPlugin>

• amazon xml

<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/">
<ShortName>Amazon.com</ShortName>
<InputEncoding>ISO-8859-1</InputEncoding>
<Image width="16" height="16">data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAA2ZJREFUeNrEV01IFGEYfndXNzq0QdalJOoUbQiBUuopD2k/GFSG/Vxyq0OHbCUjwn7AEooUzUNRkdkhS1o9JBVaUDctKqhoRQi0WjtlxG6XVtfpfb6Z2caZ2ZnZXZdeeJxvx+973+f9+775XJIkkUaWMcoYG5TxfMpvxlvGMGM8+RYEFOxn/JJyLzOMiwwv7LqUCNQwHjpxIxKJ0Gg4LABZUVhIfr+f1jLSlNuMwyCwEHoZS6xmvxoZoc6Oq+JpJoVMpOncWdpcWZkOiRIQ2MKDp1az+kIhOtV40pHG6zdvpEOixc1/VtqFvKX5gmOX0pkL2yDgsZpxt+sORaPROe8Q6ncf3tPAk8eG3Ks14lA8brsZYZ2yukBAwOfzCeMHA3WGNXrCVpLnJKeqRyhAfX5RfNmILQF4urG0VIzxBBkU5aQI9agIeU4JqNLZ0UH9ob6sDWZE4MDefSn7P1txO/FcbxypQG18nhinew/u5zYC3dyG+qLL1qjjCKDg9C21q2a3oe9zRsCsn2PR2JzfKEy9PB96Nj8E0IJm54IaGZwPZsWJOU4jY1kD2OlAQhsJjKu3bSe7yPUzifpgMPsuOBY8brtR1evmyFt0IL0IzH4fJtcCH7kK1hn2/hh71G1yKKEdz/DBBIOTkUkRemzVl1uvCGKzE4OMIaFbiv1LSX51L7mXl4kxvgeO8vMaJk0PHiHvjl4DCTWs2lMOX0L6cwD/Bxlp6hNNv2gUT9MjsLiB8koaMOxJRgCMgPijWsqvaCP3qqqUZ4JVzUASYyFyL/WTqyggPxWH4qGtBlJzagCGIYjEzJv2zHe38vOUt6mNPGv20OyPMMV7yuVwL5IjBlLmRej1UX7VLdmLt+2CMVKTiSDn0wO1NPPyBOtdLL+MyzWkTbGhC5AGsBdKkEtWAjglgjUwCq/FGjhV0ZosdOhXI5FyH0DoIIK9slB0CS8UCgr8AlpvpamwqHpttcO4WtSqA57ioKENE05IqIYSY46uD4Is0qmGG2s9RYeS7adI3Paz3K6lTMmzIdFmXp/d1Gb1YvLF7i4IzxMfu1ITYWOe1VWix7U5tlKpXkzwYycONqfVLcW+cU7lQ0jePf360DqRS4zT/+Ny+ofRxPBoL6fa6zmu5uvtbkwZyE/lev6a8VV9+VeAAQADg36zc4GRNAAAAABJRU5ErkJggg==</Image>
<Url type="text/html" method="GET" template="http://www.amazon.com/exec/obidos/external-search/">
  <Param name="field-keywords" value="{searchTerms}"/>
  <Param name="mode" value="blended"/>
  <Param name="tag" value="mozilla-20"/>
  <Param name="sourceid" value="Mozilla-search"/>
</Url>
<SearchForm>http://www.amazon.com/</SearchForm>
</SearchPlugin>

• Now let's turn those xml documents into JSON (records must contain JSONable code) (google, amz, custom(youtube))

{
  "SearchPlugin": {
    "-xmlns": "http://www.mozilla.org/2006/browser/search/",
    "ShortName": "Google",
    "Description": "Google Search",
    "InputEncoding": "UTF-8",
    "Image": {
      "-width": "16",
      "-height": "16",
      "#text": "data:image/png;base64,AAABAAEAEBAAAAEAGABoAwAAFgAAACgAAAAQAAAAIAAAAAEAGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADs9Pt8xetPtu9FsfFNtu%2BTzvb2%2B%2Fne4dFJeBw0egA%2FfAJAfAA8ewBBegAAAAD%2B%2FPtft98Mp%2BwWsfAVsvEbs%2FQeqvF8xO7%2F%2F%2F63yqkxdgM7gwE%2FggM%2BfQA%2BegBDeQDe7PIbotgQufcMufEPtfIPsvAbs%2FQvq%2Bfz%2Bf%2F%2B%2B%2FZKhR05hgBBhQI8hgBAgAI9ewD0%2B%2Fg3pswAtO8Cxf4Kw%2FsJvvYAqupKsNv%2B%2Fv7%2F%2FP5VkSU0iQA7jQA9hgBDgQU%2BfQH%2F%2Ff%2FQ6fM4sM4KsN8AteMCruIqqdbZ7PH8%2Fv%2Fg6Nc%2Fhg05kAA8jAM9iQI%2BhQA%2BgQDQu6b97uv%2F%2F%2F7V8Pqw3eiWz97q8%2Ff%2F%2F%2F%2F7%2FPptpkkqjQE4kwA7kAA5iwI8iAA8hQCOSSKdXjiyflbAkG7u2s%2F%2B%2F%2F39%2F%2F7r8utrqEYtjQE8lgA7kwA7kwA9jwA9igA9hACiWSekVRyeSgiYSBHx6N%2F%2B%2Fv7k7OFRmiYtlAA5lwI7lwI4lAA7kgI9jwE9iwI4iQCoVhWcTxCmb0K%2BooT8%2Fv%2F7%2F%2F%2FJ2r8fdwI1mwA3mQA3mgA8lAE8lAE4jwA9iwE%2BhwGfXifWvqz%2B%2Ff%2F58u%2Fev6Dt4tr%2B%2F%2F2ZuIUsggA7mgM6mAM3lgA5lgA6kQE%2FkwBChwHt4dv%2F%2F%2F728ei1bCi7VAC5XQ7kz7n%2F%2F%2F6bsZkgcB03lQA9lgM7kwA2iQktZToPK4r9%2F%2F%2F9%2F%2F%2FSqYK5UwDKZAS9WALIkFn%2B%2F%2F3%2F%2BP8oKccGGcIRJrERILYFEMwAAuEAAdX%2F%2Ff7%2F%2FP%2B%2BfDvGXQLIZgLEWgLOjlf7%2F%2F%2F%2F%2F%2F9QU90EAPQAAf8DAP0AAfMAAOUDAtr%2F%2F%2F%2F7%2B%2Fu2bCTIYwDPZgDBWQDSr4P%2F%2Fv%2F%2F%2FP5GRuABAPkAA%2FwBAfkDAPAAAesAAN%2F%2F%2B%2Fz%2F%2F%2F64g1C5VwDMYwK8Yg7y5tz8%2Fv%2FV1PYKDOcAAP0DAf4AAf0AAfYEAOwAAuAAAAD%2F%2FPvi28ymXyChTATRrIb8%2F%2F3v8fk6P8MAAdUCAvoAAP0CAP0AAfYAAO4AAACAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAQAA"
    },
    "Url": [
      {
        "-type": "application/x-suggestions+json",
        "-method": "GET",
        "-template": "https://www.google.com/complete/search?client=firefox&q={searchTerms}"
      },
      {
        "-type": "text/html",
        "-method": "GET",
        "-template": "https://www.google.com/search",
        "Param": [
          {
            "-name": "q",
            "-value": "{searchTerms}"
          },
          {
            "-name": "ie",
            "-value": "utf-8"
          },
          {
            "-name": "oe",
            "-value": "utf-8"
          },
          {
            "-name": "aq",
            "-value": "t"
          },
          {
            "-name": "rls",
            "-value": "{moz:distributionID}:{moz:locale}:{moz:official}"
          }
        ],
        "MozParam": {
          "-name": "client",
          "-condition": "defaultEngine",
          "-trueValue": "firefox-a",
          "-falseValue": "firefox"
        }
      },
      {
        "-type": "application/x-moz-keywordsearch",
        "-method": "GET",
        "-template": "https://www.google.com/search",
        "Param": [
          {
            "-name": "q",
            "-value": "{searchTerms}"
          },
          {
            "-name": "ie",
            "-value": "utf-8"
          },
          {
            "-name": "oe",
            "-value": "utf-8"
          },
          {
            "-name": "aq",
            "-value": "t"
          },
          {
            "-name": "rls",
            "-value": "{moz:distributionID}:{moz:locale}:{moz:official}"
          },
          {
            "-name": "channel",
            "-value": "fflb"
          }
        ],
        "MozParam": {
          "-name": "client",
          "-condition": "defaultEngine",
          "-trueValue": "firefox-a",
          "-falseValue": "firefox"
        }
      },
      {
        "-type": "application/x-moz-contextsearch",
        "-method": "GET",
        "-template": "https://www.google.com/search",
        "Param": [
          {
            "-name": "q",
            "-value": "{searchTerms}"
          },
          {
            "-name": "ie",
            "-value": "utf-8"
          },
          {
            "-name": "oe",
            "-value": "utf-8"
          },
          {
            "-name": "aq",
            "-value": "t"
          },
          {
            "-name": "rls",
            "-value": "{moz:distributionID}:{moz:locale}:{moz:official}"
          },
          {
            "-name": "channel",
            "-value": "rcs"
          }
        ],
        "MozParam": {
          "-name": "client",
          "-condition": "defaultEngine",
          "-trueValue": "firefox-a",
          "-falseValue": "firefox"
        }
      }
    ],
    "SearchForm": "https://www.google.com/"
  }
}

{
  "SearchPlugin": {
    "-xmlns": "http://www.mozilla.org/2006/browser/search/",
    "ShortName": "Amazon.com",
    "InputEncoding": "ISO-8859-1",
    "Image": {
      "-width": "16",
      "-height": "16",
      "#text": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAA2ZJREFUeNrEV01IFGEYfndXNzq0QdalJOoUbQiBUuopD2k/GFSG/Vxyq0OHbCUjwn7AEooUzUNRkdkhS1o9JBVaUDctKqhoRQi0WjtlxG6XVtfpfb6Z2caZ2ZnZXZdeeJxvx+973+f9+775XJIkkUaWMcoYG5TxfMpvxlvGMGM8+RYEFOxn/JJyLzOMiwwv7LqUCNQwHjpxIxKJ0Gg4LABZUVhIfr+f1jLSlNuMwyCwEHoZS6xmvxoZoc6Oq+JpJoVMpOncWdpcWZkOiRIQ2MKDp1az+kIhOtV40pHG6zdvpEOixc1/VtqFvKX5gmOX0pkL2yDgsZpxt+sORaPROe8Q6ncf3tPAk8eG3Ks14lA8brsZYZ2yukBAwOfzCeMHA3WGNXrCVpLnJKeqRyhAfX5RfNmILQF4urG0VIzxBBkU5aQI9agIeU4JqNLZ0UH9ob6sDWZE4MDefSn7P1txO/FcbxypQG18nhinew/u5zYC3dyG+qLL1qjjCKDg9C21q2a3oe9zRsCsn2PR2JzfKEy9PB96Nj8E0IJm54IaGZwPZsWJOU4jY1kD2OlAQhsJjKu3bSe7yPUzifpgMPsuOBY8brtR1evmyFt0IL0IzH4fJtcCH7kK1hn2/hh71G1yKKEdz/DBBIOTkUkRemzVl1uvCGKzE4OMIaFbiv1LSX51L7mXl4kxvgeO8vMaJk0PHiHvjl4DCTWs2lMOX0L6cwD/Bxlp6hNNv2gUT9MjsLiB8koaMOxJRgCMgPijWsqvaCP3qqqUZ4JVzUASYyFyL/WTqyggPxWH4qGtBlJzagCGIYjEzJv2zHe38vOUt6mNPGv20OyPMMV7yuVwL5IjBlLmRej1UX7VLdmLt+2CMVKTiSDn0wO1NPPyBOtdLL+MyzWkTbGhC5AGsBdKkEtWAjglgjUwCq/FGjhV0ZosdOhXI5FyH0DoIIK9slB0CS8UCgr8AlpvpamwqHpttcO4WtSqA57ioKENE05IqIYSY46uD4Is0qmGG2s9RYeS7adI3Paz3K6lTMmzIdFmXp/d1Gb1YvLF7i4IzxMfu1ITYWOe1VWix7U5tlKpXkzwYycONqfVLcW+cU7lQ0jePf360DqRS4zT/+Ny+ofRxPBoL6fa6zmu5uvtbkwZyE/lev6a8VV9+VeAAQADg36zc4GRNAAAAABJRU5ErkJggg=="
    },
    "Url": {
      "-type": "text/html",
      "-method": "GET",
      "-template": "http://www.amazon.com/exec/obidos/external-search/",
      "Param": [
        {
          "-name": "field-keywords",
          "-value": "{searchTerms}"
        },
        {
          "-name": "mode",
          "-value": "blended"
        },
        {
          "-name": "tag",
          "-value": "mozilla-20"
        },
        {
          "-name": "sourceid",
          "-value": "Mozilla-search"
        }
      ]
    },
    "SearchForm": "http://www.amazon.com/"
  }
}

{
  "SearchPlugin": {
    "-xmlns": "http://www.mozilla.org/2006/browser/search/",
    "-xmlns:os": "http://a9.com/-/spec/opensearch/1.1/",
    "os:ShortName": "YouTube Video Search",
    "os:Description": "Search for videos on YouTube",
    "os:InputEncoding": "UTF-8",
    "os:Image": {
      "-width": "16",
      "-height": "16",
      "#text": "data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAQAQAABMLAAATCwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgNkQkIDZGiCA2RzAgNkcwIDZH/CA2R/wgNkf8IDZH/CA2R/wgNkf8IDZH/CA2R2AgNkcwIDZHMCA2RhAgNkQYIDpWHCA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpX/CA6V/wgOlf8IDpWHCQ6ZzAkOmf8JDpn/CQ6Z/wkOmf8JDpb/BQhc/wgMgf8JDpn/CQ6Z/wkOmf8JDpn/CQ6Z/wkOmf8JDpn/CQ6ZzAkOnuoJDp7/CQ6e/wkOnv8JDp7/Exed/8jIy/9RU4j/Bwp0/wkOm/8JDp7/CQ6e/wkOnv8JDp7/CQ6e/wkOnuoJD6T8CQ+k/wkPpP8JD6T/CQ+k/xUbo//V1dX/1dXV/4yNrP8QFG//CA6Y/wkPpP8JD6T/CQ+k/wkPpP8JD6T8CQ+q/wkPqv8JD6r/CQ+q/wkPqv8WG6n/3d3d/93d3f/d3d3/v7/M/y0wjv8JD6r/CQ+q/wkPqv8JD6r/CQ+q/woQr/8KEK//ChCv/woQr/8KEK//Fx2v/+fn5//n5+f/5+fn/+jo6P+YmtP/ChCv/woQr/8KEK//ChCv/woQr/8KELX8ChC1/woQtf8KELX/ChC1/xgdtf/x8fH/8fHx//Ly8v+bndv/Ehi3/woQtf8KELX/ChC1/woQtf8KELX8ChG76goRu/8KEbv/ChG7/woRu/8YH77/+fn5/+/v9/9fY9H/ChG7/woRu/8KEbv/ChG7/woRu/8KEbv/ChG76goRwMwKEcD/ChHA/woRwP8KEcD/EBfB/6Ol5/8tM8n/ChHA/woRwP8KEcD/ChHA/woRwP8KEcD/ChHA/woRwMwLEcSHCxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcT/CxHE/wsRxP8LEcSHCxLICQsSyKULEsjMCxLI+QsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI/wsSyP8LEsj/CxLI0gsSyMwLEsiiAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP//AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAD//wAA//8AAA=="
    },
    "os:Url": {
      "-type": "text/html",
      "-method": "GET",
      "-template": "http://www.youtube.com/results?search_query={searchTerms}&page={startPage?}&utm_source=opensearch"
    }
  }
}

Digging around in the code Pointers

• • http://jsontoxml.utilities-online.info/ convert between the two data formats
  • changes frequently dont show up immediate (but get written on close)
  • ordering in data store includes hidden engines. the number may not match the observed number
    • note in the example below all engines are off by 1 because google is order:1 and hidden, so youtube appears first in the list but has order:2
  • getting a 3rd party search thingy
    • right click on a website search bar (ie youtube) and /profile/searchplugins will be created with an OpenSearch .xml
    • searchplugins doesn't exist until you create a non-default one
  • SQLite table: install addon, go to tools -> SQLiteManager -> select db, go
  • default engines have a hidden value, which afaic, is never applied to custom engines?