Extension Manager:Sqlite Storage

Current Situation

Currently the extension manager caches add-on metadata, blocklist and disabled status, downloads, pending installs and known updates. This is an incomplete list of the highlights:

• Add-on metadata (majority from install.rdf)
  • id
  • version
  • type
  • hidden
  • appManaged
  • targetApplication entries (modified by update checks)
  • default metadata
  • localized metadata
  • dependencies
  • appDisabled (set based on blocklist, compatibility and security)
  • userDisabled (user controlled)
  • available update (added by update checks)

• Pending installs (essentially a subset of the above items, added so they appear in the UI before restarting)

• Downloads (Used to display download progress in the UI)

The UI uses a template to display items, however it does not use the real extensions.rdf datasource. Instead a dynamic datasource is provided that calculates values for certain attributes:

• Add-on metadata
  • iconURL (may come from the real datasource, may be a direct file uri to a theme's icon)
  • previewImage (a direct file uri to the theme's preview image)
  • aboutURL (hidden if the add-on is disabled)
  • installDate (returns the mtime of the extension's directory)
  • compatible (returns whether the extension is compatible with the application)
  • providesUpdatesSecurely (whether the update path is secure)
  • satisfiesDependencies (whether all dependencies are available)
  • blocklisted (asks the blocklist service if the add-on is blocklisted)
  • opType (any pending operation for the add-on)
  • appManaged (can only return true if the extension is installed by the app)
  • hidden (can only return true if the extension is in a restricted install location)
  • locked (can only return true if the extension is in a restricted install location)
  • textual data (retrieved from properties, the localized area or the default area as appropriate)
  • isDisabled (based on appDisabled and userDisabled)
  • updateable (checks preferences and permissions to see if this add-on can be updated)

• Downloads
  • state (current install state)
  • progress (current download progress)

Every pane of the UI is currently built using an rdf based template. Switching panes swaps in a new template in a bunch of complex code. The Get Add-ons pane and Plugins pane are made available through the use of two in-memory datasources added to the templates composite datasource.

Much of the UI is dependant on the dynamically generated properties listed above to display the text in the correct locales and show state correctly.

Potential Improvements

While it may not be necessary to actually do these during this there are a few issues that can be thought about while changing to sqlite:

• XPInstall is now just a downloader for the extension manager. There isn't much point in doing this in a separate component and doing so introduces complexity and communication issues. We can just handle the downloading and certificate check directly from the EM. This will also allow us to check the cert for updates bug 248218
• Multiple target application ranges bug 301236
• Separate UI from backend bug 407329. Moving to pure notifications to the UI achieves a lot of this.
• Dynamic theme switching bug 226791
• Install add-ons without restart bug 256509
• Handle updates that change guid bug 404968

Proposed Changes

If we were to go the route of using sqlite templates to build the UI then it would be necessary to fill the database with all of the generated properties that are currently available on the EM rdf datasource as well as keep them up to date when things like the locale and preferences are changed in the application.

I don't believe that this is the sensible route to take. However instead this essentially requires that a new API be provided such that the UI can get the lists of installed add-ons and that contains methods to get those generated properties when necessary.

Database Schema

The database then should only contain the fixed information about installed add-ons, essentially all of the data read from the install.rdf, as well as any information needed to persist restarts like available update information.

This means that information about add-ons waiting to be installed and downloaded no longer need to be in the database. They can instead be held in in-memory data structures.

The database will look roughly like this:

There are some key points suggested in this design.

The addon table contains all known add-ons in all install locations. This is different from the current situation where we only cache data for the add-ons in the install location with the highest priority. This makes various future behaviours possible including using the newest version of an add-on regardless of install location, using any of the same add-on based on compatibility/blocklisting etc. We can create a view in the database that returns only the currently active add-ons for the common query cases.

The target_application data is linked to add-ons by addon id and version rather than being a foreign key into the addon table. Since an add-on might be installed with the same version in different install locations there is no need to duplicate the compatibility information (which should be the same) in the database. Likewise known updates have their target application info here. We could potentially think about caching this information almost permanently such that future installs of a previously installed add-on may not need another compatibility update check.

Note the locale_strings table. This is used for the contributors, localizers and other multi-string localized metadata. The type column will be an enumeration of the different types.

Might want to do the same for the requirements, however right now they come only out of the install.rdf and are never updated by remote data.

Do we actually need to keep all the target application data? We do currently but all we need for compatibility checks is minVersion and maxVersion and the app (toolkit/firefox). I'm generally in favour of keeping as much data but querying compatibility would be much simpler with the min/maxversion in the addon table

UI

Currently the UI is purely template based which has meant that plugin and add-on search results have ended up being put into in-memory rdf datasources. Switching away from templates will allow us to break this into a more straightforward approach. We just have one richlistbox per pane in a deck, on window startup we create richlistitems for every installed add-on, plugin and search results when available.

For the install and add-on lists we will have to have some kind of notifications from the EM backend to the UI to let it know that a change has been made that might require updating the UI.

For installs the already present nsIAddonInstallListener interface will be used. There will be some cleaning up so it gets passed the same nsIAddonDownload object for every single call.

Properties for installed add-ons don't change all that much, mainly pending operation changes. We already have state changes exposed through observer notifications, we can likely reuse/adapt that method.

API

Interface changes are necessary so that the UI can query for all of the necessary information. The main changes are to nsIExtensionManager and nsIUpdateItem (renamed to nsIAddon).

Extension Manager:Sqlite Storage:nsIExtensionManager

Extension Manager:Sqlite Storage:nsIAddon

Installed items would be initialised from the addon table in the database. This gives enough information for internal operations. Accesses to other properties like the localised information will cause additional db queries to find them.

For a pending install/downloading add-on the nsIAddon interface is mostly unfilled, things like the version are unavailable until after the download is complete and the install.rdf read. If they are updates then the id and name and some other information can be prefilled from the original item. This would enable us to reject updates that turn out to not be the id and version expected.

We could consider moving the add-on operations (uninstall, disable, enable etc.) onto the nsIAddon interface.

This is an overview of the data stores in the extension manager:

The extension manager has two main stores of information, the database (which holds installed add-ons and available updates) and an install cache which holds all of the add-ons being downloaded or pending restart. It also includes the available updates so that these pending items are managed in the same place.

This is not massively different to the current situation except that previously items in the install cache had to be added to the container in the database to allow the UI to function correctly.

The methods for retrieving addons on nsIExtensionManager pulls items from one or other of the database and install cache. getPendingInstalls, getAvailableUpdates both retrieve from the install cache. getItemList and getItemForID retrieve from the database.

The install cache is initially empty at every startup. When first queried it adds mirrored copies of all the available updates in the database. The installation methods create new items in the install cache. Update checks create new items in the database and a mirror of the same item in the install cache. Items remain in the install cache until shutdown.

Maybe items can be removed if they have been installed and no restart was required

Major Tasks

• API Implementation
• UI Work
• Create suitable test harness