MozillaWiki

Places:Plan

Revision as of 16:28, 28 August 2009 by Mak77 (talk | contribs) (→‎Untargeted stuff)

Roadmap for future work on Places, the SQLite-driven Bookmarks and History system.

Firefox 3.6

Database related

VACUUM Places database

Some user reports slow database performances, due to fragmentation, we should start investigate and implement Vacuum in Places, and in future make that app-wide.

  • PROS: smaller and faster db.
  • CONS: vacuuming too much can cause slower INSERTs, vacuum takes long time.
  • EST: 1 week.
  • BUG: bug 512854
  • DEPENDS:
  • STATUS: active
  • TEAM: mak

Fix dynamic containers and convert livemark children to them

All queries involving bookmarks have to filter out livemarks children that are not real bookmarks, also checking if an uri is bookmarked requires a similar filtering. Moreover the new async locationbar is not correctly filtering these out. Plus livemarks service is slow in updating livemarks because bookmarks are sync written to disk. Dynamic containers stuff can be useful for extensions.

  • PROS: faster bookmarks queries, simpler bookmarks code. no more UI-blocking while updating livemarks (make livemark service as async as possible).
  • CONS: Will require a separate db to hold livermarks children. Will need some API change to livemarks service to allow extension developers to access livemarks children, and add their own "annotations" on them.
  • EST: 3 weeks.
  • BUG: bug 453530
  • DEPENDS: bug 457708
  • STATUS: active
  • TEAM: mak

Backend related

Find async query wins [p2]

Need to find points that require async love.

Tests re-enable/cleanup

We have a bunch of tests (also folders of tests) that have been disabled and never touched again, and bugs filed about queries tests containing tests that were never pushed due to minor issues that now could be fixed (or is worth fixing them).

  • PROS: larger test base at a relatively low cost.
  • CONS:
  • EST: 1 week.
  • BUG: bug 509868
  • DEPENDS:
  • STATUS: active
  • TEAM: mak

Frontend related

Firefox 3.7

Database related

Move tags out of bookmarks table

Tags are actually bookmarks folders, so we have to duplicate bookmarks inside them, pay attention to not add containers/separators inside them, move bookmarks when merging/splitting two tags, filter out them from all bookmarks queries. Having tags in a separated table would make all of this much easier and faster. Apart performance, we also have bugs that are coming from our "fancy" tags implementation and should be workaround if we don't fix the implementation.

  • PROS: faster bookmarks queries, simpler bookmarks code. faster autocomplete queries. no data duplication. easier merge/update tags.
  • CONS: Will require a separate table to hold tags. Need to revise all tagging related code and tagging service.
  • EST: 2 weeks.
  • BUG: bug 424160
  • DEPENDS:
  • STATUS: inactive

Denormalize info into moz_places and revise PlacesSQLQueryBuilder

Actually the query builder code is in history.cpp, but there is much related code out of it. Ideally it should just receive queries and options, and return an OPTIMIZED sql query. As it is code is confusing and returned queries are not always optimized. We should flatten most informations in moz_places and rewrite the builder so that it will return better queries. For example it should not JOIN history table if we are not interested in querying history values (like querying by timeframe).

Actually we could just convert nsNavHistoryQuery.cpp to JS, and make the SQL query be a property of a query, we could provide an inclusion mode that will use: UNION, INTERSECT, EXCEPT... the query system should be able to get a query string in places format, or json, or anything other, having a parser to convert it to an internal representation, and use that to build SQL queries in a de-normalized and performant way.

  • PROS: more manageable and separated code between history and SQL. Easily to build more optimized queries (better performances for uri based queries).
  • CONS: this is a rewrite with some risk.
  • EST: 1.5 weeks.
  • BUG: bug 500290
  • DEPENDS:
  • STATUS: inactive

make GUIDs not annotations

This is a Weave need, we need to talk to them and understand what are the problems andhow we can fix them.

Backend related

Make Expiration async and performant

Expiration currently runs on shutdown and on idle, if possible it should instead run in background and be async. We should try to unify it with queries that run in nsPlacesDBFlush, so that we will write to disk the less often as possible. Expiration should be removed from history, and moved to a separate JS component for sake of simplifying history code and separating not completely related code. Expiration algorithm could even be reviewed a bit to make it simpler.

  • PROS: no UI-blocking or standby blocking expiration. Less disk writes. easier to manage code.
  • CONS: It's a rewrite, with some possible privacy risk.
  • EST: 2 weeks.
  • BUG:
  • DEPENDS:
  • STATUS: inactive

Asynchronous isVisited

This is needed for Electrolysis and Tsnappiness, isvisited sould get data async.

  • PROS: no UI-blocking and Tp wins.
  • CONS:
  • EST:
  • BUG: bug 511960
  • DEPENDS:
  • STATUS: active
  • TEAM: sdwilsh

Open containers asynchronously

Opening containers is slowing down and blocking UI, it should be done async.

  • PROS: UI responsiveness.
  • CONS: Can cause views bugs, needs to move back some code to the backend.
  • EST:
  • BUG: bug 490714
  • DEPENDS:
  • STATUS: active
  • TEAM: adw

Frontend related

  • cut and paste is slow
  • dnd is slow
  • delayed bookmarks init (delayed toolbar init, like chrome)
  • history menu (cache w/ json? live update?)

Firefox 4.0

Database related

Convert bookmarks table to a preordered nested tree

Nested Trees are a good way to manage a hierachy in sql, you can most likely select anything with one query, and no recursion is needed at all. So you can with one query know if a node is descendant or ancestor of another one. This will allow to print out bookmarks paths in the UI, search for folders, check ancestors/descendant with better performances and so on.

  • PROS: allows to completely avoid recursion when querying descendant/ancestors of a bookmark. fast children count. Self protection from creating cycles of bookmarks.
  • CONS: Needs schema changes (2 or 3 columns in moz_bookmarks). INSERTS/UPDATES/DELETES could be a bit slower and harder (triggers could help for complexity).
  • EST: 2 weeks.
  • BUG: bug 408991
  • DEPENDS: bug 424160, bug 453530
  • STATUS: inactive

Denormalize referrer in moz_places table

Actually from_visit reports the page we have come from, so in case of a double redirect we would have to recurse twice to go to the original site. It would be nice having a new column reporting a sort of referer, or a new table schema allowing to query without recursion. We could ideally reuse session column (that is actually minused) to track redirects, querying first visit with the same session would give back a somehow correct referrer.

  • PROS: Easily query for referer and visit-chain, avoiding recursive queries. Useful for extsnsions and security checks (how did i come here?)
  • CONS: Needs schema changes.
  • EST: 1,5 weeks.
  • BUG: bug 468710
  • DEPENDS:
  • STATUS: inactive

remove temp tables

Change keywords schema to drop multiple keywords support

Keywords live in a separated table, but they are linked through a keyword_id column in moz_bookmarks, this is often null, and that's not good since it will take a lot of useless space in the db We should instead have a (keyword, item_id) table, and no column in moz_bookmarks.

  • PROS: less wasted space in the db.
  • CONS: Needs hard schema change.
  • EST:
  • BUG:
  • DEPENDS:
  • STATUS: inactive

figure out why annotations are slow (weave)

Backend related

Notifications should give more informations to the observers

Our actual notifications don't tell anything to the observer, it will have to query again to know informations that the notifier already has. For example if i get onItemAdded, i have to query bookmarks service to know which kind of item, and at what time it was added. Having better notifications would reduce queries load and make observers more performant.

  • PROS: less queries. faster observers.
  • CONS: Compatibility with extensions.
  • EST:
  • BUG: bug 494380
  • DEPENDS: bug 498130
  • STATUS: active
  • TEAM: mano, mak
  • non crappy notifications (avoid removed/added on index changes)
  • reduce observer traffic
  • async addBookmark
  • async addVisit

Remove LAZY_ADD, use async storage instead

LAZY_ADD is the old workaround to implement an async addition of visits and favicons, we should get rid of it and use async storage instead.

  • PROS: less code bloat and UI-blocking.
  • CONS: Views will be updated faster, will that be an issue? We are just moving work earlier.
  • EST: 1 week.
  • BUG: bug 499828, bug 429989
  • DEPENDS:
  • STATUS: inactive

Frontend related

investigate use of templates

Not-yet targeted stuff

Review Places views drag & drop code

Places drag & drop code needs some further cleaning: menus code should be cleaned up to just use new d&d API, trees code should be fixed since trees are now partially using new d&d API, we should support more natural dragging through menus (diagonal dragging).

  • PROS: more natural D&D experience for users. cleaner code.
  • CONS: no deep tests are possible, regressions risk.
  • EST: 1,5 weeks.
  • BUG: bug 419911
  • DEPENDS:
  • STATUS: inactive

Detect real database corruption (integrity_check)

Actually we don't detect real database file corruptions because it's slow, we should still, because there is no easy path to bring users out of it. Better if done at a browser/app wide level.

  • PROS: allow users to recover data
  • CONS: slow
  • EST:
  • BUG: bug 506905
  • DEPENDS:
  • STATUS: inactive

Places performance tests

Tests and graphs for UI performance: microbenchmarking, dirty profile testing, etc. Needs to hook into Talos for continuous integration, and Perfomatic for reporting.

  • PROS:
  • CONS:
  • EST:
  • BUG:
  • DEPENDS:
  • STATUS: partly active
  • TEAM: ddahl, alice, dietrich

Create a Places Page-Thumbnails service

This would be useful in various parts of the UI and good for extensions. Some extensions are reimplementing this, and if not done correctly this will hurt us badly (See old Google Toolbar for example, putting all thumbnails in annotations, and causing places.sqlite to bloat hundred of MBs)

  • PROS: Usable in library, and many other points of access to bookmarks/history, maybe in tooltips too. Built to be async. Extensions that need thumbnails won't hurt us.
  • CONS: will use some disk space, privacy concerns.
  • EST: 2 weeks.
  • BUG: bug 497543
  • DEPENDS:
  • STATUS: inactive

Queries improvements

Support querying for the exact data set that the location bar can, support querying on transition type, implement RESULTS_AS_FULL_VISIT or put everything needed into a visit node and deprecate FULL_VISIT, expose frecency.

  • PROS: better queries, allow for further enhancements.
  • CONS:
  • EST: 2 weeks.
  • BUG:
  • DEPENDS:
  • STATUS: inactive

Better JSON-friendly querying API

Querying Places is hard, having a better JSON API would be cool, this is actually under experimentation in JetPack, but we would like to uplift that.

  • PROS: easier to query for extensions.
  • CONS:
  • EST:
  • BUG:
  • DEPENDS:
  • STATUS: active
  • TEAM: ddahl

Provide extended API to insert or get bookmark properties

Transaction manager, sync services and restore/import have to insert bookmarks and then set various additional properties. We also do not provide a way to get back all properties for a bookmark with a single call. This causes more queries traffic and notifications than needed. Would be cool to provide insertBookmarkExt and getItemProperties to allow setting getting all properties at once.

  • PROS: faster bulk inserting and reading
  • CONS: API noise
  • EST: 1 week.
  • BUG: bug 473666, bug 473664
  • DEPENDS:
  • STATUS: inactive

Fix Cut in Places frontend, to avoid dataloss

Actually Cut is doing Remove&Copy, this means to undo it user will have to undo twice, and after a Cut if user closes browser or it crashes the cutted elements will be lost.

  • PROS: no dataloss, single action undo
  • CONS:
  • EST: 2 days.
  • BUG: bug 484578
  • DEPENDS:
  • STATUS: inactive

Make bookmarks service initialization fault tolerant and faster

Check roots are in a good shape on start, but only do that when we are sure something has changed in the db.

  • PROS: avoid bloating places.sqlite
  • CONS: harder to query
  • EST:
  • BUG: bug 478912
  • DEPENDS:
  • STATUS: active
  • TEAM: adw

deCOM internal bookmarks and history observers

Avoid xpcom overhead, use results globally to observe changes.

Add UI for quick tagging

Quick tagging UI through keyword.

  • PROS: faster tagging for heavy taggers.
  • CONS: discoverability?
  • EST: practically done
  • BUG: bug 434946
  • DEPENDS:
  • STATUS: active
  • TEAM: dietrich

Add hooks for extensions in Library

Library needs to provide hooks for extensions content. Maybe a specific root where they can put dynamic containers or personalized content. There should be some sort of "registration" for both content and special views.

  • PROS: easier for extensions to provide their own views for the Library
  • CONS: adds complexity to the Library, Library perf/bloat.
  • EST:
  • BUG:
  • DEPENDS:
  • STATUS: inactive

Use Places Temp Tables for Private Browsing

Actually when private browsing starts it "freeze" Places, so that no new visits are added. In this state i could not use the awesomebar, the history menu or sidebar to see where i was 5 minutes ago or find a website i've closed. IDeally all Places features should continue working.

The private visits could be showed differently in the primary UI, maybe by putting a special private icon near them or changing their color to grey, or their background. They would be marked as private in the temp table through a special "private" column, on sync we would avoid syncing to disk "private" visits. On exiting private browsing we would clear all private items from the temp tables (they would so disappear from the UI)

  • PROS: all Places features would work and help user during private browsing.
  • CONS: Needs UI changes to clearly show that informations are added only "temporary", till PB is disabled.
  • EST: 1.5 weeks.
  • BUG:
  • DEPENDS:
  • STATUS: inactive

Add Advanced Search UI in Library

This has been disabled, should be re-enabled, or rethought (maybe a content page?)

  • PROS: allow users to create better personalized searches.
  • CONS:
  • EST: 2 weeks.
  • BUG: bug 436380
  • DEPENDS:
  • STATUS: inactive

Better handle large number of annotations

Google Toolbar has demonstrated that an extension heavily using annotations can easily bloat places.sqlite, they should either be saved in a separate attached db, or at least binary ones.

  • PROS: avoid bloating places.sqlite
  • CONS: harder to query
  • EST:
  • BUG:
  • DEPENDS:
  • STATUS: inactive

Enable full-text indexing

Being able to search in queries with full text indexing should give back good results, once we can vacuum.

  • PROS:
  • CONS:
  • EST:
  • BUG: bug 342913
  • DEPENDS:
  • STATUS: partly active
  • TEAM: adw

Revise Bookmarks and History sidebars

Sidebars UI is quite old, and needs rethought.

  • PROS: better UX.
  • CONS: moving away from consolidated way of interaction.
  • EST:
  • BUG:
  • DEPENDS:
  • STATUS: inactive

Revise Library UI

Library interaction is quite messed up and not perfect.

  • PROS: better UX.
  • CONS: needs lot of brainthinking
  • EST:
  • BUG:
  • DEPENDS:
  • STATUS: inactive

Allow to search folders in Places search fields and awesomebar

Allow searching and managing downloads in Library

Library was initially intended to manage not only history and bookmarks but also downloads, as personal data.

  • PROS: Central place to manage navigation data.
  • CONS: adds more bloatness to Library.
  • EST: 1,5 weeks.
  • BUG: bug 402231
  • DEPENDS:
  • STATUS: inactive

Further sprint ideas yet to be parsed

These are enhancements that need further thoughts or involve minor changes.

  • More reduction of writes and fsyncs
  • reduce query volume
  • get better tree views performance on large history trees
  • reduce copy/paste/delete operations weight bug 391836
  • awesomebar extensibility (frecency bonus, external providers, bug 399213)
  • tags in HTML export/import?
  • Pluggable views in Library based on collection type
  • Unified search in Library and Sidebars
  • make Most Visited better (show most visited typed uris)
  • support for browser.chrome.favicons (bug 331228)
Retrieved from "https://wiki.mozilla.org/index.php?title=Places:Plan&oldid=164996"