Places:Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(40 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Roadmap for future work on Places, the SQLite-driven Bookmarks and History system.
Evolution of future work on Places, the SQLite-driven Bookmarks and History system.


= Firefox 3.6 =
= Timeline =
== Database related ==
=== VACUUM Places database ===
Some user reports slow database performances, due to fragmentation, we should start investigate and implement Vacuum in Places, and in future make that app-wide.


* '''PROS:''' smaller and faster db.
Cherry-picked list of Places related fixes. Minor changes or cleanups are excluded.
* '''CONS:''' vacuuming too much can cause slower INSERTs, vacuum takes long time.
* '''EST:''' 1 week.
*'''BUG:''' {{bug|512854}}
* '''DEPENDS:'''
* '''STATUS:''' active
* '''TEAM:''' mak


=== Fix dynamic containers and convert livemark children to them ===
=== Firefox 5  ===
All queries involving bookmarks have to filter out livemarks children that are not real bookmarks, also checking if an uri is bookmarked requires a similar filtering. Moreover the new async locationbar is not correctly filtering these out. Plus livemarks service is slow in updating livemarks because bookmarks are sync written to disk.
Dynamic containers stuff can be useful for extensions.


* '''PROS:''' faster bookmarks queries, simpler bookmarks code. no more UI-blocking while updating livemarks (make livemark service as async as possible).
{| class="wikitable"
* '''CONS:''' Will require a separate db to hold livermarks children. Will need some API change to livemarks service to allow extension developers to access livemarks children, and add their own "annotations" on them.
! Bugs
* '''EST:''' 3 weeks.
! Type
*'''BUG:''' {{bug|453530}}
! Description
* '''DEPENDS:''' {{bug|457708}}
|-
* '''STATUS:''' active
| {{bug|660036}}
* '''TEAM:''' mak
| IO
| Workarounds a bug in Android < 2.3, that causes places.sqlite-wal to grows indefinitely on first Sync.
|-
| {{bug|649211}}
| SUPPORT
| Places Maintenance output was broken, not giving any feedback to users.
|-
| {{bug|636924}}
| UI
| Makes copied livemarks not appear as common folders in sidebar.
|-
| {{bug|599641}}
| PERF
| Runs ANALYZE after each expiration step. SQLite query planner uses this data.
|}


== Backend related ==
=== Firefox 6 ===
=== Find async query wins [p2] ===
Need to find points that require async love.


=== Tests re-enable/cleanup ===
{| class="wikitable"
We have a bunch of tests (also folders of tests) that have been disabled and never touched again, and bugs filed about queries tests containing tests that were never pushed due to minor issues that now could be fixed (or is worth fixing them).
! Bugs
! Type
! Description
|-
| {{bug|555474}}
| UI
| Avoids bookmark tooltips interrupting Drag'n'Drop actions.
|-
| {{bug|620627}}
| l10n
| Fixes localization of grouped history views
|-
| {{bug|652379}}
| API
| Fixes results for a Places query pointing to a wrong folder id.
|-
| {{bug|630240}}
| PERF
| Avoid full refreshes of history results to avoid locking the main-thread too long.
|-
| {{bug|359809}}
| UI
| Fixes keyword searches encoding in location bar
|-
| {{bug|639105}}
| UI
| History menu was ignoring some added visits
|-
| {{bug|656188}}
| PERF
| Caches last fetched bookmarks info to speed up requests.
|-
| {{bug|630622}}
| UI
| Updates the Library design to the browser Aero style.
|-
| {{bug|629620}}
| PERF
| Copied bookmarks won't inherit anymore all annotations, since they are new entitities.
|-
| {{bug|631374}}
| UI
| Fixes tags going out of view on toggle in bookmarks dialog.
|-
| {{bug|658135}}
| PERF
| SQlite should not create transactions for a batch of read-only statements (like Autocomplete)
|-
| {{bug|630225}}
| API
| Frecency can now be used to sort queries.
|-
| {{bug|524091}}
| API
| Microsummaries have been removed from the core product.
|-
| {{bug|633274}}
| API
| Bookmarks observers now pass out more information, included GUIDs.
|-
| {{bug|625325}}<br/>{{bug|625322}}
| UI
| Some menuitems in Bookmarks and History menu have been repositioned.
|-
| {{bug|663344}}
| SUPPORT
| Fixes a case where weekly maintenance may cause history loss.
|}


* '''PROS:''' larger test base at a relatively low cost.
=== Firefox 7 ===
* '''CONS:'''
* '''EST:''' 1 week.
*'''BUG:''' {{bug|509868}}
* '''DEPENDS:'''
* '''STATUS:''' active
* '''TEAM:''' mak


== Frontend related ==
{| class="wikitable"
! Bugs
! Type
! Description
|-
| {{bug|661135}}
| PERF
| Simplifies some autocomplete queries.
|-
| {{bug|633266}}<br/>{{bug|662806}}
| API
| Pass GUID to history observers. Helps Sync.
|-
| {{bug|661445}}<br/>{{bug|663269}}<br/>{{bug|663104}}
| PERF
| Various fixes to livemarks code.
|-
| {{bug|611568}}
| UI
| File/Import has been moved to the Library window.
|-
| {{bug|660109}}
| API
| Allow to distinguish expiration removals, to avoid history dataloss on Sync.
|-
| {{bug|416459}}
| UI
| Cut actions are now properly implemented. Fixes UI, saves some IO and has better Sync performances.
|-
| {{bug|578268}}
| CLEANUP
| MOZ_MORKREADER is no more in Firefox.
|-
| {{bug|593566}}
| SUPPORT
| Fixed a bug that caused broken bookmarks export.
|-
| {{bug|645963}}
| UI
| Text-only toolbar view now correctly shows the chevron in pinstripe.
|-
| {{bug|655270}}<br/>{{bug|655273}}
| UI
| history.pushstate causes missing title or icon
|}


= Firefox 3.7 =
=== Firefox 8 ===
== Database related ==
=== Move tags out of bookmarks table ===
Tags are actually bookmarks folders, so we have to duplicate bookmarks inside them, pay attention to not add containers/separators inside them, move bookmarks when merging/splitting two tags, filter out them from all bookmarks queries. Having tags in a separated table would make all of this much easier and faster. Apart performance, we also have bugs that are coming from our "fancy" tags implementation and should be workaround if we don't fix the implementation.


* '''PROS:''' faster bookmarks queries, simpler bookmarks code. faster autocomplete queries. no data duplication. easier merge/update tags.
{| class="wikitable"
* '''CONS:''' Will require a separate table to hold tags. Need to revise all tagging related code and tagging service.
! Bugs
* '''EST:''' 2 weeks.
! Type
* '''BUG:''' {{bug|424160}}
! Description
* '''DEPENDS:'''
! Status
* '''STATUS:''' inactive
|-
| {{bug|564900}}<br/>{{bug|591289}}<br/>{{bug|669905}}
| UI
| Integrate downloads in the Library
| IN PROGRESS
|-
| {{bug|671001}}<br/>
| Perf
| Places telemetry
| IN PROGRESS
|-
| {{bug|674210}}<br/>
| Perf
| New expiration heuristics (memshrink)
| IN PROGRESS
|-
| {{bug|658305}}<br/>
| Perf
| Limit journal for mobile
| IN PROGRESS
|-
| {{bug|669040}}<br/>
| Cleanup
| Remove Mork code
| IN PROGRESS
|-
| {{bug|636603}}<br/>
| API
| mozIAsyncHistory fixes
| IN PROGRESS
|}


=== Denormalize info into moz_places and revise PlacesSQLQueryBuilder ===
=== Future ===
Actually the query builder code is in history.cpp, but there is much related code out of it. Ideally it should just receive queries and options, and return an OPTIMIZED sql query. As it is code is confusing and returned queries are not always optimized.
We should flatten most informations in moz_places and rewrite the builder so that it will return better queries. For example it should not JOIN history table if we are not interested in querying history values (like querying by timeframe).


Actually we could just convert nsNavHistoryQuery.cpp to JS, and make the SQL query be a property of a query, we could provide an inclusion mode that will use: UNION, INTERSECT, EXCEPT...
{| class="wikitable"
the query system should be able to get a query string in places format, or json, or anything other, having a parser to convert it to an internal representation, and use that to build SQL queries in a de-normalized and performant way.
! Bugs
! Description
|-
|
| Schema changes
|-
|
| Async bookmarks API
|-
|
| Async annotations API
|-
|
| in-content UI
|-
|
| Finish UpdatePlaces
|-
|
| Audit Mobile API usage
|-
|
| [places-next-wanted]
|-
|
| Limit WAL Size
|-
|
| History backup
|-
|
| Memory usage reduction
|}


*'''PROS:''' more manageable and separated code between history and SQL. Easily to build more optimized queries (better performances for uri based queries).
= Schema Changes =
*'''CONS:''' this is a rewrite with some risk.
* '''GOAL:''' Database filesize reduction, performances, healthier management
*'''EST:''' 1.5 weeks.
* '''ETA:''' Firefox 7
*'''BUG:''' {{bug|500290}}
* '''LEADER:''' TBD
*'''DEPENDS:'''
* '''PRECONDITIONS:''' History backup
*'''STATUS:''' inactive


=== make GUIDs not annotations ===
=== Create history backups ===
This is a Weave need, we need to talk to them and understand what are the problems andhow we can fix them.
This is a precondition to stop caring about corruptions, downgrades, compatibility.
* '''PROPOSAL:''' per-host backups and expiration, csv-append-like.
* '''PRECONDITIONS''' out-of-mainthread file access? (append and remove)
* '''BUG:''' TBD


== Backend related ==
=== Coalesce titles across history and bookmarks ===
=== Make Expiration async and performant ===
Strings are the largest database offender, we never use both titles and all the queries have to coalesce titles. Users want us to update titles automatically when they change if they didn't manually set a different bookmark title.
Expiration currently runs on shutdown and on idle, if possible it should instead run in background and be async.  We should try to unify it with queries that run in nsPlacesDBFlush, so that we will write to disk the less often as possible. Expiration should be removed from history, and moved to a separate JS component for sake of simplifying history code and separating not completely related code. Expiration algorithm could even be reviewed a bit to make it simpler.
* '''PROPOSAL:''' keep only one title in moz_places, track whether it's expirable.
* '''BUG:''' TBD


* '''PROS:''' no UI-blocking or standby blocking expiration. Less disk writes.  easier to manage code.
=== Tags out of moz_bookmarks ===
* '''CONS:''' It's a rewrite, with some possible privacy risk.
Tags are actually folders, this is unefficient and takes a lot of filesize for nothing, searches are slow as well since subqueries suffer from missing indices.
* '''EST:''' 2 weeks.
* '''PROPOSAL:''' tags in a separate relational table.
* '''BUG:'''  
* '''BUG:''' TBD
* '''DEPENDS:'''
* '''STATUS:''' inactive


=== Asynchronous isVisited ===
=== Livemarks out of moz_bookmarks ===
This is needed for Electrolysis and Tsnappiness, isvisited sould get data async.
Livemarks are actually folders of temporary bookmarks, this is unefficient and causes unneeded database fragmentation.
* '''PROS:''' no UI-blocking and Tp wins.
* '''PROPOSAL:''' redesign from scratch.
* '''CONS:'''
* '''BUG:''' TBD
* '''EST:'''
* '''BUG:''' {{bug|511960}}
* '''DEPENDS:'''
* '''STATUS:''' active
* '''TEAM:''' sdwilsh


=== Open containers asynchronously ===
=== Remove dynamic containers support ===
Opening containers is slowing down and blocking UI, it should be done async.  
They never worked, nuke them.
* '''PROPOSAL:''' Remove all related code, redesign from scratch.
* '''BUG:''' TBD


*'''PROS:''' UI responsiveness.  
=== Kill moz_bookmarks table ===
*'''CONS:''' Can cause views bugs, needs to move back some code to the backend.  
Once titles are flattened to moz_places and dynamic containers killed, most of bookmarks data consists of hierarchical information, then just make them so.  
*'''EST:'''
* '''PROPOSAL:''' Flatten isBookmark tracking in moz_places, have tags and folders just be hierarchical descriptions of these entries.
*'''BUG:''' {{bug|490714}}
* '''PRECONDITIONS:''' flattened bookmarks in moz_places, tags out of bookmarks.
*'''DEPENDS:'''
* '''BUG:''' TBD
*'''STATUS:''' active
*'''TEAM:''' adw


== Frontend related ==
=== Re-evaluate history tracking information ===
*      cut and paste is slow
We track each single step in the history visit chain, but these data are not really useful as of today, the UI mostly cares about the destination page, not about the full redirects chain, and walking it up is slow and unefficient.
*      dnd is slow
Sessions tracking has been removed, and adds useless data. Sync also missed some of these data.
*     delayed bookmarks init (delayed toolbar init, like chrome)
* '''PROPOSAL:''' Simplify, remove redundant information, track only what matters.
*     history menu (cache w/ json? live update?)
* '''BUG:''' TBD


= Firefox 4.0 =
=== Keywords are unefficient ===
== Database related ==
Currently these are a mostly-null column in moz_bookmarks, we should allow a 1_keyword->n_uris but not the opposite.
=== Convert bookmarks table to a preordered nested tree ===
* '''PROPOSAL:''' Convert keywords to annotations? they are already being lazily cached by bookmarks service for performance and IO.
Nested Trees are a good way to manage a hierachy in sql, you can most likely select anything with one query, and no recursion is needed at all. So you can with one query know if a node is descendant or ancestor of another one. This will allow to print out bookmarks paths in the UI, search for folders, check ancestors/descendant with better performances and so on.
* '''BUG:''' TBD
* '''PRECONDITIONS:''' Better annotations management


*'''PROS:''' allows to completely avoid recursion when querying descendant/ancestors of a bookmark. fast children count. Self protection from creating cycles of bookmarks.
=== Annotations are unefficient ===
*'''CONS:''' Needs schema changes (2 or 3 columns in moz_bookmarks). INSERTS/UPDATES/DELETES could be a bit slower and harder (triggers could help for complexity).  
Most of the annotations are simple boolean stuff, so they could be categorized as: dataless, with-data. Dataless annotations shouldn't take all the space they are today.
*'''EST:''' 2 weeks.
With-data annotations on the other side allow for binary data, but that data should NOT be stored in the database, let's allow the filesystem to handle them.
*'''BUG:''' {{bug|408991}}
Side notes: Having separate page and item annotations is confusing for both API and code, unfortunately we allow to annotate folders and this could be problematic to flatten. Could favicons become binary annotations?
*'''DEPENDS:''' {{bug|424160}}, {{bug|453530}}
* '''PROPOSAL:''' Split dataless annotations to save space and get better performances.
*'''STATUS:''' inactive
* '''BUG:''' TBD


=== Denormalize referrer in moz_places table ===
=== Evaluate Microsummaries removal ===
Actually from_visit reports the page we have come from, so in case of a double redirect we would have to recurse twice to go to the original site. It would be nice having a new column reporting a sort of referer, or a new table schema allowing to query without recursion.
They are our largest js component, but only a few users use them, and never take off completely as a standard.
We could ideally reuse session column (that is actually minused) to track redirects, querying first visit with the same session would give back a somehow correct referrer.
* '''PROPOSAL:''' Move microsummaries to an add-on.
* '''BUG:''' {{bug|524091}}


*'''PROS:''' Easily query for referer and visit-chain, avoiding recursive queries. Useful for extsnsions and security checks (how did i come here?)
=== Evaluate bookmarks roots removal ===
*'''CONS:''' Needs schema changes.
Roots caused more bugs than benefits, UI can better use annotated folders in a simple 1-root hierarchy.
*'''EST:''' 1,5 weeks.
* '''PROPOSAL:''' Annotate special folders, 1-root hierarchy
*'''BUG:''' {{bug|468710}}
* '''BUG:''' TBD
*'''DEPENDS:'''
*'''STATUS:''' inactive


=== remove temp tables ===
=== Change keywords schema to drop multiple keywords support ===
Keywords live in a separated table, but they are linked through a keyword_id column in moz_bookmarks, this is often null, and that's not good since it will take a lot of useless space in the db We should instead have a (keyword, item_id) table, and no column in moz_bookmarks.


*'''PROS:''' less wasted space in the db.
= Async APIs =
*'''CONS:''' Needs hard schema change.
* '''GOAL:''' No UI-blocking tasks
*'''EST:'''
* '''ETA:''' Firefox 7
*'''BUG:'''
* '''LEADER:''' TBD
*'''DEPENDS:'''
*'''STATUS:''' inactive


=== figure out why annotations are slow (weave) ===
=== Simpler and async querying API in js ===
The classical querying API is complicated by xpcom, we need a deCOM version of it, this was partially done for Jetpack, must be finished.
* ''' PROPOSAL:''' Finish the work started for Jetpack, with a js module.
* ''' BUG:''' TBD


== Backend related ==
=== Finish async favicons service ===
=== Notifications should give more informations to the observers ===
Favicons service is already partially async, but the work has to be completed on all APIs.
Our actual notifications don't tell anything to the observer, it will have to query again to know informations that the notifier already has. For example if i get onItemAdded, i have to query bookmarks service to know which kind of item, and at what time it was added.
* ''' PROPOSAL:''' Make all interface methods async.
Having better notifications would reduce queries load and make observers more performant.
* ''' BUG:''' TBD


*'''PROS:''' less queries. faster observers.
=== Finish async containerOpen ===
*'''CONS:''' Compatibility with extensions.
Folders work, but it was never used live.
*'''EST:'''
* ''' PROPOSAL:''' Evaluate if this is still the wanted approach.
*'''BUG:''' {{bug|494380}}
* ''' BUG:''' TBD
*'''DEPENDS:''' {{bug|498130}}
*'''STATUS:''' active
*'''TEAM:''' mano, mak


*      non crappy notifications (avoid removed/added on index changes)
=== Async bookmarks API ===
*      reduce observer traffic
Largely wanted for Sync and mobile.
*      async addBookmark
* ''' PROPOSAL:''' split a new interface, gather from what we did for history.
*      async addVisit
* ''' BUG:''' TBD
=== Remove LAZY_ADD, use async storage instead ===
LAZY_ADD is the old workaround to implement an async addition of visits and favicons, we should get rid of it and use async storage instead.


* '''PROS:''' less code bloat and UI-blocking.
= Better Notifications =
* '''CONS:''' Views will be updated faster, will that be an issue? We are just moving work earlier.
* '''GOAL:''' More performant observers
* '''EST:''' 1 week.
* '''ETA:''' Firefox 5
* '''BUG:''' {{bug|499828}}, {{bug|429989}}
* '''LEADER:''' TBD
* '''DEPENDS:'''  
* '''STATUS:''' inactive


== Frontend related ==
=== Avoid useless notifications traffic  ===
=== investigate use of templates ===
Just changing position could sometimes fire a removed and a added notifications.
* ''' PROPOSAL:''' Reduce observers traffic.
* ''' BUG:''' TBD


= Untargeted stuff =
= In-content UI =
* '''GOAL:''' Better user experience
* '''ETA:''' Firefox 7
* '''LEADER:''' TBD


=== Review Places views drag & drop code ===
=== Thumbnails service ===
Places drag & drop code needs some further cleaning: menus code should be cleaned up to just use new d&d API, trees code should be fixed since trees are now partially using new d&d API, we should support more natural dragging through menus (diagonal dragging).
Any in content UI will probably want pages screenshots, this issue can be coalesced with Sync needs, also look at other vendors suggestions for speed dial thumbs.
* ''' PROPOSAL:''' Allow web authors to specify their own thumbs or make a new one with a tentative algo.
* ''' BUG:''' TBD


*'''PROS:''' more natural D&D experience for users. cleaner code.
=== Breadcrumb navigation ===
*'''CONS:''' no deep tests are possible, regressions risk.
As in most today-ish OS, simplify navigation in the Places hierarchies.
*'''EST:''' 1,5 weeks.
* ''' PROPOSAL:''' History or Bookmarks breadcrumbs.
*'''BUG:''' {{bug|419911}}
* ''' BUG:''' TBD
*'''DEPENDS:'''
*'''STATUS:''' inactive


=== Detect real database corruption (integrity_check) ===
=== Drag&Drop chrome-content ===
Actually we don't detect real database file corruptions because it's slow, we should still, because there is no easy path to bring users out of it. Better if done at a browser/app wide level.
D&D code needs to be simplified and allow interaction with in-content UI.
* ''' PROPOSAL:''' TBD
* ''' BUG:''' TBD


*'''PROS:''' allow users to recover data
=== Kill the Library ===
*'''CONS:''' slow
Should be moved in-content, long treeviews should be paged.
*'''EST:'''
* ''' PROPOSAL:''' TBD
*'''BUG:''' {{bug|506905}}
* ''' BUG:''' TBD
*'''DEPENDS:'''
*'''STATUS:''' inactive


=== Places performance tests ===
=== Donwloads in Places ===
Tests and graphs for UI performance: microbenchmarking, dirty profile testing, etc. Needs to hook into Talos for continuous integration, and Perfomatic for reporting.
Move downloads manager to a toolbar widget, and allow to track and manage all downloads in the in-content UI.
* ''' PROPOSAL:''' TBD
* ''' BUG:''' TBD


*'''PROS:'''
=== Better search-ability ===
*'''CONS:'''
We should allow the user to filter results from history and bookmarks better, also to allow more granular privacy removals.
*'''EST:'''
* ''' PROPOSAL:''' Revised in-content search UI
*'''BUG:'''  
* ''' BUG:''' TBD
*'''DEPENDS:'''
*'''STATUS:''' partly active
*'''TEAM:''' ddahl, alice, dietrich


=== Create a Places Page-Thumbnails service ===
This would be useful in various parts of the UI and good for extensions. Some extensions are reimplementing this, and if not done correctly this will hurt us badly (See old Google Toolbar for example, putting all thumbnails in annotations, and causing places.sqlite to bloat hundred of MBs)


*'''PROS:''' Usable in library, and many other points of access to bookmarks/history, maybe in tooltips too. Built to be async. Extensions that need thumbnails won't hurt us.
= Fulltext indexing =
*'''CONS:''' will use some disk space, privacy concerns.
* '''GOAL:''' No tables scan on search
*'''EST:''' 2 weeks.
* '''ETA:''' TBD
*'''BUG:''' {{bug|497543}}
* '''LEADER:''' TBD
*'''DEPENDS:'''
*'''STATUS:''' inactive


=== Queries improvements ===
=== Find a good tokenizer ===
Support querying for the exact data set that the location bar can, support querying on transition type, implement RESULTS_AS_FULL_VISIT or put everything needed into a visit node and deprecate FULL_VISIT, expose frecency.
The pages case is special, since the same profile can have visits in different locales, the tokenizer should be per URI. LibICU is nice but huge.
* ''' PROPOSAL:''' TBD
* ''' BUG:''' TBD


*'''PROS:''' better queries, allow for further enhancements.
=== Measure perf win/loss ===
*'''CONS:'''
FTS can give nice boosts, but also perf loss in some case, have to figure out the right compromise.
*'''EST:''' 2 weeks.
* ''' PROPOSAL:''' TBD
*'''BUG:'''  
* ''' BUG:''' TBD
*'''DEPENDS:'''
*'''STATUS:''' inactive


=== Better JSON-friendly querying API ===
= Sync Needs =
Querying Places is hard, having a better JSON API would be cool, this is actually under experimentation in JetPack, but we would like to uplift that.
* Async bookmarks ({{bug|519514}}, see also https://wiki.mozilla.org/Places/AsyncAPIsForSync#nsIBookmarkInfo)
* use GUIDs in queries (folder=guid), in json and html backups
* finish UpdatePlaces (see dependents of {{bug|606966}})
* use url strings not nsIURI ({{bug|636393}})


*'''PROS:''' easier to query for extensions.
= Mobile Needs =
*'''CONS:'''
* WAL size: force checkpoint
*'''EST:'''
* Code audit for bad API usage
*'''BUG:'''
* Async Bookmarks
*'''DEPENDS:'''
* Database size reduction
*'''STATUS:''' active
* Memory cache reduction
*'''TEAM:''' ddahl
 
=== Provide extended API to insert or get bookmark properties ===
Transaction manager, sync services and restore/import have to insert bookmarks and then set various additional properties. We also do not provide a way to get back all properties for a bookmark with a single call. This causes more queries traffic and notifications than needed. Would be cool to provide insertBookmarkExt and getItemProperties to allow setting getting all properties at once.
 
*'''PROS:''' faster bulk inserting and reading
*'''CONS:''' API noise
*'''EST:''' 1 week.
*'''BUG:''' {{bug|473666}}, {{bug|473664}}
*'''DEPENDS:'''
*'''STATUS:''' inactive
 
=== Fix Cut in Places frontend, to avoid dataloss ===
Actually Cut is doing Remove&Copy, this means to undo it user will have to undo twice, and after a Cut if user closes browser or it crashes the cutted elements will be lost.
 
*'''PROS:''' no dataloss, single action undo
*'''CONS:'''
*'''EST:''' 2 days.
*'''BUG:''' {{bug|484578}}
*'''DEPENDS:'''
*'''STATUS:''' inactive
 
=== Make bookmarks service initialization fault tolerant and faster ===
Check roots are in a good shape on start, but only do that when we are sure something has changed in the db.
 
*'''PROS:''' avoid bloating places.sqlite
*'''CONS:''' harder to query
*'''EST:'''
*'''BUG:''' {{bug|478912}}
*'''DEPENDS:'''
*'''STATUS:''' active
*'''TEAM:''' adw
 
=== deCOM internal bookmarks and history observers ===
Avoid xpcom overhead, use results globally to observe changes.
 
*'''PROS:''' less queries. faster observers.
*'''CONS:''' Compatibility with extensions.
*'''EST:'''
*'''BUG:''' {{bug|497608}}
*'''DEPENDS:''' {{bug|497387}}, {{bug|497389}},  {{bug|497605}},  {{bug|497606}},  {{bug|497607}}
*'''STATUS:''' active
*'''TEAM:''' mano
 
=== Add UI for quick tagging ===
Quick tagging UI through keyword.
 
*'''PROS:''' faster tagging for heavy taggers.
*'''CONS:''' discoverability?
*'''EST:''' practically done
*'''BUG:''' {{bug|434946}}
*'''DEPENDS:'''
*'''STATUS:''' active
*'''TEAM:''' dietrich
 
=== Add hooks for extensions in Library ===
Library needs to provide hooks for extensions content. Maybe a specific root where they can put dynamic containers or personalized content. There should be some sort of "registration" for both content and special views.
 
*'''PROS:''' easier for extensions to provide their own views for the Library
*'''CONS:''' adds complexity to the Library, Library perf/bloat.
*'''EST:'''
*'''BUG:'''
*'''DEPENDS:'''
*'''STATUS:''' inactive
 
=== Use Places Temp Tables for Private Browsing ===
Actually when private browsing starts it "freeze" Places, so that no new visits are added. In this state i could not use the awesomebar, the history menu or sidebar to see where i was 5 minutes ago or find a website i've closed. IDeally all Places features should continue working.
 
The private visits could be showed differently in the primary UI, maybe by putting a special private icon near them or changing their color to grey, or their background. They would be marked as private in the temp table through a special "private" column, on sync we would avoid syncing to disk "private" visits. On exiting private browsing we would clear all private items from the temp tables (they would so disappear from the UI)
 
*'''PROS:''' all Places features would work and help user during private browsing.
*'''CONS:''' Needs UI changes to clearly show that informations are added only "temporary", till PB is disabled.
*'''EST:''' 1.5 weeks.
*'''BUG:'''
*'''DEPENDS:'''
*'''STATUS:''' inactive
 
=== Add Advanced Search UI in Library ===
This has been disabled, should be re-enabled, or rethought (maybe a content page?)
 
*'''PROS:''' allow users to create better personalized searches.
*'''CONS:'''
*'''EST:''' 2 weeks.
*'''BUG:''' {{bug|436380}}
*'''DEPENDS:'''
*'''STATUS:''' inactive
 
=== Better handle large number of annotations ===
Google Toolbar has demonstrated that an extension heavily using annotations can easily bloat places.sqlite, they should either be saved in a separate attached db, or at least binary ones.
 
*'''PROS:''' avoid bloating places.sqlite
*'''CONS:''' harder to query
*'''EST:'''
*'''BUG:'''
*'''DEPENDS:'''
*'''STATUS:''' inactive
 
=== Enable full-text indexing ===
Being able to search in queries with full text indexing should give back good results, once we can vacuum.
 
*'''PROS:'''
*'''CONS:'''
*'''EST:'''
*'''BUG:''' {{bug|342913}}
*'''DEPENDS:'''
*'''STATUS:''' partly active
*'''TEAM:''' adw
 
=== Revise Bookmarks and History sidebars ===
Sidebars UI is quite old, and needs rethought.
 
*'''PROS:''' better UX.
*'''CONS:''' moving away from consolidated way of interaction.
*'''EST:'''
*'''BUG:'''
*'''DEPENDS:'''
*'''STATUS:''' inactive
 
=== Revise Library UI ===
Library interaction is quite messed up and not perfect.
 
*'''PROS:''' better UX.
*'''CONS:''' needs lot of brainthinking
*'''EST:'''
*'''BUG:'''
*'''DEPENDS:'''
*'''STATUS:''' inactive
 
=== Allow to search folders in Places search fields and awesomebar ===
 
*'''PROS:''' better search UX
*'''CONS:'''
*'''EST:'''
*'''BUG:''' {{bug|469424}}, {{bug|406157}}
*'''DEPENDS:'''
*'''STATUS:''' inactive
 
=== Allow searching and managing downloads in Library ===
Library was initially intended to manage not only history and bookmarks but also downloads, as personal data.
 
*'''PROS:''' Central place to manage navigation data.
*'''CONS:''' adds more bloatness to Library.
*'''EST:''' 1,5 weeks.
*'''BUG:''' {{bug|402231}}
*'''DEPENDS:'''
*'''STATUS:''' inactive
 
== Further sprint ideas yet to be parsed ==
These are enhancements that need further thoughts or involve minor changes.


= To-be-parsed stuff =
* investigate use of templates
* more natural dragging through menus (diagonal dragging)
* Places performance tests
* Fix Cut in frontend, to avoid dataloss
* deCOM internal bookmarks and history observers
* Add UI for quick tagging
* Allow to search folders
* More reduction of writes and fsyncs
* More reduction of writes and fsyncs
* reduce query volume
* reduce query volume
* get better tree views performance on large history trees
* awesomebar extensibility (frecency bonus, external providers, ...)
* reduce copy/paste/delete operations weight {{bug|391836}}
* awesomebar extensibility (frecency bonus, external providers, {{bug|399213}})
* tags in HTML export/import?
* tags in HTML export/import?
* Pluggable views in Library based on collection type
* Pluggable views
* Unified search in Library and Sidebars
* make Most Visited better (show most visited typed uris)
* make Most Visited better (show most visited typed uris)
* support for browser.chrome.favicons ({{bug|331228}})
* support for browser.chrome.favicons ({{bug|331228}})
* deCOM feedParser

Latest revision as of 05:27, 1 November 2011

Evolution of future work on Places, the SQLite-driven Bookmarks and History system.

Timeline

Cherry-picked list of Places related fixes. Minor changes or cleanups are excluded.

Firefox 5

Bugs Type Description
bug 660036 IO Workarounds a bug in Android < 2.3, that causes places.sqlite-wal to grows indefinitely on first Sync.
bug 649211 SUPPORT Places Maintenance output was broken, not giving any feedback to users.
bug 636924 UI Makes copied livemarks not appear as common folders in sidebar.
bug 599641 PERF Runs ANALYZE after each expiration step. SQLite query planner uses this data.

Firefox 6

Bugs Type Description
bug 555474 UI Avoids bookmark tooltips interrupting Drag'n'Drop actions.
bug 620627 l10n Fixes localization of grouped history views
bug 652379 API Fixes results for a Places query pointing to a wrong folder id.
bug 630240 PERF Avoid full refreshes of history results to avoid locking the main-thread too long.
bug 359809 UI Fixes keyword searches encoding in location bar
bug 639105 UI History menu was ignoring some added visits
bug 656188 PERF Caches last fetched bookmarks info to speed up requests.
bug 630622 UI Updates the Library design to the browser Aero style.
bug 629620 PERF Copied bookmarks won't inherit anymore all annotations, since they are new entitities.
bug 631374 UI Fixes tags going out of view on toggle in bookmarks dialog.
bug 658135 PERF SQlite should not create transactions for a batch of read-only statements (like Autocomplete)
bug 630225 API Frecency can now be used to sort queries.
bug 524091 API Microsummaries have been removed from the core product.
bug 633274 API Bookmarks observers now pass out more information, included GUIDs.
bug 625325
bug 625322 		UI Some menuitems in Bookmarks and History menu have been repositioned.
bug 663344 SUPPORT Fixes a case where weekly maintenance may cause history loss.

Firefox 7

Bugs Type Description
bug 661135 PERF Simplifies some autocomplete queries.
bug 633266
bug 662806 		API Pass GUID to history observers. Helps Sync.
bug 661445
bug 663269
bug 663104 		PERF Various fixes to livemarks code.
bug 611568 UI File/Import has been moved to the Library window.
bug 660109 API Allow to distinguish expiration removals, to avoid history dataloss on Sync.
bug 416459 UI Cut actions are now properly implemented. Fixes UI, saves some IO and has better Sync performances.
bug 578268 CLEANUP MOZ_MORKREADER is no more in Firefox.
bug 593566 SUPPORT Fixed a bug that caused broken bookmarks export.
bug 645963 UI Text-only toolbar view now correctly shows the chevron in pinstripe.
bug 655270
bug 655273 		UI history.pushstate causes missing title or icon

Firefox 8

Bugs Type Description Status
bug 564900
bug 591289
bug 669905 		UI Integrate downloads in the Library IN PROGRESS
bug 671001
 Perf Places telemetry IN PROGRESS
bug 674210
 Perf New expiration heuristics (memshrink) IN PROGRESS
bug 658305
 Perf Limit journal for mobile IN PROGRESS
bug 669040
 Cleanup Remove Mork code IN PROGRESS
bug 636603
 API mozIAsyncHistory fixes IN PROGRESS

Future

Bugs Description
Schema changes
Async bookmarks API
Async annotations API
in-content UI
Finish UpdatePlaces
Audit Mobile API usage
[places-next-wanted]
Limit WAL Size
History backup
Memory usage reduction

Schema Changes

  • GOAL: Database filesize reduction, performances, healthier management
  • ETA: Firefox 7
  • LEADER: TBD
  • PRECONDITIONS: History backup

Create history backups

This is a precondition to stop caring about corruptions, downgrades, compatibility.

  • PROPOSAL: per-host backups and expiration, csv-append-like.
  • PRECONDITIONS out-of-mainthread file access? (append and remove)
  • BUG: TBD

Coalesce titles across history and bookmarks

Strings are the largest database offender, we never use both titles and all the queries have to coalesce titles. Users want us to update titles automatically when they change if they didn't manually set a different bookmark title.

  • PROPOSAL: keep only one title in moz_places, track whether it's expirable.
  • BUG: TBD

Tags out of moz_bookmarks

Tags are actually folders, this is unefficient and takes a lot of filesize for nothing, searches are slow as well since subqueries suffer from missing indices.

  • PROPOSAL: tags in a separate relational table.
  • BUG: TBD

Livemarks out of moz_bookmarks

Livemarks are actually folders of temporary bookmarks, this is unefficient and causes unneeded database fragmentation.

  • PROPOSAL: redesign from scratch.
  • BUG: TBD

Remove dynamic containers support

They never worked, nuke them.

  • PROPOSAL: Remove all related code, redesign from scratch.
  • BUG: TBD

Kill moz_bookmarks table

Once titles are flattened to moz_places and dynamic containers killed, most of bookmarks data consists of hierarchical information, then just make them so.

  • PROPOSAL: Flatten isBookmark tracking in moz_places, have tags and folders just be hierarchical descriptions of these entries.
  • PRECONDITIONS: flattened bookmarks in moz_places, tags out of bookmarks.
  • BUG: TBD

Re-evaluate history tracking information

We track each single step in the history visit chain, but these data are not really useful as of today, the UI mostly cares about the destination page, not about the full redirects chain, and walking it up is slow and unefficient. Sessions tracking has been removed, and adds useless data. Sync also missed some of these data.

  • PROPOSAL: Simplify, remove redundant information, track only what matters.
  • BUG: TBD

Keywords are unefficient

Currently these are a mostly-null column in moz_bookmarks, we should allow a 1_keyword->n_uris but not the opposite.

  • PROPOSAL: Convert keywords to annotations? they are already being lazily cached by bookmarks service for performance and IO.
  • BUG: TBD
  • PRECONDITIONS: Better annotations management

Annotations are unefficient

Most of the annotations are simple boolean stuff, so they could be categorized as: dataless, with-data. Dataless annotations shouldn't take all the space they are today. With-data annotations on the other side allow for binary data, but that data should NOT be stored in the database, let's allow the filesystem to handle them. Side notes: Having separate page and item annotations is confusing for both API and code, unfortunately we allow to annotate folders and this could be problematic to flatten. Could favicons become binary annotations?

  • PROPOSAL: Split dataless annotations to save space and get better performances.
  • BUG: TBD

Evaluate Microsummaries removal

They are our largest js component, but only a few users use them, and never take off completely as a standard.

  • PROPOSAL: Move microsummaries to an add-on.
  • BUG: bug 524091

Evaluate bookmarks roots removal

Roots caused more bugs than benefits, UI can better use annotated folders in a simple 1-root hierarchy.

  • PROPOSAL: Annotate special folders, 1-root hierarchy
  • BUG: TBD


Async APIs

  • GOAL: No UI-blocking tasks
  • ETA: Firefox 7
  • LEADER: TBD

Simpler and async querying API in js

The classical querying API is complicated by xpcom, we need a deCOM version of it, this was partially done for Jetpack, must be finished.

  • PROPOSAL: Finish the work started for Jetpack, with a js module.
  • BUG: TBD

Finish async favicons service

Favicons service is already partially async, but the work has to be completed on all APIs.

  • PROPOSAL: Make all interface methods async.
  • BUG: TBD

Finish async containerOpen

Folders work, but it was never used live.

  • PROPOSAL: Evaluate if this is still the wanted approach.
  • BUG: TBD

Async bookmarks API

Largely wanted for Sync and mobile.

  • PROPOSAL: split a new interface, gather from what we did for history.
  • BUG: TBD

Better Notifications

  • GOAL: More performant observers
  • ETA: Firefox 5
  • LEADER: TBD

Avoid useless notifications traffic

Just changing position could sometimes fire a removed and a added notifications.

  • PROPOSAL: Reduce observers traffic.
  • BUG: TBD

In-content UI

  • GOAL: Better user experience
  • ETA: Firefox 7
  • LEADER: TBD

Thumbnails service

Any in content UI will probably want pages screenshots, this issue can be coalesced with Sync needs, also look at other vendors suggestions for speed dial thumbs.

  • PROPOSAL: Allow web authors to specify their own thumbs or make a new one with a tentative algo.
  • BUG: TBD

Breadcrumb navigation

As in most today-ish OS, simplify navigation in the Places hierarchies.

  • PROPOSAL: History or Bookmarks breadcrumbs.
  • BUG: TBD

Drag&Drop chrome-content

D&D code needs to be simplified and allow interaction with in-content UI.

  • PROPOSAL: TBD
  • BUG: TBD

Kill the Library

Should be moved in-content, long treeviews should be paged.

  • PROPOSAL: TBD
  • BUG: TBD

Donwloads in Places

Move downloads manager to a toolbar widget, and allow to track and manage all downloads in the in-content UI.

  • PROPOSAL: TBD
  • BUG: TBD

Better search-ability

We should allow the user to filter results from history and bookmarks better, also to allow more granular privacy removals.

  • PROPOSAL: Revised in-content search UI
  • BUG: TBD


Fulltext indexing

  • GOAL: No tables scan on search
  • ETA: TBD
  • LEADER: TBD

Find a good tokenizer

The pages case is special, since the same profile can have visits in different locales, the tokenizer should be per URI. LibICU is nice but huge.

  • PROPOSAL: TBD
  • BUG: TBD

Measure perf win/loss

FTS can give nice boosts, but also perf loss in some case, have to figure out the right compromise.

  • PROPOSAL: TBD
  • BUG: TBD

Sync Needs

Mobile Needs

  • WAL size: force checkpoint
  • Code audit for bad API usage
  • Async Bookmarks
  • Database size reduction
  • Memory cache reduction

To-be-parsed stuff

  • investigate use of templates
  • more natural dragging through menus (diagonal dragging)
  • Places performance tests
  • Fix Cut in frontend, to avoid dataloss
  • deCOM internal bookmarks and history observers
  • Add UI for quick tagging
  • Allow to search folders
  • More reduction of writes and fsyncs
  • reduce query volume
  • awesomebar extensibility (frecency bonus, external providers, ...)
  • tags in HTML export/import?
  • Pluggable views
  • make Most Visited better (show most visited typed uris)
  • support for browser.chrome.favicons (bug 331228)
  • deCOM feedParser
Retrieved from "https://wiki.mozilla.org/index.php?title=Places:Plan&oldid=362849"