View source for Places/StatusMeetings/2006-10-26 and 2006-10-27

<small>[[Places/StatusMeetings/2006-10-19| « previous week]] | [[Places/StatusMeetings|index]] | [[Places/StatusMeetings/2006-11-09|next week »]]</small>

== Places meeting: 2006-10-26 4pm PST ==

 sspitzer       thunder / myk / dietrich / et al:  meeting time?
 thunder        hola
 dietrich       howdy all
 myk            sspitzer: yes, i'm here
 dietrich       sspitzer: i'll try out the new patch on bug 356487 asap
 dietrich       anyone have a chance to check out the ERD i sent out?
 sspitzer       dietrich:  yes
 thunder        yeah
 sspitzer       I had a question
 myk            dietrich: one more naming suggestion is to call the moz_anno_names table moz_attributes, since the moz_annos table is an entity-attribute-value (EAV) table, and moz_anno_names stores the attributes that the moz_annos table uses
 dietrich       myk: moz_attributes sounds kinda generic.. moz_anno_names is specific to annotations
 dietrich       moz_anno_attributes?
 myk            dietrich: sure
 sspitzer       my question was also about moz_annos
 sspitzer       and this is not new to your ERD, but:
 sspitzer       are moz_annos required to have a moz_anno_name?
 sspitzer       can we have un-nammed annotations?
 sspitzer       I think it should be a "may have a " relationship
 myk            dietrich: i generally prefer one-word names, but perhaps expressiveness is better than brevity in this case
 sspitzer       instead of a "has a"
 myk            sspitzer: they are required to have a name
 dietrich       myk: we could remove the moz_* prefix from all the tables
 dietrich       sspitzer: i'm not sure what the use case for un-typed annotations is
 myk            dietrich: yeah, i've thought about that as well, but it makes some sense to have it given that sqlite doesn't support namespaces or cross-db queries
 sspitzer       dietrich:  here's my use case
 myk            dietrich: the use case i usually consider is that multiple extensions want to store some related data in the database; we don't want them to create colliding tables (neither colliding with us nor with each other)...
 dietrich       myk: yeah, prefixing to avoid collisions makes sense
 myk            dietrich: large "enterprise" RDBMSes use namespaces to accomplish this; medium-scale RDBMSes like MySQL allow you to join across databases, effectively making databases be namespaces
 myk            dietrich: incidentally, i dislike prefixes too and think they are often used unnecessarily, but in this case i think they make sense
 dietrich       myk: yeah, the lack of cross-db queries means extensions are likely to add to our db as opposed to creating their own
 dietrich       ugh
 myk            dietrich: yeah :-)
 sspitzer       well, I was thinking about extension developers, who might use annotations in ways other than tagging (which is how I have been thinking about them, mostly).  in ways where all annotations are of the same type, but to play nice with other extensions and our own code, they might need a moz_anno_name, like "extension_xyz", so maybe my use case is wrong.
 dietrich       sspitzer: yeah, anno_names are basically being used as a loose typing system
 sspitzer       ok, then ignore my question.
 myk            dietrich: hmm, actually it looks like i'm wrong; sqlite does support joining across databases using the ATTACH DATABASE syntax
 myk            although "There is a compile-time limit of 10 attached database files."
 thunder        it must be fairly inefficient if they do that
 myk            (http://www.sqlite.org/lang_attach.html)
 thunder        s/do that/have that restriction/
 sspitzer       speaking of SQLite, I think thunder is working on updating to a new version, one with full text indexing.
 thunder        oh; yes
 thunder        I have a patch waiting to be reviewed by vlad
 thunder        (well, and committed by vlad, since I don't have an account)
 dietrich       myk: thx for the link. yeah we should encourage extension devs to use external dbs :)
 dietrich       thunder: cool, is there a bug for that?
 myk            dietrich: yeah, you're right, we really should; i wonder what it means for our prefixes
 thunder        yeah, looking
 thunder        just a sec
 thunder        341137
 thunder        https://bugzilla.mozilla.org/show_bug.cgi?id=341137
 myk            sspitzer: brett encouraged third parties to use namespaces in the annotation names to avoid collisions
 myk            sspitzer: f.e., in my implementation of microsummaries on top of places, i used microsummary/ as the namespace, f.e. the annotation name for the generator URI is microsummary/generator_uri (or something like that; would have to look at source to recall the exact name)
 dietrich       myk: if we can't restrict table creation in our db, then we should keep the prefixes
 sspitzer       myk:  ok, that makes sense.  thanks for the background info.
 myk            dietrich: i think i concur, especially given that compile-time limit to the number of other databases one can attach
 thunder        dietrich: my patch doesn't enable the text searching stuff
 thunder        (yet)
 thunder        I thought I'd get this reviewed first, then add the interface to mozStorage
 dietrich       fyi: i sent mail to todd agulnick, asking for him to take a look at the proposed changes
 dietrich       i'll hit up the yahoo people also
 thunder        cool
 thunder        um, do we have an agenda of some sort?
 dietrich       thunder: i think just to report any progress on the task list, and then any specific issues people want to bring up
 thunder        ok,
 thunder        I'm in the process of making a couple of new tests for tinderbox
 thunder        TpMH (medium history) and TpLH (large history)
 thunder        they are basically Tp, but I copy in a history file
 thunder        however, I don't have good history files to copy in :)
 dietrich       thunder: QA might be able to help getting history files
 thunder        I'm going to shoot for ~1MB for the medium and ~10MB for the large
 thunder        ah good call
 thunder        I'll ask
 thunder        other than that, I just need to get this patch checked in and deployed, and we'll have new testing data
 thunder        hopefully it'll be useful
 dietrich       cool
 dietrich       i'm sure it will. far more realistic baselines for testing Tp
 thunder        yeah
 thunder        other than that, nothing new and exciting to report
 thunder        the 3.3.8 update seth mentioned
 thunder        I'll add interfaces for the text search stuff when I figure out how to do that :)
 dietrich       anyone else have anything to discuss?
 dietrich       the only other thing i wanted to mention is that we should be thinking about where we want UI to go after we get Fx2 parity
 dietrich       eg, considering stuff here: http://wiki.mozilla.org/Firefox/Feature_Brainstorming:Bookmarks
 thunder        yeah.
 dietrich       as well as adding any ideas to that list
 thunder        that is where the exciting stuff starts :-)
 dietrich       exactly!
 thunder        I want cheese-based bookmarks
 dietrich       mmmm
 thunder        but we should offer a lactose-free version
 dietrich       on that note: meeting adjourned :)
 thunder        um, with full text search in the db, we should have smart bookmark folders
 thunder        haha :)
 dietrich       thunder: yes, that'd be v. cool
 dietrich       hey i don't think that's on the list
 thunder        really
 *              thunder adds
 *              myk votes for cheese-based bookmarks
 thunder        hooray!
 thunder        okay, it has been totally added
 sspitzer       I don;'t have much to report, except a new patch (that address the comments from dietrich).
 thunder        woo
 thunder        commit!
 thunder        :-P

== Places meeting: 2006-10-27 2pm PST ==

 sspitzer       dietrich / thunder / myk / et al
 dietrich       hi
 sspitzer       dietrich:  first, thanks for the tip about the storage inspector.
 sspitzer       that's a very useful extension
 dietrich       yah
 sspitzer       https://addons.mozilla.org/firefox/3072/
 sspitzer       so, a couple questions that relate to the ERD
 sspitzer       http://wiki.mozilla.org/Places:BookmarksComments#Places_ERD
 sspitzer       we have title and user_title in the moz_places title (moz_history if you use the storage inspector on your trunk with places enabled "as is" today)
 sspitzer       those are usually the same, right?
 sspitzer       can we do a sort of copy-on-write trickery, where we don't store it twice if they are the same?
 sspitzer       when we do our query, we ask for both columns, and if user_title is null, we use title?
 sspitzer       or, is there a reason we do what we are doing?
 dietrich       sspitzer: iirc user_title is basically for bookmarks
 dietrich       title retains the original title from history
 dietrich       say you bookmark a page and title the bookmark differently, it populates user_title
 dietrich       user_title may be obsoleted by the schema changes actually
 sspitzer       how is that?
 dietrich       my opinion is that a bookmark title is one of the most-likely to be changed, and represented in multiple places, properties of a bookmark
 dietrich       and therefore should be tied to the bookmark, not to the history entry
 dietrich       i think that user_title should be removed, and a title property added to moz_bookmarks
 sspitzer       ok, I follow you.  I thought you were saying it has been changed in your new erd
 sspitzer       you are saying that we should.
 dietrich       right
 sspitzer       I agree, but this leads me into my next question, which is really something mconnor asked me (and he might have asked you already, too)
 thunder        dietrich: agreed
 thunder        I think
 dietrich       that's basically what i say in the wiki, but i forgot to change that in the diagram
 thunder        nod
 sspitzer       why are bookmarks and history in the same data model?  Could we have two databses, and when we need to, join across them?
 dietrich       sspitzer: since history is unique on URI and bookmarks isn't anymore, that's a possibility
 dietrich       however,
 thunder        I was thinking about that too
 dietrich       there might be performance repercussions of loading the 2 files separately
 myk            dietrich: putting user_title into moz_bookmarks seems like the right thing to do
 dietrich       i think that there are close ties between history and annos, bookmarks and annos, but not as close an association between bookmarks and history
 dietrich       wrt to how they're used in practice
 myk            dietrich: since it's a property of the bookmark, not the place itself
 dietrich       myk: yep
 dietrich       sspitzer: i think you bring up a larger question: what does a tight integration between history and bookmarks in the data model buy us?
 myk            sspitzer: it's a good question; one might also ask the corollary question, however: why ever have more than one database, no matter how many different kinds of data we start storing in sqlite?
 sspitzer       for myks question, do we want extensions to be using our database?
 sspitzer       not that I have a problem with extension authors, but it seems like they could impact the browser, even after the extension is removed.
 sspitzer       I was thinking (maybe naively) that having one giant database with everything could be costly on startup.
 sspitzer       at yesterdays 4pm meeting, we chatted here about table prefixes
 sspitzer       about why we were doing moz_*
 sspitzer       about my naive thinking, I don't know if sqlite is designed for one db with lots of tables, or better to have mutliple dbs.  we already have:
 myk            sspitzer: that may be true, since databases are represented as a single file; on the other hand, perhaps sqlite just loads a small portion of that file on startup; i don't know
 sspitzer       myk:  me neither
 sspitzer       we already have:
 sspitzer       bookmarks_history.sqlite        urlclassifier.sqlite
 sspitzer       formhistory.sqlite              urlclassifier2.sqlite
 sspitzer       search.sqlite
 sspitzer       each of those are a separate db on disk.
 sspitzer       dietrich asked:
 sspitzer       what does a tight integration between history and bookmarks in the data model buy us?
 dietrich       i think that the page URI singleton model demands a tight integration between them
 myk            sspitzer: as an aside, perhaps we should rename bookmarks_history.sqlite -> places.sqlite
 myk            (that is, unless it turns out that we should be sticking everything into a single database)
 dietrich       eg: if we remove the place_id FK in moz_bookmarks, what breaks?
 dietrich       anything that modifies a bookmark will now be annotating a bookmark URI, not the place URI
 thunder        it might make it slower to update the history table when one visits the place
 thunder        since you'd have to look up which row it is
 thunder        though, probably not
 thunder        since that is/could be indexed anyway
 dietrich       yeah
 thunder        (to look up by uri)
 sspitzer       does separating them make "clear private data" faster?
 sspitzer       if we don't have to worry about bookmarks being lost?
 dietrich       sspitzer: possibly
 dietrich       however, i think it's important to think about what we want from a functionality perspective first, then optimize on the requirements of those goals
 myk            fwiw, i agree with dietrich 
 myk            we should avoid premature optimization and instead focus on accurately modeling the concepts with the database schema
 thunder        hmm
 thunder        sure; but from that perspective
 myk            once we have an accurate model, then it makes sense to measure its performance and modify the schema accordingly (adding indexed, de-normalizing in exceptional cases, etc.)
 thunder        bookmarks is a separate concept from history
 thunder        so should live in its own table :)
 myk            thunder: right, which is why we represent the two concepts with their own tables
 thunder        ah, I thought the argument was geared toward not separating them
 dietrich       hence my question: after removing the bookmark singleton approach, is there a reason to keep that place_id FK in the moz_bookmarks table?
 thunder        my confusion, sorry
 sspitzer       dietrich: thinking...
 dietrich       i'm trying to think of use-cases for showing history data for a bookmark
 thunder        there are
 dietrich       but in those cases, you could join on URI instead of place_id
 thunder        but, right
 thunder        which is why I think it's not necessary to have the FK there
 dietrich       also: say you wanted visitation stats for a bookmark, those would have to be stored against the bookmark URI, not the original URI
 thunder        URI?
 thunder        or row?
 dietrich       thunder: same thing wrt to annotations
 dietrich       thunder: right now there are place: uris for folders, queries, etc
 thunder        it could be useful to know how many times I've been to a site vs how many times I've clicked on this bookmark
 thunder        both could be useful, really, depending on context
 dietrich       one of my recommendations was to provide URIs for all bookmarks datastore objects, for annotations, etc
 myk            dietrich: removing the place_id FK essentially turns the uri column into a FK; i don't see a problem with that offhand
 dietrich       even if that URI is something simple like place:bookmark:{PKID}
 dietrich       myk: exactly - place_id is redundant at the point
 thunder        yeah, it does do that, with the difference that we don't need to maintain it pointing to an actual row
 thunder        (right?)
 dietrich       yep
 myk            dietrich: there's a minor space hit, but i'd say it's insigificant, since users don't generally have too many bookmarks
 thunder        so you could have a bookmark with a uri that is not in the history table
 dietrich       thunder: i think so
 myk            dietrich: one consideration is that joins might be more expensive
 thunder        whereas with the FK route it seems bad if it doesn't point anywhere (though, it could just be null, I guess? - but even then you still have to walk the bookmarks when you clear history)
 dietrich       well, a bookmark's URI *is* a URI that's not already in history
 thunder        er
 thunder        no, I mean
 dietrich       myk: how so?
 thunder        the uri pointed to by a bookmark
 myk            dietrich: as we'd be joining on URI string rather than integer ID
 dietrich       myk: good point - given that we'd already be indexing the URI cols, i wonder how big that hit would be
 myk            dietrich: here's a use case that requires a join from places -> bookmarks: say i wanted to search my history for some site.  in the search results, i'd probably want to know that a particular result is bookmarked rather than being just some random site i visited once
 myk            f.e. the search results might include an icon for each result, and the icon for results that are also bookmarks would be different (f.e. a bookmark symbol overlaid over the favicon)
 dietrich       myk: so that was my next question: in use-cases like that, should it be done at the db layer?
 myk            dietrich: well, my first thought is that the db layer solution would be simplest, but maybe that's just because i know how to do it
 thunder        that is a valid use-case, but we don't know how much slower that would be sans-integer-FK
 dietrich       myk: i think it would be faster to implement that use-case at the db layer than it would be to call out to the bookmark service from the front-end
 thunder        nor do we know much much faster other operations (e.g. clear history) would be
 myk            dietrich: sure; in that case, i suspect that a join from places -> bookmarks would be faster on an integer key than a URI, but i don't know that for sure
 dietrich       any interaction w/ annotations are likely joins on URI cols
 dietrich       in the current model, and with the proposed changes
 dietrich       and likely to occur more often than the use-case we're discussing
 myk            another consideration is that it isn't clear to me that "places" is conceptually the same thing as "history"; i imagine that interesting applications could be enabled by differentiating between the two
 dietrich       so that's really an issue we have in the status quo (if it's even an issue)
 dietrich       myk: i totally agree. i think that using annotations as an extensibility mechanism is key to that.
 myk            dietrich: yes, annotations is important, but we would also need to provide for the possibility that a place in the places table doesn't necessarily represent an item in "history", if "history" means the set of places the user has visited
 thunder        hrm
 dietrich       myk: i think that the separation of moz_visits and moz_places effectively does that
 myk            dietrich: yes, indeed
 myk            dietrich: but we seem to have been talking about the places table as if it's history, and i wanted to make sure we were differentiating between them conceptually
 dietrich       ok
 dietrich       hm, i wonder if clearing history in places clears moz_places entries (ne moz_history), or just entries in moz_visits
 dietrich       b/c entries in moz_places are kind of like an implicit visit
 sspitzer       dietrich:  didn
 dietrich       sure you can put non history stuff in there, but history URIs are there because you visited them
 dietrich       i guess if there's no moz_visits entry, there's no way to tell how the moz_places entry got there
 sspitzer       't brett also tells us recently that the clearing of history happens "in chunks"?
 myk            dietrich: perhaps the solution is to clear entries which are neither bookmarks, nor history, nor have any annotations in the annotations database to indicate that some other code cares about those URIs
 sspitzer       looking for his comments...
 dietrich       myk: sure that might do it
 dietrich       when clearing history, clear moz_visits *and* remove "un-attached" entries in moz_places
 myk            dietrich: you mentioned earlier that interactions between places and annotations are likely joins between URI columns, but it looks like your ERD has them joined by ID
 sspitzer       "places history expiration happens incrementally as you browse (instead of delaying shutdown)"
 dietrich       sspitzer: yeah, i wonder if clearing history manually forces removal from the db, or if it does the expire-over-time thing
 dietrich       eg: via cpd
 myk            dietrich: also, there's a partial model for expiring annotations; i've previously proposed adding "expire when bookmark goes away" functionality; perhaps there should also be "expire when history cleared"
 sspitzer       for more on expiration, see brettw's comments on http://wiki.mozilla.org/Places/StatusMeetings/2006-10-12
 sspitzer       he writes:
 dietrich       myk: yeah i was mistaken, it's by place_id
 dietrich       myk: sure. i think brettw referenced a bug for that
 dietrich       maybe u filed it
 dietrich       :)
 myk            :-)
 myk            i probably filed the bookmarks one; if so, i'll add history to it
 myk            hope i'm not dumping too many considerations on y'all
 dietrich       myk: not at all :)
 dietrich       we're still early in the process, so it's great to have more ideas to churn on
 dietrich       i think the takeaways wrt to schema, that i need to update ERD to:
 sspitzer       dietrich:  look at http://lxr.mozilla.org/seamonkey/source/toolkit/components/places/src/nsNavHistoryExpire.cpp, clearing history manually doesn't appear to do the expire over time thing.
 dietrich       - remove place_id FK from bookmarks
 dietrich       and rename the db file to moz_places.sqlite?
 dietrich       or just places
 sspitzer       I think places.sqlite
 sspitzer       based on the names of the other .sqlite files
 dietrich       k
 sspitzer       one more item, in addition to "- remove place_id FK from bookmarks"
 sspitzer       what about user_title?
 sspitzer       or, did you already have that, but not in the ERD image?
 dietrich       sspitzer: right, that also needs to be added to the diagram
 sspitzer       any objection to me making today's chat part of yesterday's 4pm chat log, that I'll post on http://wiki.mozilla.org/Places/StatusMeetings?
 myk            i'm not so sure removing place_id is actually the right thing to do
 dietrich       myk: yeah that one's wobbly
 dietrich       i think there aren't any super arguments for keeping it
 dietrich       but no args for removing it either
 dietrich       which leaves performance, which i think would be worse (just not sure to what degree)
 dietrich       i leave it in for now
 dietrich       however, thunder, this means that you cannot bookmark something that isn't in moz_places :)
 sspitzer       dietrich:  then how does it work when I right click and bookmark a link I've never been to?
 sspitzer       do we create an entry, and set the visited to null, or some special value?
 sspitzer       (in the moz_places table)
 dietrich       sspitzer: right now it creates entries in both
 dietrich       both moz_history and moz_bookmarks
 dietrich       i'm not sure what we do for visits
 dietrich       sspitzer: http://lxr.mozilla.org/seamonkey/source/toolkit/components/places/src/nsNavHistory.cpp#868
 dietrich       that's what's called when you add a bookmark
 dietrich       "create a new hidden, untyped, unvisited entry"
 sspitzer       ok, and then the moz_bookmark references that?
 dietrich       yep
 sspitzer       ok, thanks for clarifying.