Firefox/Projects/PlacesQueryAPIRedesign

Places Query API

Create a sensible, easy to use Query API for Places for Firefox.next & Jetpack. This API should make it possible to do targeted queries against history and bookmarks with a minimum of code.

Goals/Use Cases

• An elegant and easy to use API. Fx devs, Jetpack and extension developers will all benefit.
• allows us to prototype the new ui for Fx.next
• 'Pluggable' results handling Easy wrappable querying object and results
  
• Focus on JS usage
• Paging Support - Sounds like a great use of Generators! dropped, there is no perf gain, can be a follow-up
  
• Fetch individual record by Id or other property

(After talking to the Labs guys about Weave, we should try to provide a plugin-interface whereby you can allow a developer to keep track of and index random JSON objects.

Dan Mills has example code that was created for the people store. Link coming soon.)

Non Goals

• Tailoring the API to tree views
• Creating the perfect API (lets iterate)
• Create an abstract datastore
  
• Intended as a snap-in replacement of the current API Future follow-up through a live-updating wrapper
  
• Wrap current sync and XPCOM API

Status

• Old prototype landed in Jetpack 0.8, currently being revised
  
• Team
  • API: ddahl, mak
  • UX/UI: not required so far, future users could though
• Implementation bugs: bug 522572 bug 545700 bug 531940 bug 543888
• Next Steps
  • Iterate the feel of the API with the Places team and other interested developers. Ideas and feedback are encouraged. Build working examples to encourage feedback.

Timeline / Milestones

• 2010/02 Landed in Jetpack 0.8
• 2010/03 Work on "skeleton" API design with Marco
• 2010/05 New redesign, collecting feedback and proceeding.
  

Delivery Requirements

• This is purely additive.
• Coordination with Jetpack as this should be coded once, checked into places and imported into Jetpack's implementation.

Constraints

• Final implementation should use only Async Storage APIs

Testing

• Comprehensive xpcshell tests
• Example/Documentation style Browser Chrome tests Not for now, when treeviews will use this.
  
• TDD

Related Projects

• Jetpack
  

Additional Details

The initial API "sketches" are here: [1]

Faaborg has been mocking up designs and putting a lot pf thought into the UX: http://blog.mozilla.com/faaborg/2009/10/13/browsing-your-personal-web

Inspiration

Links to code we should study...

Gloda

Gloda: [2]

facet.js: [3]

Gloda Fundamental Attribute provider: [4]

Explicit Attribute Provider: [5]

FacetView: [6]

Current Sketches

function callback(results) {
  // results is a simple array of ResultItem, it's pretty easy to wrap it
  // if more complex management is needed.
  if (results.length == 0)
      dump("En empty results array indicates last results push\n");
  else
      dump("Collected " + results.length + " results this turn\n");
}

// Results are pushed to the callback as soon as they are available.
new PlacesQuery([object]QueryConf, [function]callback, [scope]thisObject);

// It's possible to pass multiple QueryConf objects, results will be merged
// based on the .merge attribute of each config.
// First query results are always unioned.
// In case of multiple queries, sort, group and limit of the first query will
// be used for the global result.
new PlacesQuery([QueryConf1, QueryConf2], callback, thisObject);

// It's possible to create the query, but run it later.
let query = new PlacesQuery(QueryConf);
query.execute(callback, thisObject);

/* The query can be so configured:
 * QueryConf = {
 *   phrase: string.
 *           Show only pages containing this string in either title, uri or
 *           tags.  Case insensitive.  Can use ^ and $ to match beginning or
 *           end.
 *   host: string.
 *         Show only pages containing this string in the host.  Case
 *         insensitive.  Can use ^ and $ to match beginning or end.
 *   uri: string.
 *        Show only pages containing this string in the uri.  Case
 *        insensitive.  Can use ^ and $ to match beginning or end.
 *   annotated: array of strings.
 *           Show only pages with these annotations (Either page or item).
 *   bookmarked: object {
 *                 tags: array of strings.
 *                       Show only pages tagged with these tags.
 *                 folders: array of numbers.
 *                          Show contents of these folders. (non-recursive)
 *                 id: number.
 *                      Show only the bookmark with this id.
 *                 when: array of 2 Date objects.
 *                       Show only bookmarks created between these times.
 *                       Can use null beginning or end time to match till epoch
 *                       or now.
 *                 modified: array of 2 Date objects.
 *                           Show only bookmarks modified between these times.
 *                           Can use null beginning or end time to match till
 *                           epoch or now.
 *                 excludeNonContainers: boolean.
 *                                       Removes any non-container from results.
 *                                       Default is false.
 *                 excludeReadOnlyContainers: boolean.
 *                                            Removes read only containers from
 *                                            results.  Default is false.
 *               }
 *   visited: object {
 *              count: array of numbers.
 *                     Show only pages with these so many visits.
 *                     Can use null minimum or maximum to match anything.
 *              transitions: array fo transition types.
 *                           Show only pages with at least one visit with these
 *                           transitions.
 *              when: array of 2 Date objects.
 *                    Show only pages with visits between these times.
 *                    Can use null beginning or end time to match till epoch
 *                    or now.
 *              excludeRedirectSources: boolean.
 *                                      Removes redirects sources from results.
 *                                      Default is false.
 *              excludeRedirectTargets: boolean.
 *                                      Removes redirects targets from results.
 *                                      Default is false.
 *              includeHiddenPages: boolean.
 *                                  Returns also pages marked as hidden.
 *                                  Default is false.
 *              includeVisits: boolean.
 *                             Returns all visits.
 *                             Default is false, that means visits are grouped
 *                             by uri, and no duplicates are returned.
 *            }
 *   sort: object {
 *           by: string.
 *               Either "none", "title", "time", "uri", "accessCount" or
 *               "lastModified", "frecency".  Defaults to "none".
 *           dir: string.
 *                Either "asc" or "desc".  Defaults to "asc".
 *         }
 *   group: string.
 *          Either "tags", "folders", "day", "month", "year" or "domain".
 *          Defaults to "none".
 *   limit: number.
 *          Maximum umber of results to return.  Defaults to all results.
 *   merge: string.
 *          How to merge this query's results with others in the same request.
 *          Either "union", "intersect" or "except".
 * }
 *
 *
/* QueryConf.prototype = {
 *   phrase: string.
 *           Show only pages containing this string in either title, uri or
 *           tags.  Case insensitive.  Can use ^ and $ to match beginning or
 *           end.
 *   host: string.
 *         Show only pages containing this string in the host.  Case
 *         insensitive.  Can use ^ and $ to match beginning or end.
 *   uri: string.
 *        Show only pages containing this string in the uri.  Case
 *        insensitive.  Can use ^ and $ to match beginning or end.
 *   annotated: array of strings.
 *           Show only pages with these annotations (Either page or item).
 *   bookmarked: object {
 *                 tags: array of strings.
 *                       Show only pages tagged with these tags.
 *                 folders: array of numbers.
 *                          Show contents of these folders. (non-recursive)
 *                 id: number.
 *                      Show only the bookmark with this id.
 *                 when: array of 2 Date objects.
 *                       Show only bookmarks created between these times.
 *                       Can use null beginning or end time to match till epoch
 *                       or now.
 *                 modified: array of 2 Date objects.
 *                           Show only bookmarks modified between these times.
 *                           Can use null beginning or end time to match till
 *                           epoch or now.
 *                 excludeNonContainers: boolean.
 *                                       Removes any non-container from results.
 *                                       Default is false.
 *                 excludeReadOnlyContainers: boolean.
 *                                            Removes read only containers from
 *                                            results.  Default is false.
 *               }
 *   visited: object {
 *              count: array of numbers.
 *                     Show only pages with these so many visits.
 *                     Can use null minimum or maximum to match anything.
 *              transitions: array fo transition types.
 *                           Show only pages with at least one visit with these
 *                           transitions.
 *              when: array of 2 Date objects.
 *                    Show only pages with visits between these times.
 *                    Can use null beginning or end time to match till epoch
 *                    or now.
 *              excludeRedirectSources: boolean.
 *                                      Removes redirects sources from results.
 *                                      Default is false.
 *              excludeRedirectTargets: boolean.
 *                                      Removes redirects targets from results.
 *                                      Default is false.
 *              includeHiddenPages: boolean.
 *                                  Returns also pages marked as hidden.
 *                                  Default is false.
 *              includeVisits: boolean.
 *                             Returns all visits.
 *                             Default is false, that means visits are grouped
 *                             by uri, and no duplicates are returned.
 *            }
 *   sort: object {
 *           by: string.
 *               Either "none", "title", "time", "uri", "accessCount" or
 *               "lastModified", "frecency".  Defaults to "none".
 *           dir: string.
 *                Either "asc" or "desc".  Defaults to "asc".
 *         }
 *   group: string.
 *          Either "tags", "folders", "day", "month", "year" or "domain".
 *          Defaults to "none".
 *   limit: number.
 *          Maximum umber of results to return.  Defaults to all results.
 *   merge: string.
 *          How to merge this query's results with others in the same request.
 *          Either "union", "intersect" or "except".
 * }
 *
 *
 * The query returns an array of these simple ResultItem objects
 * ResultItem {
 *   pageId: the place_id of the page, useful for external linking of resources
 *   uri:
 *   title:
 *   host:
 *   accessCount: visit count for pages, aggregated or null for containers
 *   time: visit date, aggregated or null for containers
 *   icon: url of the icon
 *   sessionId: visit session or null if not available
 *   itemId: bookmark id or null if not bookmarked (see isBookmarked)
 *   dateAdded: bookmark creation Date() or null if not bookmarked
 *   lastModified bookmark modification Date() or null if not bookmarked
 *   parentId: bookmark folder id or null if not bookmarked
 *   tags: string of tags
 *   tagsArray: array of tags
 *   bookmarkIndex: position of the bookmark in his container or null
 *   frecency:
 *   visitId: id of the visit or null
 *   referringVisitId: id of the originating visit or null
 *   transitionType: transition of this visit or null
 *   type: old container implementation type
 *   readableType: "bookmark", "container", "separator", "visit", "page"
 *   isBookmarked: whether this is bookmarked or not
 *   query: if this is a container will return a new PlacesQuery for contents
 * }
 * 
 */

// EXAMPLES (not all of them work, they are supposed to show the conf)

// Bookmarks toolbar.
new PlacesQuery({ bookmarked: {
                      folders: [PlacesUtils.bookmarksToolbarFolderId]
                  },
                  group: "folder"
                });

// History menu.
new PlacesQuery({ visited: {},
                  sort: { by: "time", dir: "desc" },
                  limit: 10
                });

// Start-of-awesomebar (still missing adaptive, keywords, ...).
new PlacesQuery({ phrase: "someword",
                  sort: { by: "frecency", dir: "desc" },
                  limit: 12
                });

// Folder picker.
new PlacesQuery({ bookmarked: {
                      folders: [PlacesUIUtils.allBookmarksFolderId]
                      excludeNonContainers: true,
                      excludeReadOnlyContainers: true
                  },
                  group: "folder"
                });

// Most recent bookmarks.
new PlacesQuery({ bookmarked: {},
                  sort: { by: "lastModified", dir: "desc"}
                  limit: 5
                });

// Most recent tags.
new PlacesQuery({ bookmarked: {},
                  group: "tags",
                  limit: 5
                });

// Left pane query.
new PlacesQuery({ bookmarked: { folders: [PlacesUIUtils.leftPaneFolderId] },
                  annotated: "Places/OrganizerQuery"
                  group: "folder"
                });

// Downloads
new PlacesQuery({ visited: {
                      transition: PlacesUtils.history.TRANSITION_DOWNLOAD,
                      includeHiddenPages: true,
                      includeVisits: true
                  },
                  sort: { by: "time", dir: "desc" }
                });

// Visited bookmarks containing either "foo" or "bar".
new PlacesQuery([
    { phrase: "foo",
      bookmarked: {},
      visited: {}
    },
    { phrase: "bar",
      bookmarked: {},
      visited: {}
    },
);


// Visited bookmarks containing "foo", but not "bar".
new PlacesQuery([
    { phrase: "foo",
      bookmarked: {},
      visited: {}
    },
    { phrase: "bar",
      bookmarked: {},
      visited: {},
      merge: "except"
    },
);

// Visited bookmarks containing "foo" but not "bar".
new PlacesQuery([
    { phrase: "foo",
      bookmarked: {},
      visited: {}
    },
    { phrase: "bar",
      bookmarked: {},
      visited: {},
      merge: "except"
    },
);