Firefox/Projects/PlacesQueryAPIRedesign

Places Query API

Create a sensible, easy to use Query API for Places for Firefox.next & Jetpack. This API should make it possible to do targeted queries against history and bookmarks with a minimum of code.

Goals/Use Cases

• An elegant and easy to use API. Fx devs, Jetpack and extension developers will all benefit.
• allows us to prototype the new ui for Fx.next
• 'Pluggable' results handling Easy wrappable querying object and results
  
• Focus on JS usage
• Paging Support - Sounds like a great use of Generators! dropped, there is no perf gain, can be a follow-up
  
• Fetch individual record by Id or other property

(After talking to the Labs guys about Weave, we should try to provide a plugin-interface whereby you can allow a developer to keep track of and index random JSON objects.

Dan Mills has example code that was created for the people store. Link coming soon.)

Non Goals

• Tailoring the API to tree views
• Creating the perfect API (lets iterate)
• Create an abstract datastore
  
• Intended as a snap-in replacement of the current API Future follow-up through a live-updating wrapper
  
• Wrap current sync and XPCOM API

Status

• Old prototype landed in Jetpack 0.8, currently being revised
  
• Team
  • API: ddahl, mak
  • UX/UI: not required so far, future users could though
• Implementation bugs: bug 522572 bug 545700 bug 531940 bug 543888
• Next Steps
  • Iterate the feel of the API with the Places team and other interested developers. Ideas and feedback are encouraged. Build working examples to encourage feedback.

Timeline / Milestones

• 2010/02 Landed in Jetpack 0.8
• 2010/03 Work on "skeleton" API design with Marco
• 2010/05 New redesign, collecting feedback and proceeding.
  

Delivery Requirements

• This is purely additive.
• Coordination with Jetpack as this should be coded once, checked into places and imported into Jetpack's implementation.

Constraints

• Final implementation should use only Async Storage APIs

Testing

• Comprehensive xpcshell tests
• Example/Documentation style Browser Chrome tests Not for now, when treeviews will use this.
  
• TDD

Related Projects

• Jetpack
  

Additional Details

The initial API "sketches" are here: [1]

Faaborg has been mocking up designs and putting a lot pf thought into the UX: http://blog.mozilla.com/faaborg/2009/10/13/browsing-your-personal-web

Inspiration

Links to code we should study...

Gloda

Gloda: [2]

facet.js: [3]

Gloda Fundamental Attribute provider: [4]

Explicit Attribute Provider: [5]

FacetView: [6]

Current Sketches

function callback(results) {
  // results is a simple array of ResultItem, it's pretty easy to wrap it
  // if more complex management is needed.
  if (results.length == 0)
      dump("En empty results array indicates last results push\n");
  else
      dump("Collected " + results.length + " results this turn\n");
}

// Results are pushed to the callback as soon as they are available.
new PlacesQuery([object]QueryConf, [function]callback, [scope]thisObject);

// It's possible to pass multiple QueryConf objects, results will be merged
// based on the .merge attribute of each config.
// First query results are always unioned.
// In case of multiple queries, sort, group and limit of the first query will
// be used for the global result.
new PlacesQuery([QueryConf1, QueryConf2], callback, thisObject);

// It's possible to create the query, but run it later.
let query = new PlacesQuery(QueryConf);
query.execute(callback, thisObject);

/* The query can be so configured:
 *
 * _QueryConf_ = {
 *   phrase: string.
 *           Containing this string in either title, uri or tags.  Case
 *           insensitive.  Can use ^ and $ to match at beginning or end.
 *   host: string.
 *         Containing this string in the host.  Case insensitive.
 *         Can use ^ and $ to match at beginning or end.
 *   uri: string.
 *        Containing this string in the uri.  Case insensitive.
 *        Can use ^ and $ to match beginning or end.
 *   annotated: array of strings.
 *              With these annotations (Either page or item).
 *   bookmarked: object
 *   {
 *     tags: array of strings.
 *           Tagged with these tags.
 *     at: object
 *     {
 *       folder: number.
 *               Inside this folder. (non-recursive)
 *       position: number.
 *                 At this position. (relative to folder).
 *                 If undefined or null matches all children.
 *     }
 *     id: number.
 *         Bookmarked with this id.
 *     when: object
 *     {
 *       begin: optional Date object
 *              Bookmarks created after this time (included).
 *              Defaults to epoch.
 *       end: optional Date object 
 *            Bookmarks created before this time (included).
 *            Defaults to now.
 *     }
 *     modified: object
 *     {
 *       begin: optional Date object
 *              Bookmarks modified after this time (included).
 *              Defaults to epoch.
 *       end: optional Date object 
 *            Bookmarks modified before this time (included).
 *            Defaults to now.
 *     }
 *     onlyContainers: boolean.
 *                           Removes any non-container from results.
 *                           Default is false.
 *     excludeReadOnlyContainers: boolean.
 *                                Removes read only containers from results.
 *                                Default is false.
 *   }
 *   visited: object
 *   {
 *     count: object
 *            This is lazily based on visit_count, thus is not going to work
 *            for not counted transitions: embed, download, framed_link.
 *     {
 *       min: optional number.
 *            With more than this many visits.
 *            Defaults to 0.
 *       max: optional number.
 *            With less than this many visits.
 *            Defaults to inf.
 *     }
 *     transitions: array of transition types.
 *                  With at least one visit for each of these transitions.
 *     when: object
 *     {
 *       begin: optional Date object
 *              With visits after this time (included).
 *              Defaults to epoch.
 *       end: optional Date object 
 *            With visits before this time (included).
 *            Defaults to now.
 *     }
 *     excludeRedirectSources: boolean.
 *                             Removes redirects sources from results.
 *                             Default is false.
 *     excludeRedirectTargets: boolean.
 *                             Removes redirects targets from results.
 *                             Default is false.
 *     includeHidden: boolean.
 *                    Includes also pages marked as hidden.
 *                    Default is false.
 *     allVisits: boolean.
 *                Returns all visits ungrouped.
 *                Default is false, that means visits are grouped by uri.
 *   }
 *   sort: object
 *   {
 *     by: string.
 *         Either "none", "title", "time", "uri", "accessCount", "lastModified",
 *         "frecency".  Defaults to "none".
 *     dir: string.
 *          Either "asc" or "desc".  Defaults to "asc".
 *   }
 *   group: string.
 *          Either "tags", "containers", "days", "months", "years" or "domains".
 *          Defaults to "none".
 *          NOTE: Not yet implemented.
 *   limit: number.
 *          Maximum number of results to return.  Defaults to all results.
 *   merge: string.
 *          How to merge this query's results with others in the same request.
 *          Valid values:
 *          - "union": merge results from the 2 queries.
 *          - "except": exclude current results from the previous ones.
 *          - "intersect": only current results that are also in previous ones.
 * }
 * 
 * NOTE: In case of multiple queries, sort, group and limit of the first query
 * will be used for the global result.
 *
 *
 * The query returns an array of these simple ResultItem objects
 * ResultItem {
 *   pageId: the place_id of the page, useful for external linking of resources
 *   uri:
 *   title:
 *   host:
 *   accessCount: visit count for pages, aggregated or null for containers
 *   time: visit date, aggregated or null for containers
 *   icon: url of the icon
 *   sessionId: visit session or null if not available
 *   itemId: bookmark id or null if not bookmarked (see isBookmarked)
 *   isBookmarked: whether this is bookmarked or not
 *   dateAdded: bookmark creation Date() or null if not bookmarked
 *   lastModified bookmark modification Date() or null if not bookmarked
 *   parentId: bookmark folder id or null if not bookmarked
 *   tags: string of tags
 *   tagsArray: array of tags
 *   bookmarkIndex: position of the bookmark in his container or null
 *   frecency:
 *   visitId: id of the visit or null
 *   referringVisitId: id of the originating visit or null
 *   referringUri: uri of the originating visit or null
 *   transitionType: transition of this visit or null
 *   type: old container implementation type
 *   readableType: "bookmark", "container", "separator", "visit", "page"
 *   query: if this is a container will return a new PlacesQuery for contents
 * }
 * 
 */

// EXAMPLES (not all of them work, they are supposed to show the conf)

// Bookmarks toolbar.
new PlacesQuery({ bookmarked: {
                      folders: [PlacesUtils.bookmarksToolbarFolderId]
                  },
                  group: "container"
                });

// History menu.
new PlacesQuery({ visited: {},
                  sort: { by: "time", dir: "desc" },
                  limit: 10
                });

// Start-of-awesomebar (still missing adaptive, keywords, ...).
new PlacesQuery({ phrase: "someword",
                  sort: { by: "frecency", dir: "desc" },
                  limit: 12
                });

// Folder picker.
new PlacesQuery({ bookmarked: {
                      at: { folder: PlacesUIUtils.allBookmarksFolderId }
                      onlyContainers: true,
                      excludeReadOnlyContainers: true
                  },
                  group: "container"
                });

// Most recent bookmarks.
new PlacesQuery({ bookmarked: {},
                  sort: { by: "lastModified", dir: "desc"}
                  limit: 5
                });

// Most recent tags.
new PlacesQuery({ bookmarked: {},
                  group: "tags",
                  limit: 5
                });

// Left pane query.
new PlacesQuery({ bookmarked: { folders: [PlacesUIUtils.leftPaneFolderId] },
                  annotated: ["Places/OrganizerQuery"]
                  group: "containers"
                });

// Downloads
new PlacesQuery({ visited: {
                      transitions: [PlacesUtils.history.TRANSITION_DOWNLOAD],
                      includeHidden: true,
                      allVisits: true
                  },
                  sort: { by: "time", dir: "desc" }
                });

// Visited bookmarks containing either "foo" or "bar".
new PlacesQuery([
    { phrase: "foo",
      bookmarked: {},
      visited: {}
    },
    { phrase: "bar",
      bookmarked: {},
      visited: {}
    },
);


// Visited bookmarks containing "foo", but not "bar".
new PlacesQuery([
    { phrase: "foo",
      bookmarked: {},
      visited: {}
    },
    { phrase: "bar",
      bookmarked: {},
      visited: {},
      merge: "except"
    },
);

// Visited bookmarks containing "foo" and "bar".
new PlacesQuery([
    { phrase: "foo",
      bookmarked: {},
      visited: {}
    },
    { phrase: "bar",
      bookmarked: {},
      visited: {},
      merge: "intersect"
    },
);

Testing

• Most of this will be automated and will not need QA involvement
• QA will keep an eye out for regressions during regular release cycles
• QA will run a testday once this has landed