See the diagrams attached to https://bugzilla.mozilla.org/show_bug.cgi?id=731228

Web Server

fxhome-web

What does it do

The fxhome-web module contains the main pancake web application.

It is a Pyramid web application that is mounted on http://pancake/*

The web application is split up in several smaller web applications that are all 'glued' together by NGINX. The other services like pancake-social and pancake-search are mounted on top of this application in NGINX.

What does it store

It stores Pyramid/Beaker session data in the MemBase Server. Sessions currently only contain the email address of the logged in user.

What does it talk to

It talks to BrowserID to log users in.

It talks to the Pancake MemBase server to store/load session data

It talks to the Pancake User API Server to store and retreive user records.

It talks to the Pancake Thumbnailer API to generate thumbnails for sites when showing result 'tiles'.

fxhome-lattice

What does it do

Lattice is the service that builds graphs from a user's browsing data. It receives commands from the iOS application to records visited sites, links and organizes this data in a graph. Lattice is implemented as an HTTP REST API.

The API is documented at https://docs.google.com/spreadsheet/ccc?key=0AgzuG8vbAx63dHYwUmFBNThBbk8tdmdDTmtaVUZUaUE

It is a tornado web application that is mounted on http://pancake/lattice/*

What does it store

The lattice service stores all its data in the Neo4J Server. The data that it stores is described in the Neo4J section.

It also stores the same data in the Elastic Search server.

What does it talk to

It talks to the MemBase server to retrieve a user's session object. To find out if the request is authorized.

It talks to the Neo4J server to store and retrieve the user's browsing history in a graph based format.

It talks to Elastic Search to do search and generate 'top sites' for a user.

It talks to the Pancake Thumnail API when it returns a response with a list of sites.

pancake-social

What does it do

The pancake-social module implements the 'social' code. It is responsible for talking to Twitter and Facebook to collect the most recent messages from a user's message stream or wall.

It is a Tornado web app that is mounted on http://pancake/social_api/

What does it store

It does not store anything.

What does it talk to

It talks to the MemBase server to retrieve a user's session object

It talks to the pancake-user-api service to retrieve the user record which includes the Twitter and Facebook OAuth tokens.

pancake-search

What does it do

The pancake-search module implements the search code. It is responsible for talking to Bing and Twitter to run searches.

Searches are fully anonymous, no user info is currently passed on to Bing or Twitter.

It is a Tornado web app that is mounted on http://pancake/search/*

What does it store

It does not store any data.

What does it talk to

It talks to the Bing Search API

It talks to the Twitter Search API

User API Server

pancake-user-api

This implements our user database. It is a tiny HTTP REST API wrapped around a SQLite datbase.

It is not public facing and just runs internally for other Pancake components.

What does it store

It stores users and user_properties in a local SQLite database:

create table users (
  username text primary key,
  created integer not null
 )

create table user_properties (
  username text not null references users on delete cascade,
  property_name text not null,
  property_value text not null
)

The following properties are currently used:

• facebook_access_token
• twitter_access_key
• twitter_access_secret
• status (for beta signup, set to 'beta')

The Facebook and Twitter properties are storing OAuth tokens.

What does it talk to

It does not talk to anything.

Pancake MemBase Server

Membase

What does it do

Store temporary and semi-permanent data used by the Pancake components. Currently only fxhome-web stores session data in MemBase. Other compoentns are expected to cache data in MemBase where appropriate.

What does it store

Pancake web application session data. Session objects only contain the user's email address.

What does it talk to

Nothing.

Pancake Neo4J Server

Neo4J 1.6.0

Pancake Elastic Search Server

What does it do

Neo4J is a graph database that stores the data that is managed by the fxhome-lattice module.

What does it store

It stores the user's browsing data / history in a graph structure.

The nodes of the graph are sites that have been visited and contain the following information:

• The date of the visit
• The URL of the page that was visited
• The title of the page that was visited
• The id of the 'stack' that this visit is part of
• The id of the 'session' that this visit is part of
• A unique ID of the device that generated the visit

What does it talk to

It does not talk to anything. It only exposes an API to the fxhome-lattice service.

Pancake Elastic Search Server

Elastic Search 0.19

What does it do

It stores a full text index of the user's browsing data. It is pretty much a copy of the data in Neo4J except structured in such a way that we can fulltext search on it.

What does it store

The same data that is stored in Neo4J.

What does it talk to

It does not talk to anything. It only exposes the Elastic Search API.