Firefox Home Server Architecture

Huh What Is This!?

In short: we are turning Firefox Home into a web application.

Home will become a Web App. In case of iOS it will be likely be shipped inside a small native wrapper for the iPhone and iPad but its core will be a portable Mobile Web App.

So this means that we have to build something server side. To put it simple, we are basically building a web site that acts as a sync client. When the web app has your data, we can present it in all kinds of interesting ways.

For recent devices (Android, iOS, PlayBook) we will expose a modern HTML5 web site that is mostly build on top of JSON web services that we expose. For older devices, or less capable devices, we plan to serve simple static HTML4.

Simple plan for Firefox Home 2.0

Home becomes Web App with native iPhone and iPad wrapper

• Focus on iOS: iPhone and iPad
• Integration with Mozilla’s new identity service for secure and easy single-signon with your iPhone or iPad
• Same functionality as (native app) 1.0: access history, bookmarks?, tabs, awesomebar-style searching
• New visual design
• Optional: Personas

This is work in progress. We want to keep the first release simple.

Web Components

(Missing: memcache where appropriate)

Zeus

Description

Zeus is the SSL frontend and the load balancer. Ideally Zeus will be able to look at the web service URLs from Firefox Home and based on some value in the request, send the request to the right Web Server.

(Home can of course also ask services to what server to go, just like is done for sync clients now.)

Web Server

Description

The Web Server runs the Firefox Home web application. It serves both static content and JSON web services. The site will be very HTML5/JS based, but in a later version we probably also want to serve more static HTML4.

The Web Server also periodically pushes sync requests to the Sync Queue. For example every 15 minutes. It will simply iterate over the sync accounts and based on the status of that account it will push a work request in the sync queue. This can be optimized by syncing less frequently used accounts less often. Or even skip accounts that have been created but not used in a while.

Ideally web servers do not do this kind of work but I think that a simple cron job to queue messages will not cause a major load on these systems. If it does, or when there is a need for more periodic tasks, then we can move those to an extra machine.

Each Web Server has a database dedicated to it. The Web Server talks directly to this database. It will need to do real queries. It will of course try to cache as much as it can in for example memcache.

Software Requirements

The web application will be written in Python. Currently the prototype is a Pyramid/SQLAlchemy type application. Ideally it is hosted on the same kind of platform that Services uses or at least something they approve.

Performance Concerns

This application differs from the sync services that it will actually need to run real queries and searches on people's sync data. This will be a very different load/usage pattern.

Database

Description

Initially the databases will mainly contain the following:

• Bookmarks
• History Items
• Tabs

We will normalize or extend the data that we take from sync as much as needed so that it is possible to run queries on the data.

All data is stored unencrypted in the database.

Software Requirements

There is currently no preference for any specific type of database. MySQL, Drizzle, PostgreSQL or anything that allows queries and joins will be fine.

Uptime

Data on this server is a *copy* of what already exists in the Sync Service. I would therefore say that there is a much lower requirement to keep this running. In case of problems we can always sync the data again. This will be a relatively expensive operation but it should be pretty fast.

(This changes as soon as we are also going to generate data based on sync data, usage or when we receive new data from the app)

Future

Sync Queue

Description

The Sync Queue exists to coordinate sync operations. The Web Servers periodically push requests to the queue, which the Sync Workers will pick up.

I understand that we do not currently use any specific queue system. Anything is really fine, as long as it is relatively simple to talk to it from Python.

I do not think queue persistence is needed for this application. If the queue gets lost then we simply schedule a request to sync within the next 15 minutes or whatever the sync interval is.

Sync Worker

Description

A Sync Worker is a machine that will sync a user's account.

To make this part of the infrastructure more reliable and distributed, the workers listen to a work queue. Each worker can sync N accounts in parallel.

The worker takes a job from the queue and then talks to the database where that user's data lives and to the Public Sync Services and sync data.

The frequency and scheduling of sync operations is not in control of the Sync Workers. They simply handle sync requests.

Software Requirements

The sync client will be written in Python and can be hosted on any flavor of Linux.

Hardware Requirements

For these machines, memory is more important than disk. More memory allows us to run more sync processes in parallel.

Uptime

This is an important part of the infrastructure but there is no need to get extremely high uptime on this part. In case of failures we would simply not sync, which is not something the end user would notice in form of data loss, but only as a sync that might not have happened on time.

Because the workers are driven by a queue, another machine would simply pick up the sync request the next pass.

(We could also let the workers ack the sync request so that in case they fail, the original request that was in progress goes back in the queue to be picked up by another worker, but i don't think that is even needed here)

Public Sync Services

Simply the sync services as they exist now. Ideally no changes are needed and Firefox Home is simply another client that accesses sync using the defines APIs that all other clients use.

There is no special need for shortcuts to the Sync Services on an infrastructure level. As long as the Firefox Home Sync Workers can talk HTTPS to the Sync Services, all is good.

Pretty pictures