Support/Kitsune/Panic Button

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

We'd like to build a sort of Panic Button into Kitsune that would be used under the heaviest load to keep the site functional for most users. This page is to brainstorm ideas and work out possible implementations and issues.

In the next several months, we know we'll see an Android beta, Firefox 4 beta, Android release, and Firefox 4 release. All of these have potential to hit SUMO particularly hard, and this is one part of preparing. But as much as a Panic Button is useful for the things we know about, it's so much more useful for the next Pac Man incident.

Goals

The Panic Button should:

• Prevent replication lag from climbing.
• Keep SUMO running with some reduced functionality.†
• Maintain all read-access to the site.
• Be extremely easy to activate and deactivate (eg: "more hardware" is probably too slow).

A "nice to have" would be "Still allow full superuser access" but it's secondary to keeping the site up.

†Obviously, if it keeps the site running with full functionality, it's not a Panic Button, it's something we should do normally. :)

Proposals

Add suggestions here.

Read-only Mode

All credit to Dave Dash for this idea.

A mode, probably enabled with an environment variable, that essentially shut off all views that write to the database. (Optionally we could still let superusers/admins log in and perform writes. This is somewhat more complex.)

This might be implemented via a decorator on views, eg @write_required, that would check os.environ (and possibly superuser status). It could return a 503 Service Unavailable if the check fails, with a message that the site is experiencing heavy load and a Retry-After header (extremely optionally, we could make this also configurable via an environment variable).

Advantages

• Relatively easy to implement.
• Very easy to turn on and off.
• Stopping writes should prevent replication lag issues.

Disadvantages

• Hiding all the UI that points to writes would be more complicated.
• Not hiding all the UI that points to writes could frustrate users.
• Not entirely sure how to allow superusers to log in in the first place.