Support/Scaling

Scaling of support.mozilla.com

• Firefox 3 beta 5 will be replacing in-product help with support.mozilla.com, likely to happen next week (wed/thurs)
  • help will redirect user to sumo front page
  • from there user can click on common articles and tutorials
• Already having issues this week combined with other complications from release traffic

• what we need to do
  • Get redirects up and fixed (Thurs. AM) [1] [2] - DONE
  • Turn SUMO back on post-peak-time (12pm PST?) [3] - DONE
  • Fix P1 bugs in 0.6 list before b5

Questions about where we are now

• how many boxes serve up sumo?

We are at 8 boxes. It is on the PHP5 cluster though so these are shared boxes with most of the applications that we run.

• how many databases are used by sumo?

Two databases - mrdb80, mrdb81. Although from my inspections it seems that sumo ONLY uses the master (mrdb80) and has no ability to scale on slaves. (See https://bugzilla.mozilla.org/show_bug.cgi?id=426253)

• is it possible to adjust NS cache rules to be more squid-like so entries don't randomly expire?
• has there been a noticeable performance difference after the recent optimizations?
• what's currently the bottleneck? (ex: number of hits to the database, the queries we're running, PHP execution...)

Previous bottleneck was requests to the netapp - this has now been fixed by serving pages from local disk. This took page load times from 14s to 0.5s on average.

• what would be involved in upgrading to MySQL 5? [4]

This looks like it will be a Q2 goal.

Possible outcomes

• configure NS to cache all sumo links more aggressively (think super-squid)
• use squid if necessary
• cron to auto-pull a static version of SUMO landing page from Fx help, point redirects at this static version
• cron to auto-pull static versions of all Fx in-product pages as above
• back out b5 patch to give us more time to test/scale (last resort)