BMO/performance

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

bugzilla.mozilla.org and Performance

The Problem

Bugzilla has a long history of performance issues, especially when it starts to scale up to the sizes seen on the largest sites. In comparison with modern web applications it can be sluggish to load or update a bug.

Why?

Bugzilla's core architectural design originates in early web development times, where things were simpler. By far the largest issues which is holding back Bugzilla's responsiveness is the near complete lack of cross-request caching. Unlike most modern web applications, Bugzilla doesn't utilise a caching system such as memcached or redis to avoid regenerating the same content, and retrofitting this to Bugzilla without substantial rewriting is proving to be a difficult problem to solve.

Up until reasonably recently, Bugzilla didn't do much in terms of same-request caching: for example creating an object from its ID always required a round-trip to the database, even if the object had already been created while processing that request but in another method. Bug NNNN implemented simple object caching, with other bugs building on it make it function with more classes.

• architecture not designed to scale up to large sizes
  • unlike modern web apps, not designed to use caching
  • no cross-request caching at all, difficult to retrofit
• usage increases (by both users and api-driven)
• tracking flags and the width of the bugs table
• slow template engine, exacerbated by the state of caching

What have we done already?

• lots of analysis on performance bottlenecks by bugzilla, bmo, and other teams (such as redhat's bugzilla team)
• changes to default behavour to avoid doing slow things by default (eg. don't search comments by default)
• infrastructure upgrades (more webheads, upgrading database version, etc)
• implemented and extended same-request caching (object cache, case-by-case template caching, etc)
• explored using a reverse proxy (no gains, bugzilla makes good use of cache headers)

What will be be doing?

• performance improvements are always an ongoing effort
  • eg. analysis has shown that email generation is taking a significant amount of time when a bug is updated, due to the amount of emails generated. average ~40 emails per change, however it isn't unusual for a single change to trigger over 100 emails, which are generated synchronously (but encrypted and delivered by a daemon)
• changing how tracking flags are implemented
• always evaluating caching mechanisms such as memcache and redis
• migrating from yui2 to yui3 (significantly less javascript to load on each page)
• migrating bzapi's rest proxy into bugzilla itself (tighter integration will enable performance improvements to avoid redundant work)