Changes

Bugzilla's core architectural design originates in early web development times, where things were simpler. By far the largest issues which is holding back Bugzilla's responsiveness is the near complete lack of cross-request caching. Unlike most modern web applications, Bugzilla doesn't utilise a an external caching system such as memcached or redis to avoid regenerating the same content, and retrofitting this to Bugzilla without substantial rewriting is proving to be a difficult problem to solve.

Up until reasonably recently, Bugzilla didn't do much in terms of same-request caching: for example creating an object from its ID always required a round-trip to the database, even if the object had already been created while processing that request but in another method. Bug NNNN implemented simple object caching, with other bugs building on it make it function with more classes.

* architecture not designed The templating engine Bugzilla uses - Template-Toolkit - is excellent, however some operations are slower than in other comparable engines. One feature which is lacking is the ability to scale up easily flush the generated content to the client, which means the entire page must be generated before it can be sent to large sizesthe client.** unlike modern web appsAnother factor is the schema design, not designed to specifically BMO's use caching** no crossof Bugzilla's built-request caching at all, difficult in custom fields to retrofit* usage increases (by both users for status and api-driven)* tracking flags . Bugzilla's custom fields implementation is geared around a small number of fields, and is implemented by adding columns to the width of main bugs table. Since the rapid release trains started, we've been adding 6 new fields every 6 weeks. This has resulted in a very wide bugs table(over 160 columns) which has a performance impact across both Bugzilla's business logic layer and within the database itself.* slow template engine, exacerbated by We've also seen an increase the the state amount of caching requests which Bugzilla is required to serve; from real clients but primarily from other tools built using Bugzilla's APIs.

* lots There has been significant analysis of analysis on the performance bottlenecks within Bugzilla (eg. using NYTProf) by bugzillamany participants in Bugzilla development (most noticeably from Bugzilla, bmoMozilla, and other teams (such as redhat's bugzilla teamRedHat developers)* , and quite a few changes have already been committed to default behavour upstream to avoid doing slow things improve performance by default optimising code or increasing the use of the same-request cache. To address the "every object request is loaded from the database" iss, {{Bug|811280}} implemented simple object caching (egwithin the same request). don This was backported from upstream't search comments by defaults trunk and has been running on BMO for some time. We are working on extending this mechanism so more classes make use of it. In April (?need to check?)* BMO was moved from a cluster in the PHX datacentre to SCL3, with improvements to the infrastructure upgrades (it runs on. This includes, but is not limited to, faster webheads, more webheads, upgrading moving email processing off the webheads, faster database versionservers, etc)* implemented and extended same-request caching (object cache, case-by-case template caching, etc)* explored using a reverse proxy (no gains, bugzilla makes good use more recent version of cache headers)MySQL.

* performance improvements are Work is always an ongoing with regards to the best way to implement caching mechanisms into Bugzilla, which has included investigations into reverse proxies and selected memcached usage. This effortwill continue.** eg. analysis Analysis of bug updates has shown that email generation is taking a significant amount of time when a bug is updated, due to the amount of emails generated. Due primarily to component watching (a BMO-specific customisation), Bugzilla notifies an average ~of 40 emails recipients per change, however it isn't unusual for a single change to trigger over 100 emails, which are . Currently their content is generated synchronously (but encrypted , however the actual encryption and delivered by delivery of the email happens in a background daemon). {{bug|877078}} moves the generation of bugmail to the daemon, and has shown significant time savings during testing. * changing Work on refactoring how we store tracking and status flags are implementedis nearing completion ({{Bug|750742}}), which should not only assist in overall performance, it allows us the opportunity to change how we present the tracking/status flag information without needing to heavily customise the Bugzilla core. The BzAPI (REST) API proxy, which currently is a service completely external to Bugzilla, is nearing its long journey to being included as an alternative native Bugzilla webservice endpoint, along size XML-RPC, JSON-RPC and JSON-P. Follow {{Bug|866927}} to see the progress there. This is expected to greatly improve the responsiveness of the REST API, and enables us to perform smart caching of requests. * always evaluating While the BzAPI proxy consumers themselves should consider the impact these calls have on BMO and implement caching mechanisms such as memcache , it's understandable why this doesn't always happen (I've seen many a quick-and redis-dirty solution quickly turn into production level usage without refactoring to lessen its impact). Once the REST services are integrated with Bugzilla, we plan on caching identical requests within a set time-frame to guard against accidental BzAPI driven DDOSing of BMO. * migrating One change in the pipeline which should net client-side benefits is the upgrade of our core javascript library - YUI - from yui2 version 2 to yui3 (version 3. YUI3 footprint on the browser is significantly less javascript to load when compared with YUI2, with a strong focus on each page)* migrating bzapi's rest proxy into bugzilla itself (tighter integration will enable performance improvements to avoid redundant lazy-loading of libraries. Yahoo are performing the work)for us here, in {{bug|453268}}.