View source for Bugzilla:About Bugzilla Quality

Some people might be curious about the quality of Bugzilla code and the development process it goes through, so I ([[User:MaxKanatAlexander|Max Kanat-Alexander]], one of the primary developers of Bugzilla) have written this essay for the manager or end-user who might be evaluating Bugzilla or have concerns about open-source code.

Bugzilla is the de-facto standard in open-source bug-tracking because we create a quality product. Our rate of development easily parallels or surpasses commercial firms with more paid developers than we have volunteer developers, while maintaining a well-respected level of quality. Here's how we do it:

== Our Development Process ==

Bugzilla has a far more rigorous and orderly development
process than any commercial company I've worked for. This rises out
of the necessity of a team that works together over long distances--our
administrative procedures must be very exact, or we wouldn't have a
product at all.

The first thing I'll say is that our development process
happens completely in the public eye. Everything I say here about
Bugzilla quality can be fully verified by anybody with the time and
interest. In a commercial environment, you must simply take it on faith
that there is a process, and that the developers and product managers
actually ''follow'' that process. But with Bugzilla, you can know and be
sure.

Here are some of the details of our process:

=== Fully-Tracked Changes ===

Even tiny changes must be fully documented in the bug-tracking
system, with full justifications for the change and a specific patch
posted directly to the bug report. This rule is enforced largely by
necessity--a vast number of our contributors don't have access to our
version-control system, and thus '''must''' file a bug and post a patch in
order for us to even know about it. Contrast this to a commercial
development environment where developers are ''supposed'' to post changes
to the bug-tracking system, but don't necessarily always ''have'' to do
so in order to actually get code checked in.

=== Code Review ===

Once a bug is filed, a patch must be posted to that bug and
reviewed by a qualified reviewer. Qualified reviewers are people who
have proven their ability to follow our development standards and who
themselves write excellent code. All reviewers have been constant
contributors to Bugzilla itself for months or years before they become
reviewers. All reviewers are approved directly by the Project Leader
after nomination at a meeting of current reviewers, and are only
approved if their history of writing quality code ''clearly'' warrants it.

Patches tend to go through 1 to 10 revisions, based on whether
or not they pass review. These reviews are themselves completely
documented in the bug-tracking system, along with the rationale for why
a particular piece of code was written or why the reviewer flunked the
patch on review.

Reviewers both test the actual functionality of the patch and
inspect the code for quality. Our policy is that the code in our
version-control system must always work. You could check out Bugzilla
at any time in our development process and it would be functional as
far as we were aware.

=== Code Approval Process ===

Once a patch passes review, before it can be checked in, it
must be granted "approval" by one of the leaders of the Bugzilla
Project, currently consisting of four people who are the world's
foremost experts on the internals and structure of Bugzilla, having
written much of it themselves. Approval is denied if: the patch is
implemented in a way that is in conflict with other pending patches,
the patch is implemented in a way that goes against Bugzilla's general
architecture, or it's a feature that is definitely not wanted. (Usually
unwanted features get caught before this phase, but about 0.1% of the
time they make it all the way to the approval stage.)

=== Test Suite ===

Once the code is checked in, there is a test suite that runs on
various different versions of Perl in various different configurations
to assure that the code is not broken and follows our quality
standards. Any failure of this test suite will warn the developers who
are forbidden to check any other code into version-control until the
problem is fixed.

== Our Release Process ==

Once we reach a certain point in time or on our development roadmap, we begin the process of releasing a new version of Bugzilla. Here's how that process works:

=== Code Freeze ===

Several months before a stable release, the code in the
version-control system is "frozen" and only bug-fixes can go in to the
code. This gives us a several-month stabilization period before a
release. We have a well-developed system for discovering and marking
which bugs block a release, so we know exactly what needs to be fixed
before the release can go out.

=== Development and Release Candidate Releases ===

We release several "development" and "release candidate"
releases before a final stable release, over a period of many months.
This gets our code into testing in real environments. Sometimes even
significant installations will run our development code, as it is
often stable enough to run in a production environment even before it
has undergone QA (though we never recommend this to users, on
principle).

=== Testing on a Major Installation ===

Usually before a major release, bugzilla.mozilla.org will
upgrade to our latest development code. This is a Bugzilla installation
with tens of thousands of active users, and over 400,000 bugs filed.

=== QA ===

Finally, before any stable release, an exact QA process is done
by several QA engineers testing each release in several different
environments. There are many automatic tests run by Selenium, a tool
for testing web applications. The rest of the tests are done manually
per test plans, using an application called Testopia to track the tests
and their results. 

The QA manager is himself one of the primary developers of
Bugzilla.

Any serious bugs found by QA are fixed before the release, and
no stable version of Bugzilla is ever released which has not passed QA.

== General Documentation About Bugzilla's Process ==

	There are many other aspects to our process for other areas of
Bugzilla.

* The [http://www.bugzilla.org/developers/ general documentation for Bugzilla developers] describes specific guidelines and processes for different areas of the Bugzilla Project.
* [[Bugzilla:Developers]] and its various links describe, for new contributors, our development process.
* The [http://www.bugzilla.org/docs/developer.html Developer's Guide] describes our code standards.

== Advantages of Open-Source Development ==

In addition to all of the process aspects that I've listed above, there are many benefits to Bugzilla's being developed in the open:

* If you encounter a bug in Bugzilla, and it has been fixed but the release isn't available yet, you can easily get the patch from our bug-tracking system or from our version-control system, both of which are fully available to the public.
* If you encounter an issue that hasn't been fixed, you not only have the code, and can fix it yourself, but you have an entire community of contributors and users who can advise you and help you fix it.
* The primary developers of Bugzilla are available easily and we provide support directly to users. I myself am the most frequent writer on the Bugzilla support mailing list, and nearly all the developers are available through IRC and provide support to the many users who come by. (Note that I myself ''managed'' an entire technical support department as the senior technician for a commercial software company before I started my own business providing Bugzilla services, and I am now one of those people providing support for Bugzilla directly to users on IRC and the mailing list.)
* Bugzilla isn't dependent upon a single entity for support or development. In commercial software, if a firm fails and nobody buys their intellectual property, releases are ended forever. With Bugzilla, even if all the developers were to suddenly disappear, development could be continued by any entity. Witness Linux--in the 1990's, various Linux companies went bankrupt, but other successful companies (like Red Hat) were still able to continue and provide Linux as a product, because it's open source.

The idea that open source is unstable or unreliable is an ancient myth, long disproved by industry experience and evidence. Most of the Internet is run by open-source software--the major DNS services and the largest number of web sites. Major corporations such as Google, Yahoo, and eBay depend on open source software such as Linux and Bugzilla for their business and development processes.

== Conclusion ==

Bugzilla is a high-quality product, developed to exact standards, and will remain that way for the foreseeable future.

And if it didn't, then you could fix it yourself.