Process

A few approaches were attempted in order to gather metrics. The first idea was to check for trends in b2g bugs marked with the whiteboard tag fromAutomation vs. those not within Gaia components (112 bugs). The problem with this is that many bugs may get triaged into other components or products such as Core.. Expanding the search to all products would of course include non-b2g related bugs and so did not work out. Using the smoketest tag yielded many more results (444 bugs), but when restricted to only relevant Web QA contributors it was reduced to 12 bugs. Finally we decided on a query that would include only bugs submitted by the previously mentioned contributors in addition to whether the b2g-blocking field was empty or not. This means that bugs must have been triaged first, and would guarantee b2g relevance. This resulted in 78 bugs. These ranged from March 2013 to June 2014. This feels incomplete as testing began in November 2012, but it was at least confirmed relevant to the metrics at hand.

Interpretation of Data

The method for analysis was to compare valid bugs vs. invalid ones (WORKSFORME, INVALID, WONTFIX, etc.). The following is a bar graph showing raw bug count over time as well as a filled graph showing percentage of valid vs. invalid bugs. Blue is valid, Orange is invalid.

Figure 1: Raw Total of B2G Bugs Submitted Over Time

Figure 2: Percentage of Invalid vs Valid Bugs Submitted Over Time

Using simple linear regression general trends in valid and invalid bugs can be found:

Figure 3: Linear Regression Analysis used to find best fit lines

It was found that while the number of valid bugs reported increased, the number of invalid bugs remained relatively the same. In previous calculations that included non-b2g data, a similar trend was seen where reporting of valid bugs saw a higher growth than invalid bugs. In general though there seems to be no relation between valid and invalid bugs over time.

Takeaways

There needs to be more consistency in how bugs are tagged such that relevant b2g bugs can be easily queried in the future. The blocking-b2g field is a good place to start as long as it is used consistently. Bug reporting also has improved and grown over time as B2G becomes a more mature product and the tools and people become more acquainted with the platform.