QA/qawanted proposal

Overview

This is a proposal for aligning QA usage of the Action Request keywords associated with testing in Bugzilla.

Current Situation

Over time, we have defined a number of keywords in Bugzilla for requesting testing or test-related actions on a bug. They include the following:

qawanted, in particular, has been used inconsistently.

In the Bugzilla definitions, qawanted is a catchall for any testing-related work and overlaps with the other keywords. Any given request might use a specific keyword or just qawanted. There is no differentiation between an urgent request for the internal QA team and a request to be left open for the larger community.

For the Desktop and Mobile teams, qawanted is used in combination along with another testing request keyword to mean that it's an urgent request for the internal QA team. All others are left open for the community. A request that isn't urgent and which isn't one of the other defined tasks might have no keyword at all.

For Firefox OS team, qawanted means a miscellaneous testing request that no other keyword covers. Urgency is determined by blocking flags with an assumption that everything is handled internally, and there's no little to no concept of a request that's left available for the community.

Issues With the Current Situation

This divergence in usage of qawanted leads to issues:

• Shared components (DOM, Apps, Marketplace) have keyword collisions. Multiple teams cover these components, and they expect the same keywords to mean different things.

• Triagers and developers need to know multiple sets of rules, even where collisions don't exist. As the same people frequently cover multiple projects, they have to switch between confusing semantics that are almost but not quite the same.

• There is no consistent way for community members to search for testing tasks. For Desktop and Mobile teams, many requests are only in the comments and have no keyword; these cannot be queried. Firefox OS doesn't have the community concept represented in their keyword strategy at all.

• Differentiated queries for specific actions aren't reliable, as requests are present with either no keyword or the qawanted catchall. Understanding request load requires parsing comments.

• Cross-product queries for dashboarding are impossible, as the keywords don't mean the same thing across products.

Proposed Changes

• Any outstanding testing request will be expected to have at least one request keyword on it so that it can be queried.

• qaurgent will be introduced as a new keyword to request direct involvement by the Mozilla QA staff and will be used in conjunction with one of the other existing keywords.

• qawanted will be defined as an miscellaneous request for the QA involvement not covered by any of the other existing keywords.

• Bugzilla's keyword definitions will be updated to reflect this proposal.

• All Product QA teams will adopt these semantics, though the specific commitment to qaurgent response time will vary contextually between team, product, and even release, as negotiated with other stakeholders. qawanted will no longer be used to indicate an urgent response is required and should not be used in conjunction with other QA related keywords.

How Proposed Changes Address Issues

• There will be no collision between keywords; semantics will be consistent no matter which team.

• Triagers and other external parties only have to learn one set of keywording rules for testing requests.

• We can create and publish queries for the community to search for all outstanding testing-related requests.

• We can create differentiated queries for every type of request, allowing us to process them at different urgencies as appropriate.

• We can dashboard across products to understand outstanding request load better.

Transition Issues

• Legacy bugs with request keywords will continue to show up in queries, as is appropriate.

• Existing qawanted keyword bugs can remain as is without being impacted by the new qaurgent keyword

• Initially, we will likely have to re-keyword new bugs as we educate external parties on how to use the new qaurgent keyword as well as the use specific requests. Faster turnaround can be promoted as an incentive for them learning the scheme.

• Firefox OS can continue to honor the current commitment that a blocking flag + a request keyword is an implicit qaurgent request, while in transition. An appropriate time between releases would be chosen to fully transition.