B2G/QA/Triage: Difference between revisions
Jump to navigation
Jump to search
Joshwalker (talk | contribs) |
|||
| (122 intermediate revisions by 7 users not shown) | |||
| Line 4: | Line 4: | ||
== Summary of Keywords and Flags == | == Summary of Keywords and Flags == | ||
=== Whiteboard === | |||
Project identifiers such as [red tai] may be used to identify a group of bugs associated with a specific project.<br /> | |||
More detail [[BMO/Whiteboard|click here]] | |||
=== Keywords === | === Keywords === | ||
'''Note: ''' Only one keyword should be used optimally to represent what is needed by QA. | '''Note: ''' Only one keyword should be used optimally to represent what is immediately needed by QA, except for the case of qaurgent. qaurgent is used in conjunction with a QA keyword. | ||
==== steps-wanted ==== | ==== steps-wanted ==== | ||
| Line 19: | Line 23: | ||
==== regressionwindow-wanted ==== | ==== regressionwindow-wanted ==== | ||
When a regression range is necessary to identify a first good build and first bad build for a particular bug. | When a regression range is necessary to identify a first good build and first bad build for a particular bug. This should be flagged only when the following criteria is met: | ||
* Branch checks for a bug are complete (note: this is done to ensure that we truly know that the bug is a regression) | |||
* The bug has been determined to be a nomination or a blocker (note: this is done since windows are expensive to conduct) | |||
* The reproduction rate is greater than or equal to 50% | |||
==== verifyme ==== | ==== verifyme ==== | ||
| Line 27: | Line 35: | ||
==== qawanted ==== | ==== qawanted ==== | ||
When QA support is needed on the bug, but none of the above keywords apply. When this keyword is used, the person flagging qawanted needs to indicate what QA support is needed. | When QA support is needed on the bug, but none of the above keywords apply. When this keyword is used, the person flagging qawanted needs to indicate what QA support is needed. Example requests include: | ||
* Branch checks | |||
* Retesting to see if the issue reproduces on the latest build | |||
* Getting clarification on the reproduction rate of the bug | |||
==== qaurgent ==== | |||
When QA support is urgently needed on a bug for a particular request. Use this only when you think the request has a high severity (e.g. smoketest issue). | |||
====More keywords==== | |||
[https://bugzilla.mozilla.org/describekeywords.cgi click here] | |||
=== Flags === | === Flags === | ||
| Line 37: | Line 56: | ||
== QA Contact == | == QA Contact == | ||
The use of QA contact field should be set optimally when the person is actively investigating something in relation to the bug, usually in connection to one of the keywords above. The QA contact field should aim to only be set by the person who is planning on working on the bug for some piece of QA. | The use of QA contact field should be set optimally when the person is actively investigating something in relation to the bug, usually in connection to one of the keywords above. The QA contact field should aim to only be set by the person who is planning on working on the bug for some piece of QA. | ||
However, there is one exception to this rule. If a high need is raised in triage to find an owner for a bug, then triagers should set the QA contact to the QA representative in triage to find a QA owner to address the QA needs for the bug. If no QA representative is present, then [http://mailto:pyaramada@mozilla.com Pallavi Yaramada] or [http://mailto:atsai@mozilla.com Al Tsai] should be set as the QA contact as a fallback. | |||
== | == QA Requests == | ||
For addressing bugs that need QA involvement with the keywords stated above except verifyme, the table below summarizes the urgency of response we follow for QA requests. | For addressing bugs that need QA involvement with the keywords stated above except verifyme, the table below summarizes the urgency of response we follow for QA requests. | ||
| Line 58: | Line 69: | ||
| '''Bugzilla Flags''' | | '''Bugzilla Flags''' | ||
| '''Query''' | | '''Query''' | ||
|- | |- | ||
| High | | High | ||
| blocking-b2g = | | blocking-b2g = ?, + | ||
| [https://bugzilla.mozilla.org/buglist.cgi? | | [https://bugzilla.mozilla.org/buglist.cgi?v4=%3F&f10=cf_blocking_b2g&o5=equals&o3=anywordssubstr&list_id=11415349&j8=OR&f8=OP&o11=equals&v3=qawanted%20testcase-wanted%20regressionwindow-wanted%20steps-wanted&j2=OR&o9=substring&v10=%3F&resolution=---&resolution=FIXED&f9=cf_blocking_b2g&f4=cf_status_b2g_2_0&v5=%3F&o10=substring&query_format=advanced&v9=%2B&f3=keywords&o4=equals&f2=OP&f11=cf_blocking_b2g&f5=cf_status_b2g_2_1&f7=CP Bug Query] | ||
|- | |- | ||
| Medium | | Medium | ||
| | | blocking-b2g = - | ||
| [https://bugzilla.mozilla.org/buglist.cgi? | | [https://bugzilla.mozilla.org/buglist.cgi?v4=%3F&f10=cf_blocking_b2g&o5=equals&o3=anywordssubstr&list_id=11348248&j8=OR&f8=OP&o11=equals&v3=qawanted%20testcase-wanted%20regressionwindow-wanted%20steps-wanted&j2=OR&o9=equals&resolution=---&resolution=FIXED&f9=cf_blocking_b2g&f4=cf_status_b2g_2_0&v5=%3F&o10=equals&query_format=advanced&v9=-&f3=keywords&o4=equals&f2=OP&f11=cf_blocking_b2g&f5=cf_status_b2g_2_1&f7=CP Bug Query] | ||
|- | |- | ||
| Low | | Low | ||
| Anything else | | Anything else | ||
| [https://bugzilla.mozilla.org/buglist.cgi? | | [https://bugzilla.mozilla.org/buglist.cgi?v4=qawanted%20steps-wanted%20regressionwindow-wanted%20testcase-wanted&o5=equals&f1=cf_blocking_b2g&v6=%3F&list_id=11348249&o1=equals&resolution=---&resolution=FIXED&o6=equals&o2=nowordssubstr&f4=keywords&v5=%3F&query_format=advanced&f3=OP&o4=anywordssubstr&f2=cf_tracking_b2g18&f5=cf_status_b2g_2_1&j3=OR&v1=---&f6=cf_status_b2g_2_0 Bug Query] | ||
|} | |} | ||
'''Note: ''' For | '''Note: ''' For high urgency bugs, we plan to provide an initial response on the bug within two business days. | ||
== Test Case Coverage | == Test Case Coverage Wanted == | ||
=== Summary === | === Summary === | ||
| Line 84: | Line 91: | ||
For requesting test case coverage, we shall make use of the in-moztrap flag. This flag aims to capture when someone thinks test case coverage would be a good for a feature or bug filed in Bugzilla. From a feature perspective for a release, this aims to help capture a secondary checklist to make sure that our QA team has captured test cases for all features for a particular release. However, for feature release test case development, usage of this flag is not supposed to be used for being the sole tracking area - a separate spreadsheet will be maintained for tracking the details of the work being done. For miscellaneous cases needing test case coverage, this process also helps find missing test case coverage and sometimes, can identify untracked features that product did not know about. | For requesting test case coverage, we shall make use of the in-moztrap flag. This flag aims to capture when someone thinks test case coverage would be a good for a feature or bug filed in Bugzilla. From a feature perspective for a release, this aims to help capture a secondary checklist to make sure that our QA team has captured test cases for all features for a particular release. However, for feature release test case development, usage of this flag is not supposed to be used for being the sole tracking area - a separate spreadsheet will be maintained for tracking the details of the work being done. For miscellaneous cases needing test case coverage, this process also helps find missing test case coverage and sometimes, can identify untracked features that product did not know about. | ||
To learn more information about the workflow for test case coverage, see this [https://wiki.mozilla.org/File:Inmoztrapb2g.png diagram] that describes the workflow with the in-moztrap flag. | |||
=== Request Importance === | === Request Importance === | ||
The following table below summarizes the priority for test case coverage requests that come from the in-moztrap flag. | The following table below summarizes the priority for test case coverage requests that come from the in-moztrap flag. | ||
{| class="fullwidth-table" | {| class="fullwidth-table" | ||
| Line 104: | Line 101: | ||
| '''Importance''' | | '''Importance''' | ||
| '''Bugzilla Flags''' | | '''Bugzilla Flags''' | ||
| ''' | | '''in-moztrap? User Story Query''' | ||
| '''Query''' | | '''in-moztrap? General Query''' | ||
| '''Untriaged Query''' | |||
|- | |- | ||
| High | | High | ||
| blocking-b2g = | | blocking-b2g = ?, + | ||
| [https://bugzilla.mozilla.org/buglist.cgi?o5=substring&f1=OP&o3=equals&list_id=7813732&short_desc=User%20Story&o2=anywordssubstr&f4=CP&v5=in-moztrap%3F&query_format=advanced&j1=OR&f3=cf_blocking_b2g&f2=cf_blocking_b2g&short_desc_type=allwordssubstr&f5=flagtypes.name&v2=%3F%20%2B Bug Query] | |||
| [https://bugzilla.mozilla.org/buglist.cgi? | | [https://bugzilla.mozilla.org/buglist.cgi?o5=substring&f1=OP&o3=equals&list_id=7813765&o2=anywordssubstr&f4=CP&v5=in-moztrap%3F&query_format=advanced&j1=OR&f3=cf_blocking_b2g&f2=cf_blocking_b2g&f5=flagtypes.name&v2=%3F%20%2B Bug Query] | ||
|- | | [https://bugzilla.mozilla.org/buglist.cgi?f1=OP&f2=cf_blocking_b2g&f3=cf_blocking_b2g&f4=CP&f5=flagtypes.name&j1=OR&list_id=7813772&o2=anywordssubstr&o3=equals&o5=notsubstring&query_format=advanced&v2=%3F%20%2B&v5=in-moztrap&order=bug_id&limit=0 Bug Query] | ||
| [https://bugzilla.mozilla.org/buglist.cgi?f1=cf_blocking_b2g&list_id= | |||
|- | |- | ||
| Low | | Low | ||
| | | Anything Else | ||
| | | [https://bugzilla.mozilla.org/buglist.cgi?f1=OP&v6=in-moztrap%3F&o3=substring&list_id=7813810&short_desc=User%20Story&o6=substring&o2=nowordssubstr&f4=cf_tracking_b2g18&query_format=advanced&j1=OR&f3=flagtypes.name&o4=equals&f2=cf_blocking_b2g&short_desc_type=allwordssubstr&f5=CP&f6=flagtypes.name&v2=%3F%20%2B Bug Query] | ||
| [https://bugzilla.mozilla.org/buglist.cgi? | | [https://bugzilla.mozilla.org/buglist.cgi?f1=OP&v6=in-moztrap%3F&o3=substring&list_id=7813831&o6=substring&o2=nowordssubstr&f4=cf_tracking_b2g18&query_format=advanced&j1=OR&f3=flagtypes.name&o4=equals&f2=cf_blocking_b2g&f5=CP&f6=flagtypes.name&v2=%3F%20%2B&product=Boot2Gecko Bug Query] | ||
| [https://bugzilla.mozilla.org/buglist.cgi?f1=OP&f2=cf_blocking_b2g&f3=flagtypes.name&f4=cf_tracking_b2g18&f5=CP&f6=flagtypes.name&j1=OR&list_id=7813841&o2=nowordssubstr&o3=substring&o4=equals&o6=notsubstring&product=Boot2Gecko&query_format=advanced&v2=%3F%20%2B&v6=in-moztrap&order=bug_id&limit=0 Bug Query] | |||
|} | |} | ||
== | === Test Case Prioritization === | ||
When creating the test cases, please tag them with the following prioritization levels within MozTrap: | |||
* P1 - Feature smoketests | |||
* P2 - Feature basic functional tests | |||
* P3 - Boundary, edge, stress test cases | |||
* P4 - Anything else that doesn't fall in the above buckets | |||
== Verifications == | |||
For | For addressing bugs that need verification, our QA team will follow a manual process of analyzing resolved fixed bugs, determining which bugs need verification, and verifying those bugs on the appropriate branches. If someone internal or external to the team knows that a bug is worthwhile verifying, then they will flag the bug as verifyme. The following table below summarizes what bugs should be verified and in what order. | ||
{| class="fullwidth-table" | {| class="fullwidth-table" | ||
|- | |- | ||
| ''' | | '''Importance''' | ||
| ''' | | '''Bugzilla Flags''' | ||
| '''Query''' | |||
|- | |- | ||
| | | Very High | ||
| | | blocking-b2g = 2.1 + & verifyme | ||
| [https://bugzilla.mozilla.org/buglist.cgi?j_top=OR&keywords=verifyme%2C%20&keywords_type=allwords&f1=cf_blocking_b2g&list_id=11348272&o1=anywordssubstr&o2=equals&query_format=advanced&f2=cf_blocking_b2g&v1=2.1%2B Bug Query] | |||
|- | |- | ||
| | | High | ||
| | | blocking-b2g = 2.1+, status-b2g-v1.3 fixed, no qa-, no [Qanalyst-Triage-] | ||
| [https://bugzilla.mozilla.org/buglist.cgi?o5=notregexp&f1=cf_status_b2g_2_1&v6=QAnalyst-Triage-&o3=notregexp&list_id=11348294&o1=equals&resolution=FIXED&o6=notsubstring&o2=equals&v5=qa-&query_format=advanced&f3=status_whiteboard&f2=cf_blocking_b2g&f5=status_whiteboard&v1=fixed&f6=status_whiteboard&v2=2.1%2B Bug Query] | |||
|- | |- | ||
| | | Medium | ||
| | | blocking-b2g = - & verifyme | ||
| [https://bugzilla.mozilla.org/buglist.cgi?keywords=verifyme%2C%20&f1=cf_blocking_b2g&keywords_type=allwords&list_id=7813949&o1=equals&query_format=advanced&v1=- Bug Query] | |||
|- | |- | ||
| | | Low | ||
| blocking-b2g = --- & verifyme | |||
|- | | [https://bugzilla.mozilla.org/buglist.cgi?keywords=verifyme&f1=cf_blocking_b2g&keywords_type=allwords&list_id=11348291&o1=equals&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&v1=--- Bug Query] | ||
| | |||
|} | |} | ||
Latest revision as of 14:53, 12 July 2015
Overview
This document provides a summary of how we manage requests for QA support through bugzilla, more specifically for the B2G QA team.
Summary of Keywords and Flags
Whiteboard
Project identifiers such as [red tai] may be used to identify a group of bugs associated with a specific project.
More detail click here
Keywords
Note: Only one keyword should be used optimally to represent what is immediately needed by QA, except for the case of qaurgent. qaurgent is used in conjunction with a QA keyword.
steps-wanted
When reproducible steps to reproduce are needed for a certain bug.
testcase-wanted
When a reduced test case is needed for a certain bug.
regressionwindow-wanted
When a regression range is necessary to identify a first good build and first bad build for a particular bug. This should be flagged only when the following criteria is met:
- Branch checks for a bug are complete (note: this is done to ensure that we truly know that the bug is a regression)
- The bug has been determined to be a nomination or a blocker (note: this is done since windows are expensive to conduct)
- The reproduction rate is greater than or equal to 50%
verifyme
When a verification of a bug is needed to ensure that the patch on the bug actually fixed the problem.
qawanted
When QA support is needed on the bug, but none of the above keywords apply. When this keyword is used, the person flagging qawanted needs to indicate what QA support is needed. Example requests include:
- Branch checks
- Retesting to see if the issue reproduces on the latest build
- Getting clarification on the reproduction rate of the bug
qaurgent
When QA support is urgently needed on a bug for a particular request. Use this only when you think the request has a high severity (e.g. smoketest issue).
More keywords
Flags
in-moztrap
When a bug or feature could benefit from having formal test case coverage. See below for a more detailed discussion on the workflow of using this flag.
QA Contact
The use of QA contact field should be set optimally when the person is actively investigating something in relation to the bug, usually in connection to one of the keywords above. The QA contact field should aim to only be set by the person who is planning on working on the bug for some piece of QA.
However, there is one exception to this rule. If a high need is raised in triage to find an owner for a bug, then triagers should set the QA contact to the QA representative in triage to find a QA owner to address the QA needs for the bug. If no QA representative is present, then Pallavi Yaramada or Al Tsai should be set as the QA contact as a fallback.
QA Requests
For addressing bugs that need QA involvement with the keywords stated above except verifyme, the table below summarizes the urgency of response we follow for QA requests.
| Urgency | Bugzilla Flags | Query |
| High | blocking-b2g = ?, + | Bug Query |
| Medium | blocking-b2g = - | Bug Query |
| Low | Anything else | Bug Query |
Note: For high urgency bugs, we plan to provide an initial response on the bug within two business days.
Test Case Coverage Wanted
Summary
For requesting test case coverage, we shall make use of the in-moztrap flag. This flag aims to capture when someone thinks test case coverage would be a good for a feature or bug filed in Bugzilla. From a feature perspective for a release, this aims to help capture a secondary checklist to make sure that our QA team has captured test cases for all features for a particular release. However, for feature release test case development, usage of this flag is not supposed to be used for being the sole tracking area - a separate spreadsheet will be maintained for tracking the details of the work being done. For miscellaneous cases needing test case coverage, this process also helps find missing test case coverage and sometimes, can identify untracked features that product did not know about.
To learn more information about the workflow for test case coverage, see this diagram that describes the workflow with the in-moztrap flag.
Request Importance
The following table below summarizes the priority for test case coverage requests that come from the in-moztrap flag.
| Importance | Bugzilla Flags | in-moztrap? User Story Query | in-moztrap? General Query | Untriaged Query |
| High | blocking-b2g = ?, + | Bug Query | Bug Query | Bug Query |
| Low | Anything Else | Bug Query | Bug Query | Bug Query |
Test Case Prioritization
When creating the test cases, please tag them with the following prioritization levels within MozTrap:
- P1 - Feature smoketests
- P2 - Feature basic functional tests
- P3 - Boundary, edge, stress test cases
- P4 - Anything else that doesn't fall in the above buckets
Verifications
For addressing bugs that need verification, our QA team will follow a manual process of analyzing resolved fixed bugs, determining which bugs need verification, and verifying those bugs on the appropriate branches. If someone internal or external to the team knows that a bug is worthwhile verifying, then they will flag the bug as verifyme. The following table below summarizes what bugs should be verified and in what order.
| Importance | Bugzilla Flags | Query |
| Very High | blocking-b2g = 2.1 + & verifyme | Bug Query |
| High | blocking-b2g = 2.1+, status-b2g-v1.3 fixed, no qa-, no [Qanalyst-Triage-] | Bug Query |
| Medium | blocking-b2g = - & verifyme | Bug Query |
| Low | blocking-b2g = --- & verifyme | Bug Query |