B2G/QA/Process

From MozillaWiki
< B2G‎ | QA
Jump to: navigation, search

QA Process

This page lists the relevant information to B2G QA processes.

Mozilla QA Process Overview

Please visit MDN QA page for an overview of quality assurance activities at Mozilla.

B2G Roadmap

Here is the overview of the current Firefox OS product roadmap. Check this page for new information regarding future releases.

B2G Test Planning

UX Specification for Each Release

2.5 UX Specs
2.6 UX Specs

B2G Test Plans

Note: Some information may be out of date, in which case you can notify the QA owner of the affected component(s).
New Device Test Plan
Email Test Plan
Browser Test Plan
Crash Reporting Test Plan
Graphics Test Plan

List of Firefox OS Modules/Dev Owners (link) in case you need to ask questions regarding the new features of the component.
Firefox OS functional teams breakdown (link)
QA owners for Firefox OS components (link)

B2G Test Process

Please review the following beforehand:

Before the B2G release, following QA activities should be completed:

  • Go through the list of new features (check for feature-b2g flag in Bugzilla), and assess for the creation of new test cases and update/deprecate existing test cases as necessary
  • Create and update test cases in Moztrap
  • Execute regression test suite, incorporated with the new test cases, and verify all release blocker bugs are fixed
  • Ensure the builds are passing daily smoketests
  • Execute performance/stability test suite

Smoke Test / Dogfood Processes

Status Tracking

Bugzilla Howtos (link)

Moztrap Howtos

Triage

Refer to this page for the detailed information on how we use Bugzilla for triage process.
Refer to this page for B2G triage process.

When to Block a Bug for Release

Should Block

  • Features our product team has committed us to
  • Major issue in new feature - especially those in which a large number of users will be impacted, or a smaller number of users will be significantly impacted
  • Major identifiable regression (perf or otherwise)
  • Non-localizable strings
  • Top Crashes
  • sec-high, sec-critical Security bugs
  • Smoke-test regression (special-case insta-blocker)
  • Data loss
  • Issues that block partner certification (basically legal issues)
  • Issues getting a lot of support calls with partners or on SUMO (exceptions need to be made after FC though)
  • Issues critical around updates (especially if there's been a repro)
  • Anything critical around the first time experience
  • Major Dialer, SMS, and VM communication issues (lower bar than other blockers - it's a phone)
  • Issues that prevent automated tests in established testsuites (visible test suites on b2g integration branches on TBPL) from running green at least 90% of the time.
  • Certification Waivers

Should Not Block - any exceptions to these rules must be discussed on b2g-release-drivers@mozilla.org or with Release Management

  • Enhancements
  • New Features (Not committed for the release)
  • New perf requirements (see enhancements)
  • Non-critical string changes (think of the l10n, people)
  • Polish and other minor issues
  • Unfinished localization (except perhaps in the last 3 weeks)
  • Issues requiring the user to "zoom in and out and in and out and in and out..."
  • Issues in languages not being shipped in the version of B2G
  • Bugs without clear STR or that are not reproducible
  • Bugs that do not impact production phones or the simulator

Needs Decision

  • Non-regression bugs that we chose not to block on in the last release
  • (though, sometimes this is because partner said "we won't block this time if you fix in the next release)

Qanalysts Tasks

Following documents describe the tasks assigned to QAnalyst:
(If you cannot view these documents, ask ktucker for permission in #fxosqa. You must have a valid mozilla LDAP account)

Marigold

Overview

Marigold (Cheng-Du, China) is Mozilla's QA outsource team. They were in charge of executing FxOS tests, and they covers following tasks requested by Mozilla. (You may need a valid Mozilla LDAP account to access following documents)

  • Manual Test
    • Daily smoke test
    • Regression test
    • Full test
    • RAT test
    • Bug work (qawanted,regressionwindow-wanted,verifyme)
  • Power consumption measurement
  • Test automation - Gaia-UI-test
  • Patch verification

Management

Retrieved from "https://wiki.mozilla.org/index.php?title=B2G/QA/Process&oldid=1122586"