MozillaWiki

B2G/QA/Process: Difference between revisions

Page Discussion

B2G/QA/Process (view source)

Revision as of 07:56, 17 March 2016

6,409 bytes added ,  17 March 2016
→‎Marigold
 
(21 intermediate revisions by 3 users not shown)
Line 3: Line 3:


=== Mozilla QA Process Overview ===
=== Mozilla QA Process Overview ===
Please visit [https://developer.mozilla.org/en-US/docs/Mozilla/QA MDN QA page] for an overview of quality assurance at Mozilla.
Please visit [https://developer.mozilla.org/en-US/docs/Mozilla/QA MDN QA page] for an overview of quality assurance activities at Mozilla.


=== B2G Roadmap ===
=== B2G Roadmap ===
Line 21: Line 21:
[https://wiki.mozilla.org/B2G/QA/Device_Test_Plan/Graphics Graphics Test Plan]<br>
[https://wiki.mozilla.org/B2G/QA/Device_Test_Plan/Graphics Graphics Test Plan]<br>


====[[Modules/FirefoxOS|List of Firefox OS Modules/Dev Owners]]====
'''[[Modules/FirefoxOS|List of Firefox OS Modules/Dev Owners (link)]]''' in case you need to ask questions regarding the new features of the component.<br>
In case you need to ask questions regarding the new features of the component.
[[Firefox_OS/functionalteams|'''Firefox OS functional teams breakdown (link)''']]<br>
[[FirefoxOS/FeatureTesting|'''QA owners for Firefox OS components (link)''']]


=== B2G Test Process ===
=== B2G Test Process ===
Please review the following beforehand:
* [[Release_Management/FirefoxOS/Release_Milestones|B2G Release Milestones]]
* [[Release_Management/B2G_Landing|B2G Landing Process]]
* [[Release_Management/FirefoxOS/Release_Milestones#Feature_Landing_Criteria|Release Landing Criteria]]
Before the B2G release, following QA activities should be completed:
* Go through the list of new features (check for feature-b2g flag in Bugzilla), and assess for the creation of new test cases and update/deprecate existing test cases as necessary
* Create and update test cases in Moztrap
* Execute regression test suite, incorporated with the new test cases, and verify all release blocker bugs are fixed
* Ensure the builds are passing daily smoketests
* Execute performance/stability test suite


=== Smoke Test / Dogfood Processes ===
* [[B2G/QA/BuildProcess|Smoke Test, Dogfood, Base Build Process]]


=== Status Tracking ===
* [http://mozilla-twqa.github.io/QA-Dashboard/ffos/frame_features.html Features on QA Dashboard]
* [http://mozilla-twqa.github.io/QA-Dashboard/ffos/frame_bugs.html Bugs on QA Dashboard]
** How to use: put [COM=NFC] in 'QA whiteboard', the bug shows up nicely to the count of NFC box, feature page filters for feature-b2g flag, bug page filters for blocking-b2g flag.
* [http://charts.mozilla.org/ Mozilla Charts]
* [[FirefoxOS/DeviceQA#Meeting_Minutes|Meeting minutes]]
=== [[B2G/QA/Bugzilla|Bugzilla Howtos (link)]] ===
=== Moztrap Howtos ===
* '''[https://quality.mozilla.org/docs/test-case-writing-primer Test case creation primer]'''
* '''[https://wiki.mozilla.org/QA/Execution/MozTrap Moztrap General]'''
* '''[https://wiki.mozilla.org/FirefoxOS/MoztrapGuideline Moztrap Guideline for Firefox OS]'''
* '''[https://addons.mozilla.org/EN-uS/firefox/addon/moztraphelper MozTrapHelper by Shing]''' - recommend to add this to your Firefox browser.
=== Triage ===
Refer to [[B2G/QA/Triage|this page]] for the detailed information on how we use Bugzilla for triage process. <br>
Refer to [[B2G/Triage|this page]] for B2G triage process.
=== When to Block a Bug for Release ===
'''Should Block'''
* Features our product team has committed us to
* Major issue in new feature - especially those in which a large number of users will be impacted, or a smaller number of users will be significantly impacted
* Major identifiable regression (perf or otherwise)
* Non-localizable strings
* Top Crashes
* sec-high, sec-critical Security bugs
* Smoke-test regression (special-case insta-blocker)
* Data loss
* Issues that block partner certification (basically legal issues)
* Issues getting a lot of support calls with partners or on SUMO (exceptions need to be made after FC though)
* Issues critical around updates (especially if there's been a repro)
* Anything critical around the first time experience
* Major Dialer, SMS, and VM communication issues (lower bar than other blockers - it's a phone)
* Issues that prevent automated tests in established testsuites (visible test suites on b2g integration branches on TBPL) from running green at least 90% of the time.
* Certification Waivers
'''Should Not Block''' - any exceptions to these rules must be discussed on b2g-release-drivers@mozilla.org or with Release Management
* Enhancements
* New Features (Not committed for the release)
* New perf requirements (see enhancements)
* Non-critical string changes (think of the l10n, people)
* Polish and other minor issues
* Unfinished localization (except perhaps in the last 3 weeks)
* Issues requiring the user to "zoom in and out and in and out and in and out..."
* Issues in languages not being shipped in the version of B2G
* Bugs without clear STR or that are not reproducible
* Bugs that do not impact production phones or the simulator
'''Needs Decision'''
* Non-regression bugs that we chose not to block on in the last release
* (though, sometimes this is because partner said "we won't block this time if you fix in the next release)


=== Qanalysts Tasks ===
=== Qanalysts Tasks ===
Following documents describe the tasks assigned to QAnalyst:<br>
(If you cannot view these documents, ask ktucker for permission in #fxosqa.  You must have a valid mozilla LDAP account)
* [https://docs.google.com/a/mozilla.com/document/d/1pVLh0sKuC6q6d1rBmJt9umlygY9bz_2qqOudm7uoLJU/edit?usp=sharing_eid&ts=5642879f QAnalyst Daily Tasks]
** [[B2G/QA/Gaia_Test_Plan/Email_Templates|Email Template for the daily report]]
* [https://docs.google.com/a/mozilla.com/document/d/19St0uLtBSDG6BILk2trEWTbq9N6E-OBOwTghMz9sztk/edit?usp=sharing_eid&ts=56426b8e Jenkins Reporting]
* [https://docs.google.com/a/mozilla.com/document/d/134G51KAVV0l_0XiEcYuk6h5vo0lnjQEIY_LyvbxEKro/edit?usp=sharing_eid&ts=56426bcd Automation Reporting]
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1tvOk6Jw6beZodjpQw2e9zbVHA-tec43VpBBg4nde_Eo/edit?usp=sharing_eid&ts=56426c54 Exploratory Tests]
* [https://docs.google.com/document/d/17_zO4ACFGblNOtV7ixenMr9f8zWEv_RzXpPcWFrEQ7Q/edit?ts=56426c94#heading=h.tejavodcjazn Exploratory Tests with Resolved Fixes]
* [https://docs.google.com/a/mozilla.com/document/d/1UmtV6IQzddV_1JZhGI5CboKJy2OcyrKqSOrX7-k52GI/edit?usp=sharing_eid&ts=56426d87 Getting Regression Window]
* [https://docs.google.com/a/mozilla.com/document/d/1Rf0i57kN9Fof_HgTKnnqZ3ADD0htSfploHCvGGBaL7o/edit?usp=sharing_eid&ts=56426dcd Executing Automation]
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1RhI81B01gzLYTeRVqFM8RC2_UiuGtFssGtX_pRycgqE/edit?usp=sharing_eid&ts=5642834b Application Coverage Spreadsheet]
=== Marigold ===
==== Overview ====
Marigold (Cheng-Du, China) is Mozilla's QA outsource team. They were in charge of executing FxOS tests, and they covers following tasks requested by Mozilla.
(You may need a valid Mozilla LDAP account to access following documents)


=== Bugzilla Howtos ===
* Manual Test
** Daily smoke test
** Regression test
** Full test
** RAT test
** Bug work (qawanted,regressionwindow-wanted,verifyme)
* Power consumption measurement
* Test automation - Gaia-UI-test
* Patch verification


=== Triage Process ===
==== Management ====
* [http://mzl.la/1xHcWOp Bugzilla:Need-Info List]
* [https://goo.gl/k5STRA Project Assignment and Schedule]
* [https://goo.gl/aBtobD Resource and Devices]
* [https://goo.gl/u604ZU Meeting minutes]
* Weekly Status/Report
** [https://goo.gl/mnfAms 2015 Q1 ~ 2015 Q3]
** [https://goo.gl/dAPcEt 2015 Q4]
Whsu
Confirmed users
452

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1105886...1122586"