MozillaWiki

MDN/QA/testplan: Difference between revisions

Page Discussion

MDN/QA/testplan (view source)

Revision as of 16:44, 24 June 2016

2,076 bytes added ,  24 June 2016
m
→‎2016 QA Priorities
 
(48 intermediate revisions by 2 users not shown)
Line 1: Line 1:
We are the [https://developer.mozilla.org/en-US/'''MDN'''] (Mozilla Developer Network) durable team.  MDN's mission is to provide complete, accurate, and helpful documentation for everything about the open Web, whether it's supported by Mozilla-built software or not.  Read more about our [https://developer.mozilla.org/en-US/docs/MDN/About mission].
For additional information about the project team [https://wiki.mozilla.org/Engagement/MDN_Durable_Team visit here].
See what the team is [https://mozilla.kanbanery.com/projects/68288/board/?key=775e5e65ceb38d40d0dc1a4e481901387d08abdft currently working on]
==2016 QA Priorities==
* Overview and article pages behave as expected.
* Editing pages is a consistent and user friendly experience.
* Feedback mechanisms on MDN exist and work.
* Localized experiences persist when a user arrives, uses, and leaves the site.
* The search experience is consistent.
==Open Questions==
==Open Questions==


* How do we prevent loss of polish over-time?
* Lack of test data - Stage currently isn't sync'd with production data, the database schema is. How can we solve this?
** For example:
* Is there telemetry in place for Kuma script rendering errors
*** CSS tweaks that have unnoticed and unintended consequences.
*** User workflows that don't make sense.
* Data integrity
* Data integrity
** How is this maintained, persisted, prevented from regressing?
** How is this maintained, persisted, prevented from regressing?
** How is user content quality maintained & updated for relevancy over time?
** How is user content quality maintained & updated for relevancy over time?
* Are there bugzilla search flows that can be created for triaging
** What needs manual verification?
** Where test coverage is lacking?
* How is technical test debt tracked?
* Reliability discussion - https://etherpad.mozilla.org/MDN-reliability-plan
* Reliability discussion - https://etherpad.mozilla.org/MDN-reliability-plan
* l10n testing?
* l10n testing
* Page load times
* Page load times


Line 32: Line 39:
===What are QA’s responsibilities===
===What are QA’s responsibilities===


* Acting as a user advocate
* Acting as a user advocate.
* Leading discussions that identify and assign risk to user stories
* Leading end of sprint post-mortem discussions that identify and realign the risk areas that need to be mitigated.
* Driving exploratory testing efforts around feature areas that represent high risk
* Moving user stories (acceptance criteria) to TestRail
* With a focus on community and developer inclusion; collaborative creation and maintenance of end-to-end test automation.


===What are the dev team’s responsibilities===
===What are the dev team’s responsibilities===
Line 42: Line 50:
* Maintaining code quality via code review and documentation of standards
* Maintaining code quality via code review and documentation of standards


==ToDO Complete a risk assessment==
==Complete a risk assessment==


* The team discusses and agrees on a set of risks that they are comfortable with assuming. Assumed risk isn’t a static conversation, but it is good to build a skeleton that the team can agree on and enhance over time.
* Stage - schema is up-to-date, data is 9 months out-of-date
* Dev environments are significantly different than stage and production
* Removal of low value features - tracking of feature usage and sunsetting of features that don't see uptake.


* The product should be broken into user stories so a team can apply a hierarchy of risk. It is important to identify and engage as many stakeholders as possible, each individual will have a unique and important point of view.
==User stories and features are grouped into three categories==
 
* The list of stakeholders often include; project managers, developers, testers, UX, IT, DBAs, the security team, etc. Sometimes the product has high enough visibility in the organization that a proxy for the CEO is beneficial.
User stories and features are grouped into three categories


===Cannot regress in production===
===Cannot regress in production===


* Features that users should never see broken in production (in order of throughput/traffic)
* Features that users should never see broken in production (in order of throughput/traffic)
** View a doc article (Zones & non-zones)
** Edit a doc article
*** Edit a doc article
** Translate a doc article
*** Translate a doc article
** View the home page
** View the home page
** Log in
** Log in
** View a user profile
** Search docs
** Search docs
** View a demo
*** Topic pages and articles render
** Ensure we don't leak PII
** Localization persists


===Can regress in production for a finite amount of time===
===Can regress in production for a finite amount of time===
Line 79: Line 84:
An important note about feature verification
An important note about feature verification


If a feature is deemed as low risk and it's functionality is easy to verify, anyone on the team is empowered to test and verify it. If it is a feature that requires deeper investigation and poses high risk, the QA team is the group who verifies it. If a feature set is big or user workflow is changed it is important to engage the test team to flush out defects and usability concerns.
If a feature is deemed as low to medium risk, anyone on the team is empowered to test and verify it. If it is a feature that requires deeper investigation and poses high risk, the team can reach out to QA and request our input. If a feature set is big or user workflow is changed it is important to engage the test team to flush out defects and usability concerns.


* Features sent to QA for manual verification must include Bug numbers
* Features sent to QA for manual verification must include Bug numbers.
* Important workflows that need to be tested will live in [http://testrail.stage.mozaws.net/index.php?/dashboard TestRail] TODO: update url to prod testrail once it's launched.
** Workflows that live in TestRail are candidates for automation.


==Test Automation==
==Test Automation==


===Blockers===
Currently, writing and maintenance of the tests written using Intern are owned by the developer team.
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1186081 Bug 1186081] - [meta] Implement load-testing for MDN
* This test runner and the test suites will be phased out in Q3 of 2016.
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1184662 Bug 1184662] - Update intern tests to run reliably
* Test will be rewritten in Python and use pytest and pytest-selenium
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1184670 Bug 1184670] - Automatically run Intern test suite against stage & prod servers after deployments]
* Test ownership and maintenance will be expanded to include QA.


===Overview===
===Overview===
Line 98: Line 105:
A component of the automation strategy includes reviewing the amount of test coverage that follows a developers pull request. Often times the team is able to influence and enhance the depth of coverage by simply asking for more tests earlier in the workflow [Unit + Integration].
A component of the automation strategy includes reviewing the amount of test coverage that follows a developers pull request. Often times the team is able to influence and enhance the depth of coverage by simply asking for more tests earlier in the workflow [Unit + Integration].


===TODO Coverage Areas===
===Coverage Areas===
* Topic and article pages behave as expected.
* Editing pages is a consistent and user friendly experience.
* Feedback mechanisms on MDN exist and work.
* Localized experiences persist when a user arrives, uses, and leaves the site.
* The search experience is consistent.


====TODO Environment/test matrix====
====TODO Environment/test matrix====
* Linux?
* Mobile?
* Windows 8
** IE
* Windows 10+
** IE
** Firefox latest
** Chrome
** Opera?
** Brave?
* OSX
** Firefox latest
** Safari
** Chrome
** Opera?
** Brave?


====Unit tests====
====Unit tests====
* Run by [https://travis-ci.org/mozilla/kuma TravisCI]
=====Blocks Build=====
* Run by TravisCI
** Unit tests
** Flake8
** Sphinx Docs build
** Dennis


====Functional UI tests - theIntern.js====
=====Non-Blocking=====
* Run by TravisCI
** [https://codecov.io/github/mozilla/kuma?branch=master Code Coverage]
** [https://requires.io/github/mozilla/kuma/requirements/?branch=master Requires.io]
* Functional UI Tests
** Run by developers on their local machines - [https://github.com/mozilla/kuma/tree/master/tests/ui kuma/tests/ui/]
 
====Functional UI tests====
* Written in JS -  https://github.com/mozilla/kuma/tree/master/tests/ui
* Q3 2016 - move tests to Python


====End-to-end tests====
====End-to-end tests====


* Integration tests
* Integration tests - work will begin late Q2 and through Q3
* API tests?
* API tests - none planned


==== Stress and load testing====
==== Stress and load testing====


* locust.io? Loads v2?
* None planned for Q2
* https://etherpad.mozilla.org/MDN-load-testing
*  Create and run stress test (based on the catalog) on MDN staging to learn max load limit of MDN code https://bugzilla.mozilla.org/showdependencytree.cgi?id=1186081
* Convert stress test into load test for use after stage deployments https://trello.com/b/p56Gwq46/mdndev-rocks
* https://bugzilla.mozilla.org/showdependencytree.cgi?id=1182182


====Monitoring tools====
====Monitoring tools====


*New Relic? DataDog?
* New Relic
** [https://rpm.newrelic.com/accounts/263620/applications/3172075 MDN Prod]
 
* Sentry
** [https://sentry.prod.mozaws.net/operations/mdn-stage/ MDN Stage]
** [https://sentry.prod.mozaws.net/operations/mdn-prod/ MDN Prod]
 
* Spam
** [https://groups.google.com/a/mozilla.com/forum/#!forum/mdn-spam-watch Spam watch mailing list]
** [http://53b54fe99233.web.akismet.com/1.0/user-stats.php?api_key=53b54fe99233&blog=developer.mozilla.org&show=60-day Akismiet stats]
** [https://developer.allizom.org/en-US/dashboards/revisions Revision dashboard]


===Style Guides===
===Style Guides===


* Link to style guides
* Link to style guides
** code quality, linting tools, etc
** [https://wiki.mozilla.org/QA/Execution/Web_Testing/Docs/Automation/StyleGuide Functional tests]
** What are the best practices for writing Intern tests?
** [https://wiki.mozilla.org/QA/Execution/Web_Testing/Automation/Flake8_Pre_Commit_Hook Linting with Flak8]
** Best practices for writing API endpoint tests?
** [https://wiki.mozilla.org/QA/Execution/Web_Testing/Automation Additional test automation documentation]


==One-time Events==
==One-time Events==
Line 195: Line 247:


* For less time sensitive testing tasks we can use [https://oneanddone.mozilla.org/ oneanddone.org] as a task manager to aid in funneling community members into our onboarding flows.
* For less time sensitive testing tasks we can use [https://oneanddone.mozilla.org/ oneanddone.org] as a task manager to aid in funneling community members into our onboarding flows.
==Other==
* [https://docs.google.com/document/d/1sYpvaDaYnXSTZz2-zQUYt5KYLHsamNcaj6tkLC6y4EY/ Spam administration documentation]
Mbrandt
Confirmed users
1,867

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1090535...1137559"