352
edits
QA/Mobile/FocusAndroidTestPlan: Difference between revisions
QA/Mobile/FocusAndroidTestPlan (view source)
Revision as of 20:42, 27 December 2018
, 27 December 2018→Release Criteria: rename criteria
(→Feature Lists: update feature list url) |
(→Release Criteria: rename criteria) |
||
| (15 intermediate revisions by 3 users not shown) | |||
| Line 21: | Line 21: | ||
= Personnel = | = Personnel = | ||
== QA Team == | == QA Team == | ||
* [https://phonebook.mozilla.org/?search/ | * [https://phonebook.mozilla.org/?search/richard%20pappalardo Richard Pappalardo] | ||
* [https://phonebook.mozilla.org/?search/Oana%20Horvath Oana Horvath] | |||
= Resource Plan = | = Resource Plan = | ||
| Line 35: | Line 29: | ||
= Test Strategy = | = Test Strategy = | ||
== Process == | |||
=== Current Sprint === | |||
SUMMARY | |||
# Label Issues | |||
# Create/Update Test Cases | |||
# Verify Issues | |||
DESCRIPTION | |||
* When the development work on github issue is completed, whether before/after code freeze, the issue will be closed so that the issue is available for testing. If a bug requires extra attention, it'll be marked with 'QAReady' label. | |||
* The issue should belong to a correct milestone before development work can be started and may include either new features or bugs. | |||
* Once an issue has been closed, when applicable, a (new) test case should be added to /updated in TestRail. | |||
* TestRail type field should be set to: "Functional" and automation field to: "Untriaged" | |||
* QA will verify the issue, and make sure to outline clear test steps in the (new) TestRail test case. | |||
* Upon completion, github issue should be marked with '''QAVerified''', removing '''QAReady''' label if exists. | |||
* After the QA signoff, ideally, all verifiable issues should be marked with '''QAVerified'''. | |||
* NOTE: the meta-label '''testing''' is also added by both devs and QA for any bug or feature requiring testing of any kind). | |||
* Once the issue is closed, if suitable for automation, label the it in github as '''automation-TBD''' and change the automation dropdown in TestRail to "Suitable". | |||
=== Current or Post Sprint === | |||
* Query github for issues labelled "automation-TBD" | |||
* Create new automated test(s) to verify proper function of feature based on steps outlined in TestRail. | |||
* Cross-link TestRail test case within automated test (add link to java test file on github in Pre-Conditions section) | |||
* Cross-link automated test to TestRail test case (add link to TestRail test case in java file). | |||
* Submit PR for review and, once UI tests are passing, change github label to '''automated''' and set TestRail type to: "Automated" and automation field to: "Completed" | |||
=== QA Criteria === | |||
* Once Dev team announces code freeze, a date for the QA signoff is decided, usually a week from the code freeze. | |||
* QA will run the regression test suite, updated with the coverage for new features, and inform the Dev team with any findings. | |||
* During triage, it will be decided that whether a found issue is a release blocker or not. As a minimum, a bug shall be a release blocker if it causes crash/hang on a relatively common use case scenario. The Dev team will provide fix for the release blockers during the code freeze, and the fix will be verified. | |||
* QA will sign off the release when the (up-to-date) regression test suite run has been completed, all items under [[#Test_Deliverables|Test_Deliverables]] has been completed, and there are no release blockers. | |||
== Manual Tests == | == Manual Tests == | ||
UI | UI test case suite for project: "Focus for Android" will be located in [https://testrail.stage.mozaws.net/index.php?/suites/overview/48 TestRail]. As a minimum, each of the P1 UI features will have corresponding test cases. The test suite will be executed by Softvision for the acceptance testing. | ||
== Automated Tests == | == Automated Tests == | ||
Automation tests will be mainly used for quick regression check of key UI features. Overall, automation tests will be added to the same Github repo, and executed on | Automation tests will be mainly used for quick regression check of key UI features. Overall, automation tests will be added to the same Github repo, and executed on Google Firebase. While the unit tests will be run for each commit, UI Tests (in Espresso/UIAutomator framework) will be executed in master branch only. The test result will be inspected for new failures, and github issue will be raised for tracking. | ||
Automation tests are located [https://github.com/mozilla-mobile/focus-android/tree/master/app/src/androidTest here]. | Automation tests are located [https://github.com/mozilla-mobile/focus-android/tree/master/app/src/androidTest here]. | ||
| Line 126: | Line 152: | ||
* Commented: | * Commented: | ||
** [https://github.com/mozilla-mobile/focus-android/issues/928 Focus 1.1. Beta Build #2202 - always prompts that there is an update available, opens #2196] | ** [https://github.com/mozilla-mobile/focus-android/issues/928 Focus 1.1. Beta Build #2202 - always prompts that there is an update available, opens #2196] | ||
===='''December 6th, 2017 (Sorina Florean):'''==== | |||
* Ran [https://testrail.stage.mozaws.net/index.php?/runs/view/7433&group_by=cases:section_id&group_order=asc&group_id=3479 Functional test suite] on Nexus 5 with Android 6.0.1 | |||
* New logged issues: | |||
** [https://github.com/mozilla-mobile/focus-android/issues/1926 Video exit full screen mode when device orientation is changed #1926] | |||
** [https://github.com/mozilla-mobile/focus-android/issues/1917 Trash can icon active when no search engine are selected #1917] | |||
** [https://github.com/mozilla-mobile/focus-android/issues/1920 Inconsistency when closing pages from Your Rights #1920] | |||
** [https://github.com/mozilla-mobile/focus-android/issues/1922 Settings Menu not displayed properly #1922] | |||
* Issues still reproducible: | |||
** [https://github.com/mozilla-mobile/focus-android/issues/1883 Lastpass is not filtering the autofill suggestion / only shows first few sites #1883] | |||
= Dependencies/Risks = | = Dependencies/Risks = | ||