QA/Fenix/4/1.0/Beta.1/: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
No edit summary
 
(42 intermediate revisions by 8 users not shown)
Line 1: Line 1:
<small>« [[QA/Fenix/4/0.0/Beta.5/ | Fenix RC 4.0.0-beta.5]] | [[QA/Fenix/4/1.0/Beta.1/ | Fenix RC 4.1.0-beta.1]] »</small>
<small>« [[QA/Fenix/4/0.0/Beta.5/ | Fenix RC 4.0.0-beta.5]] | [[QA/Fenix/4/1.0/Beta.1/ | Fenix RC 4.1.0-beta.1]] | [[QA/Fenix/4/2.0/Beta.1/ | Fenix RC 4.2.0-beta.1]] »</small>


= Firefox Preview 4.0.0-beta.5 Test Plan =
= Firefox Preview 4.1.0-beta.1 Test Plan =
* Release date: March 9th, 2020
* Release date: March 12th, 2020
* QA Contact: [mailto:ohorvath@mozilla.com Oana Horvath]; [mailto:fennec@softvision.ro SV QA team]
* QA Contact: [mailto:ohorvath@mozilla.com Oana Horvath]; [mailto:fennec@softvision.ro SV QA team]
* Slack: [https://mozilla.slack.com/messages/D58NA0CB0 Oana];  
* Slack: [https://mozilla.slack.com/messages/D58NA0CB0 Oana];  
* Build: [https://github.com/mozilla-mobile/fenix/releases/tag/v4.1.0-beta.1 Firefox Preview 4.1.0-beta.1]
* Build: [https://github.com/mozilla-mobile/fenix/releases/tag/v4.1.0-beta.1 Firefox Preview 4.1.0-beta.1]
* Build: [https://github.com/mozilla-mobile/fenix/releases/tag/v4.1.0-beta.2 Firefox Preview 4.1.0-beta.2]
* TestRail test plan: [https://testrail.stage.mozaws.net/index.php?/plans/view/26856 Testrail]  
* TestRail test plan: [https://testrail.stage.mozaws.net/index.php?/plans/view/26856 Testrail]  


== Features ==  
== Features ==  
=== Smoke and sanity tests ===
=== Full functional & UI tests ===
{|width=100%
{|width=100%
|-
|-
Line 18: Line 19:
| Alternating
| Alternating
|}
|}
[https://testrail.stage.mozaws.net/index.php?/plans/view/26856 Smoke and sanity tests - Test Plan]
[https://testrail.stage.mozaws.net/index.php?/plans/view/26856 Full functional and UI tests - Test Plan]


== QA Assignment ==
== QA Assignment ==
Line 32: Line 33:
  | x86
  | x86
  | Xiaomi MiPad 2 (Android 5.1)  
  | Xiaomi MiPad 2 (Android 5.1)  
  | March 12th, 2020
  | March 13th, 2020
  | '''in progress'''
  | '''Pass'''
  |-
  |-
  | Andrei Joltan
  | Andrei Joltan
  | ARM
  | ARM
  | OnePlus A3 (Android 6.0.1)
  | OnePlus A3 (Android 6.0.1)
  | March 12th, 2020
  | March 13th, 2020
  | '''in progress'''
  | '''Pass'''
  |-
  |-
  | Oana Horvath
  | Oana Horvath
  | ARM
  | ARM
  | Google Pixel 4 (Android 10)
  | Google Pixel 4 (Android 10)
  | March 12th, 2020
  | March 13th, 2020
| '''in progress'''
|-
| Andrei Bodea
| ARM
|  (Android )
| March 12th, 2020
  | '''in progress'''
  | '''in progress'''
  |-
  |-
Line 56: Line 51:
  | ARM  
  | ARM  
  | Huawei P9 Lite (Android 7.0)
  | Huawei P9 Lite (Android 7.0)
  | March 12th, 2020
  | March 13th, 2020
  | '''in progress'''
  | '''Pass'''
  |-
  |-
  | Mirabela Lobontiu
  | Mirabela Lobontiu
  | ARM  
  | ARM  
  | Samsung Galaxy Tab S6 (Android 9)
  | Samsung Galaxy Tab S6 (Android 9)
  | March 12th, 2020
  | March 13th, 2020
  | '''in progress'''
  | '''Pass'''
|-
| Stefan Deiac
| ARM
| Samsung Galaxy Tab S3 (Android 8)
| March 13th, 2020
| '''Pass'''
  |-
  |-
  | Diana Rus
  | Diana Rus
  | ARM  
  | ARM  
  | Samsung Galaxy S8+ (Android 8)
  | Samsung Galaxy S8+ (Android 8)
  | March 12th, 2020
  | March 13th, 2020
  | '''in progress'''
  | '''Pass'''
|-
| Laurentiu Apahidean
| ARM
| Huawei MediaPad M3 Lite 10 (Android 7.0)
| March 13th, 2020
| '''Pass'''
|-
  |}  
  |}  
</onlyinclude>
</onlyinclude>
Line 91: Line 99:
  |-  
  |-  
  | Everyone
  | Everyone
  | March 12th, 2020
  | March 13th, 2020
  | '''in progress'''
  | '''Done'''
|}  
|}  
</onlyinclude>
</onlyinclude>


List of issues that needed QA:
List of issues that needed QA:
* https://github.com/mozilla-mobile/fenix/issues/8949
 
* https://github.com/mozilla-mobile/fenix/issues/8800
From https://github.com/mozilla-mobile/fenix/releases/tag/v4.1.0-beta.1:
* https://github.com/mozilla-mobile/fenix/issues/8474
 
* https://github.com/mozilla-mobile/fenix/issues/8908
* https://github.com/mozilla-mobile/fenix/issues/8949 - '''Verified as fixed''' ✅
* https://github.com/mozilla-mobile/fenix/issues/8017
* https://github.com/mozilla-mobile/fenix/issues/7408 - '''Verified as fixed''' ✅
* https://github.com/mozilla-mobile/fenix/issues/7087 '''Verified as fixed'''✅
* https://github.com/mozilla-mobile/fenix/issues/8474 - '''Verified as fixed''' ✅
* https://github.com/mozilla-mobile/fenix/issues/7158
* https://github.com/mozilla-mobile/fenix/issues/8908 - Not done yet. Left a comment.
* https://github.com/mozilla-mobile/fenix/issues/8017 - Not done yet. Left a comment.
* https://github.com/mozilla-mobile/fenix/issues/7087 - '''Verified as fixed''' ✅
* https://github.com/mozilla-mobile/fenix/issues/7158 - '''Verified as fixed''' ✅
* https://github.com/mozilla-mobile/fenix/issues/8897 - '''Verified as fixed''' ✅
 
From https://github.com/mozilla-mobile/fenix/releases/tag/v4.1.0-beta.2:
 
* https://github.com/mozilla-mobile/fenix/issues/9027 - Not done yet. Left a comment.


=== L10N ===
=== L10N ===
Line 111: Line 127:
{| class="wikitable" width=100%
{| class="wikitable" width=100%
  |-  
  |-  
  |  
  | Laurentiu Apahidean
  | March 12th, 2020
  | March 13th, 2020
  | '''in progress'''
  | '''Done'''
|}  
|}  
</onlyinclude>
</onlyinclude>
Line 120: Line 136:
=== New issues ===
=== New issues ===
Major:
Major:
-
* [https://github.com/mozilla-mobile/fenix/issues/9138 9138] - [Bug] Crash when denying app's access to camera, but allowing access to the site


Other:
Other:
 
* [https://github.com/mozilla-mobile/fenix/issues/9143 9143] - [Bug] Added search engine no longer displayed if it was previously undo and the browser was restarted
* [https://github.com/mozilla-mobile/fenix/issues/9128 9128] - [Bug] Some snackbars are overlapped by the URL bar
* [https://github.com/mozilla-mobile/fenix/issues/9107 9107] - [Bug] outdated "What's new" article highlighted
* [https://github.com/mozilla-mobile/fenix/issues/9126 9126] - [Bug]Share Menu - Inconsistent "Share with" window
* [https://github.com/mozilla-mobile/fenix/issues/9124 9124] - [Bug]Zoomed in about:cache
* [https://github.com/mozilla-mobile/fenix/issues/9131 9131] - [Bug]The History section is displayed for a couple of seconds while exiting to Fenix's home screen
* [https://github.com/mozilla-mobile/fenix/issues/9127 9127] - [Bug]Missing "Bookmark saved" confirmation snackbar on Fenix Beta 4.1.0-beta.1
* [https://github.com/mozilla-mobile/fenix/issues/9139 9139] - [Bug] Empty Settings view after tapping the back button


=== Known issues ===
=== Known issues ===
Major:
Major:
 
* [https://github.com/mozilla-mobile/fenix/issues/7638 7638] - Fenix not reporting correct web site to OS


Other:
Other:
* [https://github.com/mozilla-mobile/fenix/issues/7953 7953] - [Bug] Error 0/0 displayed when double tapping on Find in page toolbar
* [https://github.com/mozilla-mobile/fenix/issues/8108 8108] - [Bug] The same site can be added multiple times to home screen
* [https://github.com/mozilla-mobile/fenix/issues/9049 9049] - [Bug]Websites that are available as PWAs on Nightly are not added to home screen on first try on Fenix Beta
* [https://github.com/mozilla-mobile/android-components/issues/3108 3108] - Add new safe browsing policies for Malware in iframe and Billing warning
* [https://github.com/mozilla-mobile/fenix/issues/7953 7953] - [Bug] Error 0/0 displayed when double tapping on Find in page toolbar
* [https://github.com/mozilla-mobile/fenix/issues/8809 8809] - [Bug] Notification permissions can't be disabled/re-enabled unless you restart the browser
* [https://github.com/mozilla-mobile/fenix/issues/5769 5769] - [Bug] talky.io - Cam & mic access denied when the mic is granted permission
* [https://github.com/mozilla-mobile/fenix/issues/8582 8582] - [Bug] Sending tabs to multiple devices is not working
* [https://github.com/mozilla-mobile/fenix/issues/8965 8965] - [Bug]Broken test app color selection in both the browser and in custom tabs
* [https://github.com/mozilla-mobile/fenix/issues/8930 8930] - [Bug] Pause/resume don't work when downloading more files
* [https://github.com/mozilla-mobile/fenix/issues/6589 6589] - Empty bookmarks library view looks buggy
* [https://github.com/mozilla-mobile/fenix/issues/3900 3900] - Bug] After sharing a page from Fenix to Firefox Nightly the popup menu remains in the recent apps menu
* [https://github.com/mozilla-mobile/fenix/issues/6970 6970] - [Bug]History and Bookmark icons flicker when using multi-select feature
* [https://github.com/mozilla-mobile/fenix/issues/8567 8567] - [Bug]Can't create new bookmarks folder due to #6974
* [https://github.com/mozilla-mobile/fenix/issues/6975 6975] - [Bug]Bookmarks folder can't be renamed (following #6974)
* [https://github.com/mozilla-mobile/fenix/issues/6974 6974] - [Bug]Current Bookmark folder shouldn't be displayed as a root folder option while editing
* [https://github.com/mozilla-mobile/fenix/issues/6977 6977] - [Bug]Deleted Bookmarks folder reappears after deleting it
* [https://github.com/mozilla-mobile/fenix/issues/5298 5298] - [Bug] The same page appears in the History section multiple times
* [https://github.com/mozilla-mobile/fenix/issues/6079 6079] - [Bug]Open tabs section flickers when opening multiple history items in new tabs
* [https://github.com/mozilla-mobile/fenix/issues/4178 4178] - [Bug] Sharing multiple tabs to another browser, only opens 1 tab
* [https://github.com/mozilla-mobile/fenix/issues/4484 4484] - [Bug]Flicker when deleting History items
* [https://github.com/mozilla-mobile/fenix/issues/8571 8571] - [Bug]History isn't sorted chronologically
* [https://github.com/mozilla-mobile/fenix/issues/5636 5636] - [Bug]Blocked media autoplay doesn't work properly
* [https://github.com/mozilla-mobile/fenix/issues/8465 8465] - [Bug]Improperly displayed Enhanced Tracking Protection CFR after rotation
* [https://github.com/mozilla-mobile/fenix/issues/4375 4375] - [Bug]Mute/Unmute a video displays a white/grey rectangle over the volume bar
* [https://github.com/mozilla-mobile/fenix/issues/6213 6213] - [Bug]Facebook Stories playback issue
* [https://github.com/mozilla-mobile/fenix/issues/7369 7369] - [Bug]Soundcloud doesn't play in the background anymore
* [https://github.com/mozilla-mobile/fenix/issues/5133 5133] - [Bug] Phone goes to sleep while audio is playing
* [https://github.com/mozilla-mobile/fenix/issues/4793 4793] - [Bug] soundscloud: Audio plays without media notification showing
* [https://github.com/mozilla-mobile/fenix/issues/6211 6211] - [Bug]Instagram notification issue
* [https://github.com/mozilla-mobile/fenix/issues/8726 8726] - [Bug]Collections-Empty collection can be created
* [https://github.com/mozilla-mobile/fenix/issues/8848 8848] - [Bug]Fenix doesn't properly suggest the collections numbering order
* [https://github.com/mozilla-mobile/fenix/issues/7242 7242] - [Bug] Collection size change after expand/compress them
* [https://github.com/mozilla-mobile/fenix/issues/2783 2783] - Add snackbar to collection deletion
* [https://github.com/mozilla-mobile/fenix/issues/8716 8716] - [Bug] Private custom tab opened when signing in to sync records history
* [https://github.com/mozilla-mobile/fenix/issues/8590 8590] - [Bug] From PB shortcut, clearing tabs from notification returns to Normal mode
* [https://github.com/mozilla-mobile/fenix/issues/9100 9100] - [Bug] Clearing private tabs from notification closes the app
* [https://github.com/mozilla-mobile/fenix/issues/7996 7996] - [Bug] Private mode doesn't play nicely with sites added to home screen
* [https://github.com/mozilla-mobile/fenix/issues/8139 8139] - [Bug]Improperly displayed uBlock icons using the Light theme
* [https://github.com/mozilla-mobile/fenix/issues/8520 8520] - [Bug] Add-ons Manager and add-on sub-pages have inconsistent style compared to the rest of Settings
* [https://github.com/mozilla-mobile/fenix/issues/8773 8773] - [Bug]Snackbars are not displayed right above URL bar
* [https://github.com/mozilla-mobile/fenix/issues/8578 8578] - [Bug] Different size icons for "Not supported yet" addons
* [https://github.com/mozilla-mobile/fenix/issues/8574 8574] - [Bug] Add-on manager UI elements are misaligned in RTL languages
* [https://github.com/mozilla/fxa/issues/1657 1657] - [fxa-content-server] Canceled pairing from the mobile device should update automatically on the desktop side
* [https://github.com/mozilla-mobile/fenix/issues/6521 6521] - [Bug] Auto-sign in doesn't work on Beta
* [https://github.com/mozilla-mobile/fenix/issues/7723 7723] - [Bug] The account sync doesn't start automatically when the app is brought to the foreground
* [https://github.com/mozilla-mobile/fenix/issues/8436 8436] - [Bug] Sync isnt syncing bookmarks correctly
* [https://github.com/mozilla-mobile/fenix/issues/7672 7672] - [Bug] [a11y] Accessibility Scanner suggests increasing the "x" button from Save to Collection page
* [https://github.com/mozilla-mobile/fenix/issues/7708 7708] - [Bug] [a11y] Consider making the "Common myths" item from PB larger
* [https://github.com/mozilla-mobile/fenix/issues/7716 7716] - [Bug] [a11y] Consider making "Learn more" clickable item larger
* [https://github.com/mozilla-mobile/fenix/issues/7718 7718] - [Bug] [a11y] Consider making the "x" button from the ETP pop-up larger
* [https://github.com/mozilla-mobile/fenix/issues/7720 7720] - [Bug] [a11y] ETP toggle accessibility updates
* [https://github.com/mozilla-mobile/fenix/issues/7725 7725] - [Bug] [a11y] Consider increasing the "Info" icon items from the ETP page
* [https://github.com/mozilla-mobile/fenix/issues/7752 7752] - [Bug] [a11y] Consider making the "Send crash report to Mozilla" item larger
* [https://github.com/mozilla-mobile/fenix/issues/7754 7754] - [Bug] [a11y] Suggestions regarding download's context menus
* [https://github.com/mozilla-mobile/fenix/issues/7756 7756] - [Bug] [a11y] Increase size of elements in bottom section of BrowserToolbar
* [https://github.com/mozilla-mobile/fenix/issues/8629 8629] - [Bug][a11] Save to collection button touch target is too small

Latest revision as of 07:09, 24 March 2020

«  Fenix RC 4.0.0-beta.5 | Fenix RC 4.1.0-beta.1 | Fenix RC 4.2.0-beta.1 »

Firefox Preview 4.1.0-beta.1 Test Plan

Features

Full functional & UI tests

QA Task Person/Group
Softvision Alternating

Full functional and UI tests - Test Plan

QA Assignment

Assigned Build type Device Completed Results
Sorina Florean x86 Xiaomi MiPad 2 (Android 5.1) March 13th, 2020 Pass
Andrei Joltan ARM OnePlus A3 (Android 6.0.1) March 13th, 2020 Pass
Oana Horvath ARM Google Pixel 4 (Android 10) March 13th, 2020 in progress
Eliza Balazs ARM Huawei P9 Lite (Android 7.0) March 13th, 2020 Pass
Mirabela Lobontiu ARM Samsung Galaxy Tab S6 (Android 9) March 13th, 2020 Pass
Stefan Deiac ARM Samsung Galaxy Tab S3 (Android 8) March 13th, 2020 Pass
Diana Rus ARM Samsung Galaxy S8+ (Android 8) March 13th, 2020 Pass
Laurentiu Apahidean ARM Huawei MediaPad M3 Lite 10 (Android 7.0) March 13th, 2020 Pass


Results

Telemetry

Phones

Tablets

Accessibility scanner


Bug verification

Everyone March 13th, 2020 Done


List of issues that needed QA:

From https://github.com/mozilla-mobile/fenix/releases/tag/v4.1.0-beta.1:

From https://github.com/mozilla-mobile/fenix/releases/tag/v4.1.0-beta.2:

L10N

Light testing on featured locales to ensure strings fit the space allotted and there are no entity errors from missing or corrupt localizations.

Laurentiu Apahidean March 13th, 2020 Done


Issues

New issues

Major:

  • 9138 - [Bug] Crash when denying app's access to camera, but allowing access to the site

Other:

  • 9143 - [Bug] Added search engine no longer displayed if it was previously undo and the browser was restarted
  • 9128 - [Bug] Some snackbars are overlapped by the URL bar
  • 9107 - [Bug] outdated "What's new" article highlighted
  • 9126 - [Bug]Share Menu - Inconsistent "Share with" window
  • 9124 - [Bug]Zoomed in about:cache
  • 9131 - [Bug]The History section is displayed for a couple of seconds while exiting to Fenix's home screen
  • 9127 - [Bug]Missing "Bookmark saved" confirmation snackbar on Fenix Beta 4.1.0-beta.1
  • 9139 - [Bug] Empty Settings view after tapping the back button

Known issues

Major:

  • 7638 - Fenix not reporting correct web site to OS

Other:

  • 7953 - [Bug] Error 0/0 displayed when double tapping on Find in page toolbar
  • 8108 - [Bug] The same site can be added multiple times to home screen
  • 9049 - [Bug]Websites that are available as PWAs on Nightly are not added to home screen on first try on Fenix Beta
  • 3108 - Add new safe browsing policies for Malware in iframe and Billing warning
  • 7953 - [Bug] Error 0/0 displayed when double tapping on Find in page toolbar
  • 8809 - [Bug] Notification permissions can't be disabled/re-enabled unless you restart the browser
  • 5769 - [Bug] talky.io - Cam & mic access denied when the mic is granted permission
  • 8582 - [Bug] Sending tabs to multiple devices is not working
  • 8965 - [Bug]Broken test app color selection in both the browser and in custom tabs
  • 8930 - [Bug] Pause/resume don't work when downloading more files
  • 6589 - Empty bookmarks library view looks buggy
  • 3900 - Bug] After sharing a page from Fenix to Firefox Nightly the popup menu remains in the recent apps menu
  • 6970 - [Bug]History and Bookmark icons flicker when using multi-select feature
  • 8567 - [Bug]Can't create new bookmarks folder due to #6974
  • 6975 - [Bug]Bookmarks folder can't be renamed (following #6974)
  • 6974 - [Bug]Current Bookmark folder shouldn't be displayed as a root folder option while editing
  • 6977 - [Bug]Deleted Bookmarks folder reappears after deleting it
  • 5298 - [Bug] The same page appears in the History section multiple times
  • 6079 - [Bug]Open tabs section flickers when opening multiple history items in new tabs
  • 4178 - [Bug] Sharing multiple tabs to another browser, only opens 1 tab
  • 4484 - [Bug]Flicker when deleting History items
  • 8571 - [Bug]History isn't sorted chronologically
  • 5636 - [Bug]Blocked media autoplay doesn't work properly
  • 8465 - [Bug]Improperly displayed Enhanced Tracking Protection CFR after rotation
  • 4375 - [Bug]Mute/Unmute a video displays a white/grey rectangle over the volume bar
  • 6213 - [Bug]Facebook Stories playback issue
  • 7369 - [Bug]Soundcloud doesn't play in the background anymore
  • 5133 - [Bug] Phone goes to sleep while audio is playing
  • 4793 - [Bug] soundscloud: Audio plays without media notification showing
  • 6211 - [Bug]Instagram notification issue
  • 8726 - [Bug]Collections-Empty collection can be created
  • 8848 - [Bug]Fenix doesn't properly suggest the collections numbering order
  • 7242 - [Bug] Collection size change after expand/compress them
  • 2783 - Add snackbar to collection deletion
  • 8716 - [Bug] Private custom tab opened when signing in to sync records history
  • 8590 - [Bug] From PB shortcut, clearing tabs from notification returns to Normal mode
  • 9100 - [Bug] Clearing private tabs from notification closes the app
  • 7996 - [Bug] Private mode doesn't play nicely with sites added to home screen
  • 8139 - [Bug]Improperly displayed uBlock icons using the Light theme
  • 8520 - [Bug] Add-ons Manager and add-on sub-pages have inconsistent style compared to the rest of Settings
  • 8773 - [Bug]Snackbars are not displayed right above URL bar
  • 8578 - [Bug] Different size icons for "Not supported yet" addons
  • 8574 - [Bug] Add-on manager UI elements are misaligned in RTL languages
  • 1657 - [fxa-content-server] Canceled pairing from the mobile device should update automatically on the desktop side
  • 6521 - [Bug] Auto-sign in doesn't work on Beta
  • 7723 - [Bug] The account sync doesn't start automatically when the app is brought to the foreground
  • 8436 - [Bug] Sync isnt syncing bookmarks correctly
  • 7672 - [Bug] [a11y] Accessibility Scanner suggests increasing the "x" button from Save to Collection page
  • 7708 - [Bug] [a11y] Consider making the "Common myths" item from PB larger
  • 7716 - [Bug] [a11y] Consider making "Learn more" clickable item larger
  • 7718 - [Bug] [a11y] Consider making the "x" button from the ETP pop-up larger
  • 7720 - [Bug] [a11y] ETP toggle accessibility updates
  • 7725 - [Bug] [a11y] Consider increasing the "Info" icon items from the ETP page
  • 7752 - [Bug] [a11y] Consider making the "Send crash report to Mozilla" item larger
  • 7754 - [Bug] [a11y] Suggestions regarding download's context menus
  • 7756 - [Bug] [a11y] Increase size of elements in bottom section of BrowserToolbar
  • 8629 - [Bug][a11] Save to collection button touch target is too small
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Fenix/4/1.0/Beta.1/&oldid=1225370"