QA/Fennec/iOS/5.0

From MozillaWiki
< QA‎ | Fennec‎ | iOS
Jump to: navigation, search

Fennec (Firefox for iOS) Test Plan

Overview

This is a tracking document that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Firefox for iOS 5.0 Release. It will follow the general plan on steps and process that QA will take to accomplish a "sign off" to the release strategy. Mozilla QA strives to uphold quality software, and a successful pass corresponds to no blocking issues, 'Green'-colored test pass on test cases in both execution and automation, and a thorough bug verification window of all Blocking bugs and Critical bugs.

Areas Covered

Features

Any major feature item will have a test plan and a tester who owns that area.

Feature QA Owner Tracking Bug Documentation/Test Plan Feature Health
Combine history-related home panels Simon Basca bug 1250749 https://wiki.mozilla.org/QA/Fennec/iOS/Combine_history-related_home_panels {mok}
Menu and Toolbar Redesign Catalin Suciu bug 1210109 https://wiki.mozilla.org/QA/Fennec/iOS/tools_menu {mok}
Set a home page Simon Basca bug 1260449 https://wiki.mozilla.org/QA/Fennec/iOS/set_a_homepage {mok}
Setting home portal page for China bug 1249190
Bring attention to Menu in "What's new in 5.0" Simon Basca bug 1267670
Support 3rd party search engines Catalin Suciu bug 1205877
Provide keyboard shortcuts for common actions Simion Basca bug 1167970
Support RTL Catalin Suciu bug 1160759 https://wiki.mozilla.org/QA/Fennec/iOS/Support_RTL#Overall_Status
Support Tracking Protection Simion Basca bug 1433449 https://wiki.mozilla.org/QA/Fennec/iOS/Support_Tracking_Protection

Functional Testing

Functional tests will be executed on every other Beta build. The test cases can be fond here

Bug tracking

List of bugs tracking this release

Full Query
ID Priority Summary Status Assigned to
1147912 -- Prompt specially for tel: URIs as phone numbers RESOLVED hennes
1186979 -- Long press back/forward list RESOLVED Robin Andersen [:tecgirl]
1222848 -- Bookmark titles should be editable RESOLVED alx91
1250145 -- Crash in XCGLogger: specialized XCGFileLogDestination.output(XCGLogDetails, text : String) -> () + 692 RESOLVED Brian Nicholson (:bnicholson)
1259251 -- Redesign SiteTableViewController to the new mocks RESOLVED Bryan Munar (:bkmunar)
1259253 -- Add "Synced Tabs" feature to history panel RESOLVED Bryan Munar (:bkmunar)
1260899 -- Enter an intermediate unable-to-sync mode when server bookmarks transition from consistent to inconsistent RESOLVED Stephan Leroux [:sleroux]
1261851 -- Lack of passcode timeout exposes logins on application background/restoration RESOLVED Stephan Leroux [:sleroux]
1264641 -- Timestamp updates after Syncing even if syncing fails due to network issues RESOLVED
1264707 -- Include Qwant search plug-in for FR, UK and DE RESOLVED Mike Kaply [:mkaply]
1265819 -- Add "Undo" option when closing all tabs RESOLVED Tyler Lacroix [:tlacroix]
1265846 -- Crash in Storage: specialized thunk + 112 (SQLiteBookmarksModel.swift) RESOLVED Emily Toop (:fluffyemily)
1266443 -- Customize intro card mismatch not reflective of new menu changes RESOLVED Robin Andersen [:tecgirl]
1266476 -- Core telemetry ping RESOLVED Brian Nicholson (:bnicholson)
1266628 -- Duplicate characters will be input when using original Simplified Chinese and Japanese keyboards RESOLVED Stefan Arentz | :st3fan | ⏰ EST | he/him
1267205 -- Display current sync status in Settings RESOLVED Emily Toop (:fluffyemily)
1270693 -- FindInPageBar Keyboard Improvements RESOLVED Stefan Arentz | :st3fan | ⏰ EST | he/him
1270959 -- Reading List empty state should be mirrored RESOLVED Stefan Arentz | :st3fan | ⏰ EST | he/him
1270971 -- Top Sites tile order should follow RTL rules RESOLVED Stefan Arentz | :st3fan | ⏰ EST | he/him
1270973 -- Reading list read/unread indicator is not aligned correctly RESOLVED Bryan Munar (:bkmunar)
1271378 -- Report session times in core ping RESOLVED Brian Nicholson (:bnicholson)
1271380 -- Report search counts in core ping RESOLVED Brian Nicholson (:bnicholson)
1271674 -- Xcode 7.3.1 Swift compiler crash on optimized build RESOLVED Stefan Arentz | :st3fan | ⏰ EST | he/him
1271880 -- Crash when opening link to Google Form from Outlook RESOLVED
1272056 -- Unlocalizable "devices connected" string added in bug 1259253 RESOLVED James Hugman [:jhugman] [@jhugman]
1273813 -- Menu not dismissed when switching orientation from landscape to portrait on iOS 8.x RESOLVED Emily Toop (:fluffyemily)
1273952 -- Send menu item icon is too tall and too light. RESOLVED James Hugman [:jhugman] [@jhugman]
1273963 -- Javascript confirm always returns false. RESOLVED James Hugman [:jhugman] [@jhugman]
1274070 -- Add timestamp to core ping RESOLVED Brian Nicholson (:bnicholson)
1275358 -- Reword action extensions to include Firefox and Beta RESOLVED James Hugman [:jhugman] [@jhugman]
1275527 -- 'Edit' button is not immediately hidden when removing the last 3rd party search engine RESOLVED Stephan Leroux [:sleroux]
1276164 -- Disable "Allow Third Party Keyboards" does not working when the default setting has never been changed RESOLVED Emily Toop (:fluffyemily)
1276541 -- The setting "Show Homepage Icon" should be disabled by default for China RESOLVED James Hugman [:jhugman] [@jhugman]
1276542 -- In China, Firefox should load the home page after launched, at lease in first run after installed. RESOLVED James Hugman [:jhugman] [@jhugman]
1276545 -- Delete the home page url by keyboard is difference with clear button RESOLVED James Hugman [:jhugman] [@jhugman]
1277194 -- 'Syncing has failed' text shifts left under the exclamation mark icon after switching orientation RESOLVED
1277227 -- Several thumbnail will be briefly replaced with an incorrect favicon/background RESOLVED Stephan Leroux [:sleroux]
1278053 -- Visit site anyway allows to load any domains if same invalid certificate is used. RESOLVED Brian Nicholson (:bnicholson)
1278488 -- Menu strings not localized in TestFlight beta 5.0 (9) RESOLVED
1278555 -- Desktop bookmarks are still deletable after a failed sync call RESOLVED Stephan Leroux [:sleroux]
1278564 -- Synchronization fails if all incoming bookmark records are value-only changes, resulting in no tree to merge RESOLVED Richard Newman [:rnewman]
1278771 -- Local bookmarks can't be deleted after a failed sync RESOLVED
1278814 -- Display warning instead of error on bookmark bi-directional sync failure RESOLVED Emily Toop (:fluffyemily)
1278854 -- Send Tab via Share menu not working RESOLVED Stephan Leroux [:sleroux]
1278933 -- Crash in Client: @objc LoginListViewController.(loadLogins in _BA3EBD723238F30AACDA6A4F284750A0)(String?) -> () + 52 RESOLVED
1279242 -- Logins are displayed in search results after they are all deleted RESOLVED Stephan Leroux [:sleroux]
1279319 -- Sync notification displaying warning/error when sync completion completed successfully RESOLVED
1279455 -- Only display new sync statuses in status bar RESOLVED Emily Toop (:fluffyemily)
1279478 -- Don't show sync failed messages RESOLVED Emily Toop (:fluffyemily)
1279787 -- Stealing cross origin DOM data with bypassing localhost navigation restriction RESOLVED Brian Nicholson (:bnicholson)
1279905 -- 'Today' date headline will overlap the pages saved in History RESOLVED
1281204 -- Incomplete localhost blacklist allows arbitrary pages to open local URLs RESOLVED Brian Nicholson (:bnicholson)
1281299 -- IPv6 URLs crash in FaviconFetcher RESOLVED Brian Nicholson (:bnicholson)
1281405 -- "Enter Passcode" is requested for one second and it disappears after resuming Firefox RESOLVED Stephan Leroux [:sleroux]
1281760 -- Handle all errors thrown by bookmark merging and syncing and report as warnings not failures in sync status RESOLVED Emily Toop (:fluffyemily)
1281841 -- Logins screen unaccessible via passcode if Touch ID is enabled, then unenrolled from RESOLVED Nathanael Alcock
1282380 -- Crash when backgrounding/foregrounding the app while in reader view RESOLVED
1282555 -- Menu titles are incorrectly wrapped RESOLVED
1282748 -- Handle wrapping better for menu item text for long words RESOLVED Emily Toop (:fluffyemily)
1282929 -- support protocol links do not work (request denied) RESOLVED Brian Nicholson (:bnicholson)
1283062 -- Menu arrow doesn't match menu color RESOLVED
1283333 -- Add "ast" localization to iOS 5.0 release RESOLVED Francesco Lodolo [:flod]

62 Total; 0 Open (0%); 62 Resolved (100%); 0 Verified (0%);


Web Compatibility Tests

Summary

The Web Compatibility Tests focuses on regression and compatibility with popular websites. For Fennec on iOS we are using WKWebView (WebKit) so comparative testing against Safari will be of value. Any problems discovered in the browser should be reproducible in Safari.

Test Framework
  • Top Sites - coverage of a few popular sites that drives most user traffic
Results

Test results here

Bug Verification

Summary

Verifying fixed bugs in Buzilla gives us additional testing on specific bugs that were fixed.

Planning & Scheduling

To prioritize and narrow the list of bugs, QA will focus primarily on Fennec bugs that represent highest severity.

Results
Full Query
ID Priority Summary Status Assigned to
1244133 -- Data URI's fail to load when accessed from the address-bar VERIFIED Nathanael Alcock
1251222 -- Hide context menu in private browsing tabs when in the app switcher VERIFIED Tyler Lacroix [:tlacroix]
1258695 -- Find In Page - with VKB open, the results are covered by the find in page bar on 3.5" devices VERIFIED farhanpatel.17
1259191 -- landscape remote tabs empty panel state on iPhone 4s is cut off VERIFIED farhanpatel.17
1260699 -- Incorrect behavior of first run tour when swiping between slides VERIFIED Maurya Talisetti
1262160 -- The application will crash when opening/exiting split screen feature VERIFIED Emily Toop (:fluffyemily)
1263624 -- App Extensions (Share, ViewLater, SendTo) should only allow http and https URLs VERIFIED Brian Nicholson (:bnicholson)
1263842 -- Unable to use the passcode after device restarts VERIFIED farhan
1264225 -- Unverified accounts are syncing on exit VERIFIED Stephan Leroux [:sleroux]
1264893 -- The favicon is not displayed after opening multiple tabs VERIFIED Maurya Talisetti
1265657 -- Today Widget - "Go to copied link" highlight is incorrect VERIFIED James Hugman [:jhugman] [@jhugman]
1265669 -- Today Widget - "Go to copied link" highlights only when tapping on text VERIFIED James Hugman [:jhugman] [@jhugman]
1265688 -- The tab counter animates over the Cancel button when "Go to copied link" while Firefox is in edit mode VERIFIED James Hugman [:jhugman] [@jhugman]
1266362 -- After a device rotation, the private browsing themed menu button on the browser toolbar is barely visible VERIFIED Emily Toop (:fluffyemily)
1267167 -- New tabs are opened only in Firefox Beta via Today's Widget VERIFIED James Hugman [:jhugman] [@jhugman]
1267241 -- Incorrect position of copy link in Today's Widget menu VERIFIED James Hugman [:jhugman] [@jhugman]
1267482 -- Tabs are restored with an extra blank page in the back stack VERIFIED Brian Nicholson (:bnicholson)
1267589 -- 'Maximum attempts reached....' message not displayed when entering the 'Turn Passcode Off' or 'Change Passcode' screens VERIFIED Tyler Lacroix [:tlacroix]
1267602 -- No limit in entering incorrect passcodes after the first passcode lock out expires VERIFIED Farhan Patel [:farhan]
1268033 -- Firefox becomes unresponsive when updating to 4.0.0b4 VERIFIED Stephan Leroux [:sleroux]
1268871 -- Regression: Accessing browser menu panels just yields a blank screen VERIFIED Brian Nicholson (:bnicholson)
1268932 -- Update message for certificate errors for consistency on Mobile VERIFIED Tyler Lacroix [:tlacroix]
1270047 -- New Private Tab button is displayed in Today Widget for iOS 8.4 VERIFIED Stephan Leroux [:sleroux]
1270087 -- Passcode bullets are not visible in landscape view VERIFIED Tyler Lacroix [:tlacroix]
1270292 -- Tab Counter should display infinity symbol when user opens 100+ tabs VERIFIED Tyler Lacroix [:tlacroix]
1270456 -- Tools Menu not dismissed when switching orientation on iPhones VERIFIED Emily Toop (:fluffyemily)
1270567 -- Incorrect tab thumbnail shown (thumbnail not yet fetched/captured) when accessing the tab drawer after opening a new tab VERIFIED Tyler Lacroix [:tlacroix]
1270573 -- Favicon lost in session restoration VERIFIED Tyler Lacroix [:tlacroix]
1270765 -- Visible tile invalidation flicker on application restoration VERIFIED Stephan Leroux [:sleroux]
1270767 -- The application will crash after opening a new tab and switching to private browsing VERIFIED Tyler Lacroix [:tlacroix]
1270865 -- Top sites are not correctly displayed after changing device orientation VERIFIED Bryan Munar (:bkmunar)
1271337 -- Clicking on the Reading List menu item opens a blank page VERIFIED Emily Toop (:fluffyemily)
1271568 -- The menu bar is not fully displayed in tabs tray VERIFIED Stephan Leroux [:sleroux]
1271572 -- New tab and New private browsing icons are overlapped VERIFIED Emily Toop (:fluffyemily)
1272007 -- Passcode not requested when resuming Firefox with the delete confirmation dialog open VERIFIED
1272008 -- Logins menu is not dismissed when opening a new tab from Today Widget VERIFIED
1272071 -- Synced Tabs title label jumps down when navigating to panel VERIFIED Bryan Munar (:bkmunar)
1272234 -- The application will crash when deleting a login after lock screen VERIFIED Tyler Lacroix [:tlacroix]
1272293 -- Set/Change passcode screen is not correctly displayed when changing device orientation VERIFIED
1272299 -- Setting up passcode functionality is not correct on IPhone 6 Plus when rotating device VERIFIED
1272602 -- The menu is cut off on phones when VKB is open VERIFIED Emily Toop (:fluffyemily)
1273143 -- Sync status message is displayed as 'Syncing' when an unverified account is created VERIFIED Stephan Leroux [:sleroux]
1273155 -- Menu animation sticks and displays lower than expected when using Personal Hotspot VERIFIED Stephan Leroux [:sleroux]
1273864 -- Menu pagination dots in Normal Browsing are the wrong color VERIFIED Emily Toop (:fluffyemily)
1273915 -- On new private tab (via widget) the private browsing information text will overlap the new opened tab VERIFIED Stephan Leroux [:sleroux]
1273927 -- "Learn more" button button is not fully displayed on screen smaller than 4 inch VERIFIED Stephan Leroux [:sleroux]
1274243 -- The menu is partially displayed when loading a web page VERIFIED Emily Toop (:fluffyemily)
1274245 -- The toolbar is barely visible when changing device orientation in private browsing VERIFIED Emily Toop (:fluffyemily)
1274278 -- The Homepage button is not positioned correct on devcies with 4'' screen VERIFIED
1274398 -- Homepage URL does not default to http when entered manually VERIFIED James Hugman [:jhugman] [@jhugman]
1274508 -- The Homepage icon should not be displayed in the toolbar VERIFIED James Hugman [:jhugman] [@jhugman]
1274521 -- The URL text cannot be selected / highlighted when manually setting a Homepage VERIFIED James Hugman [:jhugman] [@jhugman]
1274542 -- Open new tab via context menu is no longer working as expected VERIFIED James Hugman [:jhugman] [@jhugman]
1274547 -- Crash when "Close All Tabs" if the tabs were open from the context menu VERIFIED Emily Toop (:fluffyemily)
1274549 -- Missing top and bottom padding for multi-line settings rows text on small screen devices VERIFIED Stephan Leroux [:sleroux]
1274679 -- Toolbar loses button colors switching from landscape to portrait in Private Browsing Mode VERIFIED Tyler Lacroix [:tlacroix]
1275527 -- 'Edit' button is not immediately hidden when removing the last 3rd party search engine RESOLVED Stephan Leroux [:sleroux]
1276204 -- The buttons are not displayed in the toolbar after opening a new tab via Today Widget VERIFIED Emily Toop (:fluffyemily)
1276639 -- Home icon layout breaks when Firefox is backgrounded and restored VERIFIED James Hugman [:jhugman] [@jhugman]
1276648 -- The private browsing information text will overlap the closed tabs VERIFIED Stephan Leroux [:sleroux]
1278224 -- The application will crash when accessing logins while syncing VERIFIED Stephan Leroux [:sleroux]
1278854 -- Send Tab via Share menu not working RESOLVED Stephan Leroux [:sleroux]
1279242 -- Logins are displayed in search results after they are all deleted RESOLVED Stephan Leroux [:sleroux]
1279245 -- Firefox becomes unresponsive after opening a new tab via Today Widget VERIFIED Farhan Patel [:farhan]
1279901 -- Long pressing on Back/Forward button will not display the page title in a private browsing session VERIFIED
1280455 -- [IOS 10] Top sites are not correctly displayed after changing device orientation VERIFIED
1280488 -- [IOS 10] Passcode not requested when resuming Firefox with the delete confirmation dialog open VERIFIED
1280489 -- [IOS 10] Logins menu is not dismissed when opening a new tab from Today Widget VERIFIED
1280490 -- [IOS 10] The application will crash when deleting a login after lock screen VERIFIED
1281403 -- Incorrect theme applied to the bottom toolbar when opening a new blank tab VERIFIED James Hugman [:jhugman] [@jhugman]
1281405 -- "Enter Passcode" is requested for one second and it disappears after resuming Firefox RESOLVED Stephan Leroux [:sleroux]
1281777 -- Top sites is not displayed when tapping the URL Bar on a new tab with homepage VERIFIED Bryan Munar (:bkmunar)
1282061 -- Firefox will not enter 'Reader Mode' after background and restoration VERIFIED Stephan Leroux [:sleroux]
1283767 -- Top sites are displayed incorrect after 'First tour'/Settings is presented VERIFIED Bryan Munar (:bkmunar)
1285491 -- Incorrect menu version displayed in browsing mode after closing an about:home tab VERIFIED Nathanael Alcock

75 Total; 0 Open (0%); 4 Resolved (5.33%); 71 Verified (94.67%);


Collecting Feedback

Reporting tools
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Fennec/iOS/5.0&oldid=1190423"