QA/Pocket integration

From MozillaWiki
< QA
Jump to: navigation, search

Desktop — Firefox Pocket Integration

Ownership

Developer contacts: ---

QA Lead: Andrei Vaida, :avaida on IRC

Status

Overview

[LANDED] Nightly 40: first landed, disabled by default
[ON TRACK] Aurora 39: ---
[ON TRACK] Beta 38: ---
[ON TRACK] Release 38: ---

Risk analysis

Event Probability Impact Current situation Mitigation steps
Users that have the Pocket add-on already installed are not handled properly after updating to a Pocket-integrated Firefox build. probable high --- Periodical regression testing covers this area.
Users that have the Pocket social media service already installed are not handled properly after updating to a Pocket-integrated Firefox build. probable high --- Periodical regression testing covers this area.
User fails to log into Pocket using the FxA sign-in flow. probable high --- Periodical regression testing covers this area.
Page fails to be saved to Pocket. probable high --- Periodical regression testing covers this area.
Saved page fails to be opened from Pocket. probable high --- Periodical regression testing covers this area.

Summary

Context

The features made available by this integration:

  • Save to Pocket button in the toolbar
  • Open Pocket item from the bookmarks menu
  • Save to Pocket button in the context menu
  • FxA sign-in for Pocket
  • Save to Pocket button from Reader View's controls
  • UI tour
  • Reader View aesthetic to align with what Pocket is currently offering

Enabling the feature

Pocket is available starting with Firefox 40 and it can be enabled by setting the following pref to true:

browser.pocket.enabled

References

  • The Pocket toolbar panel mock is available here.
  • A design mock depicting a few Pocket flows is available here
  • A design mock showing confirmation panel's variations and states is available here.
  • A design mock showing the bookmarks menu entry is available here.

Test cases

Overview

Targeted scenarios

  • The Pocket button is displayed by default between the Bookmarks and the Downloads buttons from the toolbar, IF Pocket is enabled by default.
  • IF it's the first time, clicking the Pocket button from any page --regular, in-content, etc-- should bring up the Pocket sign-up panel.
  • The user is able to sign-up for a new Pocket account using FxA.
  • The user is able to log into Pocket using FxA.
  • The user is able to sign-up for a new Pocket account using a regular email address.
  • The user is able to log into Pocket using a regular email address.
  • The user is able to log into Pocket using an existing, premium account.
  • The user is able to save a page to Pocket by clicking the associated button from the toolbar.
  • The user is able to save a page to Pocket by clicking the associated button from Reader View's controls.
  • The user is able to save a page to Pocket by clicking the associated option from the context menu.
  • The user is able to open the Pocket list by clicking the associated option from the bookmarks menu.
  • The user is able to open the Pocket list by clicking the associated link from the Pocket confirmation panel displayed after saving a page.
  • The user is able to open a saved page from the Pocket list.
  • The user is able to open a saved Reader View page from the Pocket list.
  • The user is able to make use of all Pocket functionalities from a Firefox profile that already has Pocket installed as an add-on.
  • The user is able to make use of all Pocket functionalities from a Firefox profile that already has Pocket installed as a social media service (Social API).
  • The user is notified in a friendly manner if his internet connection or the Pocket servers are down.
  • The user is able to log out of Pocket using the associated option from Pocket's web application.

Draft test suites:

MozTrap test suites:

  • [DONE] Smoke Test Suite
    • Please note that the Smoke Test Suite is subject to change due to pending requirements for specific scenarios (e.g. offline experience, Reader View integration, etc.)
  • [ON TRACK] Regression Test Suite

Bug work

Tracking bugs

[META] Bug 1155467 - [meta] Firefox Pocket integration

[META] Bug 1157495 - [meta] Pocket quality and test eng tasks

[META] Bug 1157492 - [meta] Pocket integration metrics tracking

Bug triage +/- for verification

[QE-VERIFY+] Bug 1155523 - Implement rough first-pass at Pocket toolbar button UI
[QE-VERIFY+] Bug 1155517 - Change Reader View to have a "Save Page to Pocket" button instead of "Add To Reader List"
[QE-VERIFY+] Bug 1155518 - Implement "Save to Pocket" context menu item
[QE-VERIFY+] Bug 1155519 - Add "View Pocket Items" menuitem to the bookmarks menu
[QE-VERIFY-] Bug 1155520 - Add telemetry for Pocket integration
[QE-VERIFY-] Bug 1158289 - Optimize Reader View's line length to have between 45 and 75 characters
[QE-VERIFY+] Bug 1158281 - Match Pocket's Reader View Sepia Theme
[QE-VERIFY+] Bug 1161514 - The sign-up panel shown by the Pocket button displays top and bottom borders on Mac and Windows
[QE-VERIFY+] Bug 1160678 - Pocket door hangers arent automatically closed
[QE-VERIFY+] Bug 1161881 - Enable Pocket by default (in supported locales)
[QE-VERIFY+] Bug 1162147 - "View Pocket List" link in bookmarks menu doesn't match UI spec

2015-05-07: set qe-verify- instead since the bug was marked as wontfix

[QE-VERIFY+] Bug 1162193 - Pocket initial panel load displays incorrectly
[QE-VERIFY+] Bug 1161810 - Allow opening the Pocket panel
[QE-VERIFY+] Bug 1162713 - Implement "Save Link to Pocket" context menu item
[QE-VERIFY+] Bug 1162014 - Long tags cut off after trying to save them, overlayed/hidden by white area
[QE-VERIFY+] Bug 1162033 - Pocket panel closes itself while hovering over pocket toolbar button
[QE-VERIFY+] Bug 1162056 - While signed into Firefox Accounts for Sync: 'Sign In with Firefox' button links to sign up page, should be sign in page
[QE-VERIFY+] Bug 1162697 - spinner when saving to pocket feels like a responsiveness problem
[QE-VERIFY+] Bug 1162816 - After signing up for Pocket using Firefox, the onboarding experience shows Chrome screenshots
[QE-VERIFY-] Bug 1162253 - Update the Pocket Menu Icon with the correct aspect ratios
[QE-VERIFY-] Bug 1162660 - Pocket Save Panel should use the system font
[QE-VERIFY+] Bug 1155521 - Figure out what to do with users to have the Pocket add-on / SocialAPI button installed
[QE-VERIFY+] Bug 1163360 - Update Pocket code to latest version (May 9th code drop)
[QE-VERIFY+] Bug 1157017 - Pocket offline experience
[QE-VERIFY+] Bug 1161504 - The "Save" button from the Pocket panel appears to have a visited styling applied
[QE-VERIFY+] Bug 1163319 - Pocket button in hamburger menu breaks layout
[QE-VERIFY+] Bug 1163349 - "View Pocket List" menuitem not working

Bug fix verification

[VERI] Bug 1155523 - Implement rough first-pass at Pocket toolbar button UI

2015-04-30: partial functionality confirmed fixed on 40.0a1, across platforms

[VERI] Bug 1156878 - Send a request to the server when clicking the Pocket toolbar button

2015-04-30: confirmed fixed on 40.0a1, across platforms

[VERI] Bug 1160578 - Odd empty door hanger panel when you're on an internal Firefox page

2015-05-07: confirmed fixed on 40.0a1, across platforms

[VERI] Bug 1155518 - Implement "Save to Pocket" context menu item

2015-05-07: confirmed fixed on 40.0a1, across platforms

[VERI] Bug 1161510 - main.js:481:12 throws an error in the Browser Console when "Sign Up with Firefox" is clicked

2015-05-07: confirmed fixed on 40.0a1, across platforms

[VERI] Bug 1155519 - Add "View Pocket Items" menuitem to the bookmarks menu

2015-05-08: confirmed fixed on 40.0a1, across platforms

[VERI] Bug 1160577 - Error message is no longer shown when a page isn't readable

2015-05-08: confirmed fixed on 40.0a1, across platforms

[VERI] Bug 1160678 - Pocket door hangers arent automatically closed

2015-05-08: confirmed fixed on 40.0a1, across platforms

[VERI] Bug 1161504 - The "Save" button from the Pocket panel appears to have a visited styling applied

2015-05-11: confirmed fixed across branches and platforms

[VERI] Bug 1163319 - Pocket button in hamburger menu breaks layout

2015-05-11: confirmed fixed across branches and platforms

[VERI] Bug 1163349 - "View Pocket List" menuitem not working

2015-05-11: confirmed fixed on 39.0a2 and 38.0.5b1 (build1), across platforms

[VERI] Bug 1157682 - Should ignore query string / hashes for checking for "home page"

2015-05-11: confirmed fixed across branches and platforms

[VERI] Bug 1157197 - Please enable reader view for 38.0.5

2015-05-11: confirmed fixed on 38.0.5b1 (build1) and platforms
Logged bugs

Bug 1161502 - The question mark icon next to the "Sign Up with Firefox" button seems misaligned

2015-05-05: found while performing functional testing on 40.0a1

Bug 1161504 - The "Save" button from the Pocket panel appears to have a visited styling applied

2015-05-05: found while performing functional testing on 40.0a1

Bug 1161508 - The panel shown by the Pocket button after saving a page displays top and bottom borders

2015-05-05: found while performing functional testing on 40.0a1

Bug 1161510 - main.js:481:12 throws an error in the Browser Console when "Sign Up with Firefox" is clicked

2015-05-05: found while performing functional testing on 40.0a1

Bug 1161514 - The sign-up panel shown by the Pocket button displays top and bottom borders on Mac and Windows

2015-05-05: found while performing functional testing on 40.0a1

Bug 1161518 - The user is able to drag and drop the links from the Pocket panel

2015-05-05: found while performing functional testing on 40.0a1

Bug 1161522 - The user is able to save Pocket's "My List" to itself, twice

2015-05-05: found while performing functional testing on 40.0a1

Bug 1162445 - The Pocket toolbar button's functionality is not properly adapting to private browsing, where the user is no longer signed in

2015-05-07: found while performing functional testing on 40.0a1

Bug 1163576 - Unable to remove pages from the Pocket list

2015-05-11: found while validating Firefox 38.0.5b1 (build1)

Bug 1163613 - The panel shown when attempting to Save to Pocket in offline mode is dismissed too soon

2015-05-11: found while validating Firefox 38.0.5b1 (build1)

Bug 1163581 - Unexpected error message displayed when navigating from Pocket list

2015-05-11: found while validating Firefox 38.0.5b1 (build1)
2015-05-11: closed as invalid by Gavin Sharp, as the issue will be tracked Pocket-side

Bug 1163577 - Enter Reader View button available in Pocket List page

2015-05-11: found while validating Firefox 38.0.5b1 (build1)

Bug 1163917 - The built-in Pocket is NOT completely disabled if Pocket is already installed and used as an add-on

2015-05-12: found while validating Firefox 38.0.5b1 (build2)

Milestones

Firefox 38.0.5 Beta 1

Build 1 Build 2
Builds FTP, HTTP FTP, HTTP
ID 20150510205200 20150511143336
Changeset 32b69592b334 (pushlog) fff143cacb66 (pushlog)

Manual testing summary

  • Test coverage
    • Smoke testing — targeting the primary functionalities of the feature
    • Exploratory testing — targeting various other scenarios surrounding the main functionalities available, including negative and edge cases
    • Bug fix verification — Bugzilla query
  • Platform coverage
    • Windows 7 (x64)
    • Windows 8.1 (x86)
    • Mac OS X 10.9.5
    • Ubuntu 14.04 (x64)
  • Reported bugs for this build
    • [normal] Bug 1163576 - Unable to remove pages from the Pocket list
      • UPDATE 2015-05-12 marked as Resolved Fixed and pending uplift
    • [normal] Bug 1163577 - Enter Reader View button available in Pocket List page
    • [normal] Bug 1163613 - The panel shown when attempting to Save to Pocket in offline mode is dismissed too soon
    • [normal] Bug 1163581 - Unexpected error message displayed when navigating from Pocket list
      • UPDATE 2015-05-12 closed as Invalid by Gavin Sharp, as the issue will be tracked on Pocket's end
    • [normal] Bug 1163651 - [Windows]View Pocket List icon from Bookmarks menu is missing.
    • [normal] Bug 1163655 - View Pocket List menuitem is not at top if Bookmarks button is moved to menu panel.
    • [normal] Bug 1163917 - The built-in Pocket is NOT completely disabled if Pocket is already installed and used as an add-on
  • Test results
    • Smoke test results:
Platform Results (report)
Windows 7 (x64)
[FAILED]
Windows 8.1 (x86)
[FAILED]
Mac OS X 10.9.5
[FAILED]
Ubuntu 14.04 (x64)
[FAILED]
  • Overall feature status after testing: [AT RISK]
  • Recommendation from QA: [CONDITIONAL SIGN OFF] — pending the review of newly introduced bugs

Firefox 38.0.5 Beta 2

Builds FTP, HTTP
ID 20150514163436
Changeset 195e873a8ab1 (pushlog)

Manual testing summary

  • Test coverage
    • Smoke testing — targeting the primary functionalities of the feature
    • Exploratory testing — targeting various other scenarios surrounding the main functionalities available, including negative and edge cases
    • Bug fix verification — Bugzilla query
  • Platform coverage
    • Windows XP (x86)
    • Windows 8 (x64)
    • Mac OS X 10.8.5
    • Ubuntu (x86)
  • Reported bugs for this build
    • [normal] Bug 1165291 - Pocket Social Api is not correctly disabled
  • Test results
    • Smoke test results:
Platform Results (report)
Windows XP (x86)
[FAILED]
Windows 8 (x64)
[FAILED]
Mac OS X 10.8.5
[FAILED]
Ubuntu (x86)
[FAILED]
  • Overall feature status after testing: [GOOD]
  • Recommendation from QA: [SIGN OFF]

Sign off

Criteria

  • All the test cases were executed.
  • All the blocker, critical, major bugs have been fixed.

Results

[ON TRACK] Beta 38.0.5 (yyyy-mm-dd)

  • [CONDITIONALLY SIGNED OFF] 38.0.5b1-build2 (2015-05-12)
  • [SIGNED OFF] 38.0.5b2-build1 (2015-05-15)


Pocket System Add-on

Ownership

Developer contacts: Shane Caraveo

QA Lead: Paul Silaghi, :pauly on IRC

Summary

Pocket moved to a system add-on in Bug 1215694, which landed on Firefox 46.

Test Cases

Draft test suites:

Important bugs

[META] Bug 1215694 - Move Pocket to a built-in add-on

[NEW] Bug 1263599 - "View Pocket List" option from the bookmarks menu gone after restart

Sign off

  • [SIGNED OFF] 46.0b11 (2016-04-18)