Apps/ID and Payments/Test Plan

From MozillaWiki
Jump to: navigation, search

Purpose

The document intends to provide a high level overview of the testing that's needed for testing the identity flow, the client-side payments implementation, the marketplace payments integration, and other notable areas that affect payments and identity.

Feature Ownership

Feature QA Lead
Marketplace Integration Krupa Raj
WebPayment Web API and Trustworthy UI Jason Smith
Identity Primary: John Morrison

Secondary: Jason Smith

Other Resources

Resource Lead
Security Raymond Forbes
Load Testing and High Availability Alex, Tarek, Hanno

Testing Scope

In Scope

Functionality

  • All consumer-facing marketplace interactions during payment flow
  • All Navigator.mozPay flows, including paid applications and in-app payments
  • All trustworthy UI Gaia flows
  • All identity flows, including areas inside and outside of marketplace

Software Qualities

  • Security testing for any marketplace, trustworthy UI, and identity flow against the known threat model
  • Performance testing more to validate user responsiveness throughout the various marketplace, trustworthy UI, and identity flows, especially on slow connections
  • Availability and resilience testing for the marketplace payment provider with bango integration to ensure fault tolerance is handled and downtime is minimized
  • Localization testing specifically targeting v1 locales to verify that various character sets and text presentation in the various functional flows align with the locale
  • Location testing to validate that no surprising problems are seen in the functional flows when operating directly in target countries

Out of Scope

  • Mozilla QA team is not responsible for making sure all Bango dataflows work as expected.
  • Mozilla QA team is not responsible for making sure all TEF dataflows work as expected.

Risk Management

The open risks are summarized below from a quality perspective.

Rank Risk Indicator Prevention Mitigation
1 Unstable b2g builds halt, regress, or slow down testing of the payments and identity flow Smoke test bustage on trunk, repeated regressions on payments and identity flow Automation is in place early and often to ensure regressions are caught before check ins are made to the main tree Raise awareness quickly when bustage occurs that affects the payment and identity flow and work quickly to get those bugs fixed
2 Resource unavailability during the Holiday season Developers, Testers and Product resources being on PTO around the Holiday season. Propound atheism to team members Bake the period of unavailability into our schedule planning

Testing Lifecycle

Overview

The following section gives an overview of the lifecycle that shall be followed to test the different major pieces involved in this feature set and how it works towards testing the full flow.

Validate the Web Payment API with the Mock Payment Provider

Summary

The first phase of testing aims to ensure that the basic mozPay Web API with the trustworthy UI is setup correctly independently of the marketplace integration. At the end of this phase upon signoff, we should feel confident that major client-side issues related to the mozPay Web API and trustworthy UI are caught and addressed quickly, resulting in this portion of the implementation to be stable as an independent entity.

Status

Test pass completed. Only a couple blocker bug followups and verifications needed before signoff can be completed.

Owner

Jason Smith

Entry Criteria

  • Need to be feature complete in the implementation on the web payment API
  • Need to be feature complete in the implementation of the trustworthy UI
  • Need to have defined user stories that affect trustworthy UI and web payment Web API directly

Exit Criteria

  • All basecamp blocking bugs need to be fixed and verified
  • Test pass for this phase specifically needs to be ran and not catch any basecamp blocking bugs

Validate the Identity Integration with Trustworthy UI Independently of Marketplace

Summary

The second phase of testing aims to ensure that the identity integration pieces (client and server) work independently of the marketplace-specific flows. At the end of this phase upon signoff, we should feel confident that the persona integration on-device and server side changes have issues caught early and effectively, resulting in the identity pieces being stable independently of the marketplace payment flows.

Status

Ramp-up completed and initial exploratory testing done to understand the feature set. Brainstorming test plan has been done. Currently working on integrating perspectives from identity QA, trustworthy QA, and general b2g QA to facilitate a more disciplined test plan.

Owners

Jason Smith & John Morrison

Entry Criteria

  • Need to be feature complete in the on-device changes of the identity DOM and Gaia changes
  • Need to be feature complete in the identity server side changes required (e.g. unverified emails)
  • Need to be feature complete in the implementation of the trustworthy UI
  • Need to have defined user stories that affect trustworthy UI and identity

Exit Criteria

  • All basecamp blocker bugs need to be fixed and verified
  • A test pass is ran on the DOM ID and trustworthy identity work and finds no basecamp blocker bugs
  • A test pass is ran on the identity server side (regression and new feature work) and finds no basecamp blocker bugs
  • Gaia smoke tests in relation to identity do not experience bustage for at least one week
  • Dogfooding of the identity flow for at least one week reveals no basecamp blocking bugs

Validate the Payments Support on Marketplace

Overview

QA will vet all the Payment userflows for different usertypes : End-user, Developer, App Reviewer and Admins. The aim here is to catch and fix all issues earlier in the development cycle. Tests will be run to ensure that the Marketplace works seamlessly with the client and Identity pieces. We will also ensure that Developer and review centric features are tested and relatively bug-free.

Status

TBD

Owner

Krupa Raj

Entry Criteria

  • Feature is testable and has landed on the dev server (marketplace-dev.allizom.org)
  • QA has access to relevant test accounts for Bango
  • QA is enabled to test on the dev server with fake money
  • QA has access to all relevant mocks and specifications docs from UX
  • All the necessaries Hardware requirements and SLAs are in place

Exit Criteria

  • All basecamp blockers are fixed and verified
  • All P1-P3 bugs are fixed and verified
  • The entire Payments flow is polished and user-friendly
  • All the major user stories have been tested with real money
  • All user stories have been implemented/validated

Infrastructure

  • Mock payment provider exists to allow testing of the web payment web api and trustworthy UI independently of marketplace
  • Various sites making use of persona can be reused to test the identity flow on device
  • Marketplace staging server can be used to test the marketplace integration into the client-side payments and identity flow with Bango

Test Case Management

Test cases for each major feature in the payments and identity flow shall be managed in MozTrap. On-device test cases can be found under the B2G product with the tag of payments&id. Marketplace test cases can be found in the Marketplace product with the tag of payments.

Automation

Marionette front-end automation in Gaia is the target for doing automation from the quality side. The automation aims to broken into these distinct areas:

  • The web payment api, trustworthy UI, and mock payment provider
  • The identity api and trustworthy UI
  • The marketplace integration with trustworthy UI

Development repository for these tests can be found here.

Dogfooding

Overview

Dogfooding by b2g test drivers shall be broken into these phases:

  • Identity enabled on trunk for persona and trustworthy UI dogfooding
  • Marketplace full payments and identity flow with Bango integration with real money

Identity and Trustworthy UI

Summary

Opens the doors to general b2g test drivers to try out the persona integration into Firefox OS with the trustworthy UI. The dogfooding aims to target both general use by test drivers and at least one focused dogfooding session.

Owner

Jason Smith

Dependencies

  • Need to be feature complete on identity and trustworthy UI features that existing persona sites already make use of for client and server
  • Need to have any dogfooding blockers fixed and verified for client and server
  • Need to have a test pass ran to find no dogfooding blockers for client and server

Crowdsourcing in Brazil

Summary

After feature-complete across all components, we would like to have some testing done "out in the wild" - as in, with real money in our target countries. This may include contracting with a Brazilian company or mobilizing our Brazilian community or both. The idea is that they will use Firefox OS phones on the Telefonica carriers with real money and test our Payments flow. Details are still TBD.

Owner

Krupa Raj

Test Data

These are the requisite data for testing the end-to-end payments flow

    • Access to Bango test server (?)
    • Multiple test credit cards (Visa, Mastercard, Discovery, AMEX)
    • SIM cards to test carrier billing (?)

Open Questions

  • What infrastructure does Bango have available for QA to take advantage of?
    • Is there any payment API that exists for testing with fake money?

Resources

Retrieved from "https://wiki.mozilla.org/index.php?title=Apps/ID_and_Payments/Test_Plan&oldid=493331"