TestEngineering/Services/SyncTestPlanV1: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 341: Line 341:
** more TBD
** more TBD


== TS9: Marketplace Support (all platforms) ==
== TS9: Marketplace Support ==
* Test FxA account activity:
* Test FxA account activity:
** On FFos
** On FFos
** From a Firefox browser (desktop/android)
** From a Firefox browser (desktop/android)
** From a non-Fx browser using gherkin support
** From a non-Fx browser using the web interface
* See the Marketplace Test Plan for more information: TBD
* See the Marketplace Test Plan for more information: TBD


Revision as of 23:25, 19 February 2014

This is the Test Plan for FxA, Tokenserver, and Sync 1.5

Intro/Summary/Notes

Intro

  • The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.5.
  • It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.

Summary

  • A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc...

Notes

  • FxA is scheduled to go out with Firefox 29. This will replace old sync account functionality and use.
  • Sync users from Firefox 28 and earlier will need to create new FxA accounts when they upgrade to Fx29.
  • Once that happens, devices hooked to sync for Fx28 and older will have to be unlinked and then added to the new FxA account in order to sync.
  • If a user wants to hold all devices at old sync, he/she must not upgrade to Fx29.
  • A user can not have accounts with devices across versions of Fx older than 29.

User Stories

Building Test Cases

  • QA needs at least one resource, probably from SoftVision, to convert the current Android, desktop/web, and FFos user stories into client-side test cases.
  • Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
  • Scheduling: TBD

Schedule

QA Team

  • Services QA
    • Edwin Wong - Manager
    • James Bonacci
    • John Morrison
    • Karl Thiessen
    • Peter deHaan
    • Richard Pappalardo
  • QA
    • Various members of the Firefox OS, Desktop, and Mobile teams will be involved in the testing.

OPs Information

  • OPs has a significant number of really detailed Mana pages for deploying, testing, maintaining, and monitoring our Stage and Production environments.
  • Talk to a member of the OPs team for more information and for the Mana links to get you started.

Email Lists and IRC

Bug Lists and Reporting

  • Dev-related issues will be mixed within Bugzilla and GitHub (but see trees/meta info below)
  • Product issues will all live in Bugzilla
  • User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
    • Classification: TBD
    • Products: TBD
    • Components: TBD
  • Note: We may need to cross link Bugzilla bugs with GitHub issues

Firefox Releases

Platforms/OS

  • Mac 10.7.x and 10.8.x
  • Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
  • Windows 8 (Metro mode and desktop mode) 32bit/64bit
  • Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
  • Android 2.x - 4.x

ELM Nightly Builds

Dev Test Environments

LoadTest Environment

Stage Environments

Test Sections/Suites

TS 1: Authentication and Accounts - Backend

  • This is Authentication Server/fxa-auth (IdP), but not device-specific
  • Main parts:
    • fxa-auth (IdP) authentication service
    • MySQL data storage
    • Heka data/log aggregation
  • Areas of focus for testing: primarily server-side
    • Manual testing - unit/functional/API
    • Test automation - unit/functional/API - an API Test Suite, for example
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Server load testing with Loads tool (with heka for stats collection), monitoring, logging
    • Performance testing with Loads tool (possibly with some logging, monitoring, hekad stuff hooked up)
    • Security (end to end)
    • Scaling - AWS stack (CF) - single-region/multi-region, ELB, instances, use of PHX1
    • Documentation review and support for Doc and SUMO teams
    • Env-specific deployments on localhosts and in AWS
    • Build/deploy your own - similar to Sync 1.5 and Persona
  • Track/Test (as needed)
    • Internal dependencies: repos, packages, and libs to track and to test
      • restmail, node-kvstore, node-srp, jwcrypto, picl-spec-crypto
    • Loads integration
    • SimplePush integration (has this been defined for v1/milestone 1?)
    • Heka integration
    • External dependencies: Hawk, Hapi, Good, Kibana, etc.
  • Bug reporting will happen through Bugzilla
  • Client manual testing - see client sections below
  • Client automation - see client sections below
  • Notes
    • DB/storage issues and dependencies
      • What level of testing vs. monitoring is required
    • Server-specific notes and information:
      • Stats collection is going to happen with Heka
      • Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite

TS 2: Authentication and Accounts - Frontend/Clients

  • This is Content Server and gherkin, not device- or platform-specific
  • Main parts:
    • Content Server (fxa-content-server)
    • Scrypt-Helper (fxa-scrypt-helper)
    • various gherkin support pieces (gherkin, fxa-js-client, fxa-js-client-old)
    • Desktop client - gherkin/about:accounts
    • Android client
    • Web support - gherkin/about:accounts
    • FFos
  • Full content: TBD - see TS1

TS 4: UX/UI (device/platform-specfic)

  • This is across product (desktop vs. mobile client vs. FFos vs. web)
  • See also TS 12: Localization
  • Areas of focus for testing:
    • Flow, functionality, feel per/on each client
    • Consistency, repeatability, stability between devices/platforms
    • Functional tests against all important features - Sign Up, Sign In, Password change/reset, etc.
    • Functional tests cases for all UX/UI related user stories
    • Functional tests specific to mobile clients
    • Functional tests specific to desktop clients
    • UX/UI verification against V1/Milestone 1 goals and user stories (see REFs below...)
    • QA will help or at least support the localization testing of the UI
    • Documentation review and support for Doc and SUMO teams
    • Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)
  • Bug reporting will happen through Bugzilla
  • REFs:

TS 5: Firefox OS, FTU/FTE, and Sync 1.5

TS 6: Desktop Client and Sync 1.5

  • This is possibly old Sync plus new client auth
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
  • NOTE: Add in Master Password testing
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of desktop client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional with Selenium/Mozmill
      • To be coordinated with the Fx desktop/client QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Desktop-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all desktop-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Desktop integration - all supported platforms and OS
      • Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
        • Match to Fx support (including Fx ESR)
    • FireFox command line for testing, setting up sync, authing, profiles, who knows?
  • Bug reporting will happen through Bugzilla
  • Tools
  • See Section 5 for Sync 1.5 and new Sync integration/interaction
  • See Section 6 for Sync 1.5 migration to new Sync
  • Notes
    • What about users new to sync?
  • REFs:

TS7: Web-based Testing

  • To cover basic FxA support in non-Fx browsers across various platforms/OS
  • Pull from desktop section above, since it looks like both "platforms" will share the same client structure/code.
  • Clients - only one right now - Marketplace - so build manual and automated tests around the use of FxA and Marketplace on Fx releases and other common browsers.

TS 8: Mobile Client (Android) and Sync 1.5

  • This is possibly old Sync plus new client auth
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
  • NOTE: Add in Master Password testing
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of mobile client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional - do we have anything here besides phone/os emulation?
      • To be coordinated with the Fx Mobile QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Mobile-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all mobile-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
      • Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
        • Match to Fx support
  • Bug reporting will happen through Bugzilla
  • Tools
    • Use of Android specific adk and debug tools/methods
  • See Section 5 for Sync 1.5 and new Sync integration/interaction
  • See Section 6 for Sync 1.5 migration to new Sync
  • Notes
    • What about users who only have mobile devices. How do we make sure they can sync across those devices?
    • How do they do first setup if they never used sync on a desktop device
    • What about users new to sync?
  • REFs:

TS9: Marketplace Support

  • Test FxA account activity:
    • On FFos
    • From a Firefox browser (desktop/android)
    • From a non-Fx browser using the web interface
  • See the Marketplace Test Plan for more information: TBD

TS10: Where Is My Fox/Find My Device Support

  • Test FxA account activity:
    • On FFos
    • From a Firefox browser (desktop/android)
    • From a non-Fx browser using the web interface

TS 11: FxA Documentation and SUMO

Why? Because current documentation (all of it really) is a pain point

  • Note: For QA, this is mostly a tracking responsibility
  • Updates to the old/current Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Creation of new Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Dev and QA can help define, write, and review documentation
    • UX/UI Demos and training
    • Usage demos and training (through SUMO?)
    • Documentation team support is needed
    • SUMO team support is needed
    • REFs: TBD

TS 12: FxA Localization

  • Note: For QA, this is mostly a tracking responsibility
  • across UX/UI/clients
  • See also TS 4: UX/UI
  • Mobile and Desktop clients - full UX/UI experience, wording, etc.
  • Dev and QA can help with string verfication before and after localization
  • Will need support from the FireFox Localization team and community
  • Bug reporting will happen through Bugzilla
  • REFs: TBD


TS 13: Metro (Win8/tablet) and Sync support

  • FxA/Sync/Metro integration - TBD


TS 14: Server/Backend Data Storage


TS 15: Sync 1.5 and "new" Sync Interaction [lead: kthiessen]

  • There will be some amount of time where
    • Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
    • A "migration" will start to happen where users voluntarily replace old sync with new sync
    • A "forced migration" or "shutdown period" where users will have to move from old sync to new sync
    • Only the first item will be tested in this section
  • Areas of focus for testing:
    • Functional test independent use of both old sync and new sync, per client/platform/device
    • Functional test interactions between the two (combined use)(especially if per device syncing is supported), per client/platform/device
    • Test/verify data storage for both sync models per account/device
    • UX/UI interactions and methods, especially for Sign Up and Sign In on both Sync models, and other supported preferences and UI features
    • Old Sync/new Sync verification against V1/Milestone 1 goals
    • General account interaction: old sync, new Sync, Persona
    • Documentation review pertaining to old Sync, new Sync
  • Bug reporting will happen through Bugzilla
  • See Section 6 for the specific migration testing tasks
  • REFs:

TS 16: Migration to "new" Sync [lead: kthiessen]

  • At some point, all old Sync users will either have to
    • Migrate to new Sync
    • Stop using Sync
  • Areas of focus for testing:
    • QA will need to prepare a set of steps, before and after tests, to verify migration
    • QA will need to interrogate/investigate account contents from old and new sync to verify migration
    • All of this QA work can be done in Stage first to verify migration will actually run as expected.
    • Dev and OPs suppport will be needed for the test and verification steps.
    • Documentation review pertaining to old Sync, new Sync and migration steps/strategies
  • Bug reporting will happen through Bugzilla
  • See Migration Strategy for steps involved and timing
  • Notes:
    • From the user end, we want to make sure as much data as possible is migrated over to new sync.
    • Old Sync data will exist in PHX1 datacenter, new Sync data may exist there or in AWS
    • I assume all this work will be done manually by the user (since data is encrypted, OPs can not automatically move data over). Will be a challenge to see how this is handled per account vs.per device (if per device syncing is supported).
  • REFs:

Main References

Wikis and Etherpads

GitHub Repositories

  • FFos specific
    • TBD
  • Server-side:
    • TBD

Travis CI

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/SyncTestPlanV1&oldid=933007"