TestEngineering/Services/SyncTestPlanV1: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 17: Line 17:


== User Stories ==
== User Stories ==
* FxA User Stories on the web: https://wiki.mozilla.org/User_Services/Sync/v1#User_stories
* Start here for Android and desktop/web:  
* FFos PM/User Stories Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941723&hide_resolved=0
** https://wiki.mozilla.org/User_Services/Sync/Relaunch#Product_Requirements_.26_User_Stories
* FxA PM/User Stories Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=909322&hide_resolved=0
* For FFos: TBD


== Building Test Cases ==
== Building Test Cases ==

Revision as of 00:55, 13 February 2014

This is the Sync/Picl Test Plan, V1

Intro/Summary/Notes

Intro

  • The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.1.
  • It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.

Summary

  • A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc...

Notes

  • FxA is scheduled to go out with Firefox 29. This will replace old sync account functionality and use.
  • Sync users from Firefox 28 and earlier will need to create new FxA accounts when they upgrade to Fx29.
  • Once that happens, devices hooked to sync for Fx28 and older will have to be unlinked and then added to the new FxA account in order to sync.
  • If a user wants to hold all devices at old sync, he/she must not upgrade to Fx29.
  • A user can not have accounts with devices across versions of Fx older than 29.

User Stories

Building Test Cases

  • QA needs at least one resource, probably from SoftVision, to convert the current v1/m1 user stories into client-side test cases.
  • Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
  • Scheduling: TBD

Testing the Major Engineering Milestones

How to keep focused while staying sane...

  • Pre-release testing
    • Unit testing of fxa-auth-server
      • localhost
      • Dev hosts
    • Unit testing of fxa-content-server
    • Unit testing for fxa-scrypt-helper
    • Unit testing of fxa-js-client and fxa-js-client-old
    • Review of mocks/UI flows: https://wiki.mozilla.org/Identity/UX
  • First try/nightly build on ELM
    • See the client-side bugs for each service
    • User stories - are they covered?
    • Engineering goals - are they covered?
    • Functional checks - does this thing work?
    • Write test cases
    • Write client automation for desktop/android/FFos
  • Successive builds on ELM
    • Same list as above
  • Official Nightly builds
    • Full testing, including Documentation, Localization, and SUMO/support information

Testing the MVP

  • TBD - this really depends on which platforms are supported for MVP. Initially, this may be just FxA on FFos version 1.4.

Schedule

QA Team

  • Services QA
    • Edwin Wong - Manager
    • James Bonacci
    • John Morrison
    • Karl Thiessen
    • Peter deHaan
  • QA
    • Various members of the Firefox OS, Desktop, and Mobile teams will be involved in the testing.
  • QA head count (DRAFT)
    • 1 FT automation/tools lead + QA team support
    • 1 FT android client lead + QA team support
    • 1 FT desktop client lead + QA team support
    • 1 FT web lead + QA team support
    • 1 FT FFos lead + QA team support
    • 1 FT Auth/Storage lead + QA team support
    • 1 FT server/security/db lead + QA team support - TBD
    • A subset of the FF and Services QA teams for testing when this goes to FF Nightly.
    • The entire FF and Services QA teams as part of normal testing of FF Aurora, Beta, Release channels
    • Challenges
      • Recruiting QA and SoftVision (or other contractors) for testing Try builds
      • Recruiting QA team for full testing during normal FF releases
      • Recruiting community for testing Firefox channels when they become available (esp. if they have experience with Sync 1.x)

OPs Information

  • OPs has a significant number of really detailed Mana pages for deploying, testing, maintaining, and monitoring our Stage and Production environments.
  • Talk to a member of the OPs team for more information and for the Mana links to get you started.

Email Lists and IRC

Bug Lists and Reporting

  • Dev-related issues will be mixed within Bugzilla and GitHub (but see trees/meta info below)
  • Product issues will all live in Bugzilla
  • User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
    • Classification: TBD
    • Products: TBD
    • Components: TBD
  • Note: We may need to cross link Bugzilla bugs with GitHub issues

Bug/Issue Testing and Verification

  • TBD
  • Need to add something here to talk about how to track and test fixes/patches/builds in Bugzilla as well as GitHub.
  • Also, what to do with regressions in GitHub vs. Bugzilla...

Firefox Releases

Platforms/OS

  • Mac 10.7.x and 10.8.x
  • Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
  • Windows 8 (Metro mode and desktop mode). Does this come in a 32bit flavor?
  • Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
  • Android 2.x - 4.x

ELM Nightly Builds

Dev Test Environments

  • Note: You will need Dev/QA AWS credentials to access these servers.

LoadTest Environment

  • Note: You will need Dev/QA AWS credentials to access these servers.

Test Sections/Suites

TS 1: Authentication and Accounts - Backend

  • This is Authentication Server/fxa-auth (IdP), but not device-specific
  • Main parts:
    • fxa-auth (IdP) authentication service
    • MySQL data storage
    • Heka data/log aggregation
  • Areas of focus for testing: primarily server-side
    • Manual testing - unit/functional/API
    • Test automation - unit/functional/API - an API Test Suite, for example
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Server load testing with Loads tool (with heka for stats collection), monitoring, logging
    • Performance testing with Loads tool (possibly with some logging, monitoring, hekad stuff hooked up)
    • Security (end to end)
    • Scaling - AWS stack (CF) - single-region/multi-region, ELB, instances, use of PHX1
    • Documentation review and support for Doc and SUMO teams
    • Env-specific deployments on localhosts and in AWS
    • Build/deploy your own - similar to Sync 1.1 and Persona
  • Track/Test (as needed)
    • Internal dependencies: repos, packages, and libs to track and to test
      • restmail, node-kvstore, node-srp, jwcrypto, picl-spec-crypto
    • Loads integration
    • SimplePush integration (has this been defined for v1/milestone 1?)
    • Heka integration
    • External dependencies: Hawk, Hapi, Good, Kibana, etc.
  • Bug reporting will happen through Bugzilla
  • Client manual testing - see client sections below
  • Client automation - see client sections below
  • Notes
    • DB/storage issues and dependencies
      • What level of testing vs. monitoring is required
    • Server-specific notes and information:
      • Stats collection is going to happen with Heka
      • Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite

TS 2: Authentication and Accounts - Frontend/Clients

  • This is Content Server and gherkin, not device- or platform-specific
  • Main parts:
    • Content Server (fxa-content-server)
    • Scrypt-Helper (fxa-scrypt-helper)
    • various gherkin support pieces (gherkin, fxa-js-client, fxa-js-client-old)
    • Desktop client - gherkin/about:accounts
    • Android client
    • Web support - gherkin/about:accounts
    • FFos
  • Full content: TBD - see TS1

TS 4: UX/UI (device/platform-specfic)

  • This is across product (desktop vs. mobile client vs. FFos vs. web)
  • See also TS 12: Localization
  • Areas of focus for testing:
    • Flow, functionality, feel per/on each client
    • Consistency, repeatability, stability between devices/platforms
    • Functional tests against all important features - Sign Up, Sign In, Password change/reset, etc.
    • Functional tests cases for all UX/UI related user stories
    • Functional tests specific to mobile clients
    • Functional tests specific to desktop clients
    • UX/UI verification against V1/Milestone 1 goals and user stories (see REFs below...)
    • QA will help or at least support the localization testing of the UI
    • Documentation review and support for Doc and SUMO teams
    • Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)
  • Bug reporting will happen through Bugzilla
  • REFs:

TS 5: Firefox OS, FTU/FTE, and Sync 1.1

TS 6: Desktop Client and Sync 1.1

  • This is possibly old Sync plus new client auth
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.1 in this suite.
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of desktop client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional with Selenium/Mozmill
      • To be coordinated with the FF desktop/client QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Desktop-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all desktop-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Desktop integration - all supported platforms and OS
      • Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
        • Match to FF support (including FF ESR)
    • FireFox command line for testing, setting up sync, authing, profiles, who knows?
  • Bug reporting will happen through Bugzilla
  • Tools
  • See Section 5 for Sync 1.1 and new Sync integration/interaction
  • See Section 6 for Sync 1.1 migration to new Sync
  • Notes
    • What about users new to sync?
  • REFs:

TS7: Web-based Client

  • TBD - to cover gherkin support in non-FF browsers across various platforms/OS
  • Pull from desktop section above, since it looks like both "platforms" will share the same client structure/code.
  • TBD - sync interaction from plain web front-end

TS 8: Mobile Client (Android) and Sync 1.1

  • This is possibly old Sync plus new client auth
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.1 in this suite.
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of mobile client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional - do we have anything here besides phone/os emulation?
      • To be coordinated with the FF Mobile QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Mobile-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all mobile-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
      • Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
        • Match to FF support
  • Bug reporting will happen through Bugzilla
  • Tools
    • Use of Android specific adk and debug tools/methods
  • See Section 5 for Sync 1.1 and new Sync integration/interaction
  • See Section 6 for Sync 1.1 migration to new Sync
  • Notes
    • What about users who only have mobile devices. How do we make sure they can sync across those devices?
    • How do they do first setup if they never used sync on a desktop device
    • What about users new to sync?
  • REFs:

TS9: Marketplace Support (all platforms)

  • TBD - to cover FxA and Marketplace interaction on FFos and all supported platforms
  • Test FxA account activity:
    • On FFos
    • From a Firefox browser (desktop/android)
    • From a non-FF browser using gherkin support

TS10: WhereIsMyFox Support (all platforms)

  • TBD - to cover FxA and WhereIsMyFox interaction on FFos and all supported platforms
  • Test FxA account activity:
    • On FFos
    • From a Firefox browser (desktop/android)
    • From a non-FF browser using gherkin support

TS 11: FxA Documentation and SUMO

Why? Because current documentation (all of it really) is a pain point

  • Note: For QA, this is mostly a tracking responsibility
  • Updates to the old/current Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Creation of new Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Dev and QA can help define, write, and review documentation
    • UX/UI Demos and training
    • Usage demos and training (through SUMO?)
    • Documentation team support is needed
    • SUMO team support is needed
    • REFs: TBD

TS 12: FxA Localization

  • Note: For QA, this is mostly a tracking responsibility
  • across UX/UI/clients
  • See also TS 4: UX/UI
  • Mobile and Desktop clients - full UX/UI experience, wording, etc.
  • Dev and QA can help with string verfication before and after localization
  • Will need support from the FireFox Localization team and community
  • Bug reporting will happen through Bugzilla
  • REFs: TBD


TS 13: Metro (Win8/tablet) and Sync support

  • FxA/Sync/Metro integration - TBD


TS 14: Server/Backend Data Storage


TS 15: Sync 1.1 and "new" Sync Interaction [lead: kthiessen]

  • There will be some amount of time where
    • Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
    • A "migration" will start to happen where users voluntarily replace old sync with new sync
    • A "forced migration" or "shutdown period" where users will have to move from old sync to new sync
    • Only the first item will be tested in this section
  • Areas of focus for testing:
    • Functional test independent use of both old sync and new sync, per client/platform/device
    • Functional test interactions between the two (combined use)(especially if per device syncing is supported), per client/platform/device
    • Test/verify data storage for both sync models per account/device
    • UX/UI interactions and methods, especially for Sign Up and Sign In on both Sync models, and other supported preferences and UI features
    • Old Sync/new Sync verification against V1/Milestone 1 goals
    • General account interaction: old sync, new Sync, Persona
    • Documentation review pertaining to old Sync, new Sync
  • Bug reporting will happen through Bugzilla
  • See Section 6 for the specific migration testing tasks
  • REFs:

TS 16: Migration to "new" Sync [lead: kthiessen]

  • At some point, all old Sync users will either have to
    • Migrate to new Sync
    • Stop using Sync
  • Areas of focus for testing:
    • QA will need to prepare a set of steps, before and after tests, to verify migration
    • QA will need to interrogate/investigate account contents from old and new sync to verify migration
    • All of this QA work can be done in Stage first to verify migration will actually run as expected.
    • Dev and OPs suppport will be needed for the test and verification steps.
    • Documentation review pertaining to old Sync, new Sync and migration steps/strategies
  • Bug reporting will happen through Bugzilla
  • See Migration Strategy for steps involved and timing
  • Notes:
    • From the user end, we want to make sure as much data as possible is migrated over to new sync.
    • Old Sync data will exist in PHX1 datacenter, new Sync data may exist there or in AWS
    • I assume all this work will be done manually by the user (since data is encrypted, OPs can not automatically move data over). Will be a challenge to see how this is handled per account vs.per device (if per device syncing is supported).
  • REFs:

Main References

Wikis and Etherpads

GitHub Repositories

  • FFos specific
    • TBD

Travis CI

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/SyncTestPlanV1&oldid=925233"