TestEngineering/Services/SyncTestPlanV1: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 163: Line 163:
= Firefox Releases =
= Firefox Releases =
* Early stages
* Early stages
** Desktop try/nightly builds, 32bit and 64bit, as required, source - ELM: TBD
** ELM builds for Android and Desktop (see below)
** Android Dev build, source: TBD
** FFos unrelease builds: TBD
** FFos Dev build, source: TBD
* Nightly
* Nightly
** http://nightly.mozilla.org/
** Similar requirements to above
** Similar requirements to above
* Three-channel focus with latest features, fixes, etc. starting in Nightly
* Three-channel focus with latest features, fixes, etc.
** But, basically, at this point, feature set should be frozen
** Aurora: Index of /pub/mozilla.org/firefox/nightly/latest-mozilla-aurora
** Beta: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/latest-beta
** Release: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases
** Android-specific: http://ftp.mozilla.org/pub/mozilla.org/mobile


= Platforms/OS =
= Platforms/OS =

Revision as of 00:38, 13 February 2014

This is the Sync/Picl Test Plan, V1

Intro/Summary/Notes

Intro

  • The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.1.
  • It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.

Summary

  • A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc...

Notes

  • Q4 2013: It looks like Firefox Account support on FFos will take priority going forward. Just behind this will be Firefox Account support on Desktop, Android, and Web (already in progress) and the short-term fixes to Sync (if any).
  • This is a living document. As such, assume very regular changes, updates, and rewrites as we go through the Dev milestones of v1.
  • This document is designed to accompany the v1 release. Although it will be used as a reference for the various Dev milestones for v1, it is not specific to any one milestone.
  • Milestone-specific test plans could be broken out from here to focus on very specific test sections, as defined below.
  • Requested sync-able collections for v1/m1:
    • Awesomebar/History
    • Passwords
    • Open Tabs
    • Bookmarks
    • (all four supported per account, but not per device)
  • Requested basic feature set for v1/m1:
    • Device compatibility (desktop/mobile only, not FFox)
    • Setup (Sign Up, Sign In)
    • Account Management
    • Password change/reset(forgot password flow)
    • Detach a device (but keep the account)
    • Better security/encryption
    • Performance/stability

User Stories

Building Test Cases

  • QA needs at least one resource, probably from SoftVision, to convert the current v1/m1 user stories into client-side test cases.
  • Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
  • Scheduling: TBD

Testing the Major Engineering Milestones

How to keep focused while staying sane...

  • Pre-release testing
    • Unit testing of fxa-auth-server
      • localhost
      • Dev hosts
    • Unit testing of fxa-content-server
    • Unit testing for fxa-scrypt-helper
    • Unit testing of fxa-js-client and fxa-js-client-old
    • Review of mocks/UI flows: https://wiki.mozilla.org/Identity/UX
  • First try/nightly build on ELM
    • See the client-side bugs for each service
    • User stories - are they covered?
    • Engineering goals - are they covered?
    • Functional checks - does this thing work?
    • Write test cases
    • Write client automation for desktop/android/FFos
  • Successive builds on ELM
    • Same list as above
  • Official Nightly builds
    • Full testing, including Documentation, Localization, and SUMO/support information

Testing the MVP

  • TBD - this really depends on which platforms are supported for MVP. Initially, this may be just FxA on FFos version 1.4.

Schedule

QA Team

  • Services QA
    • Edwin Wong - Manager
    • James Bonacci
    • John Morrison
    • Karl Thiessen
    • Peter deHaan
  • QA
    • Various members of the Firefox OS, Desktop, and Mobile teams will be involved in the testing.
  • QA head count (DRAFT)
    • 1 FT automation/tools lead + QA team support
    • 1 FT android client lead + QA team support
    • 1 FT desktop client lead + QA team support
    • 1 FT web lead + QA team support
    • 1 FT FFos lead + QA team support
    • 1 FT Auth/Storage lead + QA team support
    • 1 FT server/security/db lead + QA team support - TBD
    • A subset of the FF and Services QA teams for testing when this goes to FF Nightly.
    • The entire FF and Services QA teams as part of normal testing of FF Aurora, Beta, Release channels
    • Challenges
      • Recruiting QA and SoftVision (or other contractors) for testing Try builds
      • Recruiting QA team for full testing during normal FF releases
      • Recruiting community for testing Firefox channels when they become available (esp. if they have experience with Sync 1.x)

OPs Information

  • OPs has a significant number of really detailed Mana pages for deploying, testing, maintaining, and monitoring our Stage and Production environments.
  • Talk to a member of the OPs team for more information and for the Mana links to get you started.

Email Lists and IRC

Bug Lists and Reporting

  • Dev-related issues will be mixed within Bugzilla and GitHub (but see trees/meta info below)
  • Product issues will all live in Bugzilla
  • User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
    • Classification: TBD
    • Products: TBD
    • Components: TBD
  • Note: We may need to cross link Bugzilla bugs with GitHub issues

Bug/Issue Testing and Verification

  • TBD
  • Need to add something here to talk about how to track and test fixes/patches/builds in Bugzilla as well as GitHub.
  • Also, what to do with regressions in GitHub vs. Bugzilla...

Firefox Releases

Platforms/OS

  • Mac 10.7.x and 10.8.x
  • Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
  • Windows 8 (Metro mode and desktop mode). Does this come in a 32bit flavor?
  • Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
  • Android 2.x - 4.x

ELM Nightly Builds

Dev Test Environments

  • Note: You will need Dev/QA AWS credentials to access these servers.

LoadTest Environment

  • Note: You will need Dev/QA AWS credentials to access these servers.

Test Sections/Suites

TS 1: Authentication and Accounts - Backend

  • This is Authentication Server/fxa-auth (IdP), but not device-specific
  • Main parts:
    • fxa-auth (IdP) authentication service
    • MySQL data storage
    • Heka data/log aggregation
  • Areas of focus for testing: primarily server-side
    • Manual testing - unit/functional/API
    • Test automation - unit/functional/API - an API Test Suite, for example
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Server load testing with Loads tool (with heka for stats collection), monitoring, logging
    • Performance testing with Loads tool (possibly with some logging, monitoring, hekad stuff hooked up)
    • Security (end to end)
    • Scaling - AWS stack (CF) - single-region/multi-region, ELB, instances, use of PHX1
    • Documentation review and support for Doc and SUMO teams
    • Env-specific deployments on localhosts and in AWS
    • Build/deploy your own - similar to Sync 1.1 and Persona
  • Track/Test (as needed)
    • Internal dependencies: repos, packages, and libs to track and to test
      • restmail, node-kvstore, node-srp, jwcrypto, picl-spec-crypto
    • Loads integration
    • SimplePush integration (has this been defined for v1/milestone 1?)
    • Heka integration
    • External dependencies: Hawk, Hapi, Good, Kibana, etc.
  • Bug reporting will happen through Bugzilla
  • Client manual testing - see client sections below
  • Client automation - see client sections below
  • Notes
    • DB/storage issues and dependencies
      • What level of testing vs. monitoring is required
    • Server-specific notes and information:
      • Stats collection is going to happen with Heka
      • Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite

TS 2: Authentication and Accounts - Frontend/Clients

  • This is Content Server and gherkin, not device- or platform-specific
  • Main parts:
    • Content Server (fxa-content-server)
    • Scrypt-Helper (fxa-scrypt-helper)
    • various gherkin support pieces (gherkin, fxa-js-client, fxa-js-client-old)
    • Desktop client - gherkin/about:accounts
    • Android client
    • Web support - gherkin/about:accounts
    • FFos
  • Full content: TBD - see TS1

TS 4: UX/UI (device/platform-specfic)

  • This is across product (desktop vs. mobile client vs. FFos vs. web)
  • See also TS 12: Localization
  • Areas of focus for testing:
    • Flow, functionality, feel per/on each client
    • Consistency, repeatability, stability between devices/platforms
    • Functional tests against all important features - Sign Up, Sign In, Password change/reset, etc.
    • Functional tests cases for all UX/UI related user stories
    • Functional tests specific to mobile clients
    • Functional tests specific to desktop clients
    • UX/UI verification against V1/Milestone 1 goals and user stories (see REFs below...)
    • QA will help or at least support the localization testing of the UI
    • Documentation review and support for Doc and SUMO teams
    • Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)
  • Bug reporting will happen through Bugzilla
  • REFs:

TS 5: Firefox OS, FTU/FTE, and Sync 1.1

TS 6: Desktop Client and Sync 1.1

  • This is possibly old Sync plus new client auth
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.1 in this suite.
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of desktop client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional with Selenium/Mozmill
      • To be coordinated with the FF desktop/client QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Desktop-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all desktop-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Desktop integration - all supported platforms and OS
      • Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
        • Match to FF support (including FF ESR)
    • FireFox command line for testing, setting up sync, authing, profiles, who knows?
  • Bug reporting will happen through Bugzilla
  • Tools
  • See Section 5 for Sync 1.1 and new Sync integration/interaction
  • See Section 6 for Sync 1.1 migration to new Sync
  • Notes
    • What about users new to sync?
  • REFs:

TS7: Web-based Client

  • TBD - to cover gherkin support in non-FF browsers across various platforms/OS
  • Pull from desktop section above, since it looks like both "platforms" will share the same client structure/code.
  • TBD - sync interaction from plain web front-end

TS 8: Mobile Client (Android) and Sync 1.1

  • This is possibly old Sync plus new client auth
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.1 in this suite.
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of mobile client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional - do we have anything here besides phone/os emulation?
      • To be coordinated with the FF Mobile QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Mobile-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all mobile-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
      • Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
        • Match to FF support
  • Bug reporting will happen through Bugzilla
  • Tools
    • Use of Android specific adk and debug tools/methods
  • See Section 5 for Sync 1.1 and new Sync integration/interaction
  • See Section 6 for Sync 1.1 migration to new Sync
  • Notes
    • What about users who only have mobile devices. How do we make sure they can sync across those devices?
    • How do they do first setup if they never used sync on a desktop device
    • What about users new to sync?
  • REFs:

TS9: Marketplace Support (all platforms)

  • TBD - to cover FxA and Marketplace interaction on FFos and all supported platforms
  • Test FxA account activity:
    • On FFos
    • From a Firefox browser (desktop/android)
    • From a non-FF browser using gherkin support

TS10: WhereIsMyFox Support (all platforms)

  • TBD - to cover FxA and WhereIsMyFox interaction on FFos and all supported platforms
  • Test FxA account activity:
    • On FFos
    • From a Firefox browser (desktop/android)
    • From a non-FF browser using gherkin support

TS 11: FxA Documentation and SUMO

Why? Because current documentation (all of it really) is a pain point

  • Note: For QA, this is mostly a tracking responsibility
  • Updates to the old/current Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Creation of new Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Dev and QA can help define, write, and review documentation
    • UX/UI Demos and training
    • Usage demos and training (through SUMO?)
    • Documentation team support is needed
    • SUMO team support is needed
    • REFs: TBD

TS 12: FxA Localization

  • Note: For QA, this is mostly a tracking responsibility
  • across UX/UI/clients
  • See also TS 4: UX/UI
  • Mobile and Desktop clients - full UX/UI experience, wording, etc.
  • Dev and QA can help with string verfication before and after localization
  • Will need support from the FireFox Localization team and community
  • Bug reporting will happen through Bugzilla
  • REFs: TBD


TS 13: Metro (Win8/tablet) and Sync support

  • FxA/Sync/Metro integration - TBD


TS 14: Server/Backend Data Storage


TS 15: Sync 1.1 and "new" Sync Interaction [lead: kthiessen]

  • There will be some amount of time where
    • Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
    • A "migration" will start to happen where users voluntarily replace old sync with new sync
    • A "forced migration" or "shutdown period" where users will have to move from old sync to new sync
    • Only the first item will be tested in this section
  • Areas of focus for testing:
    • Functional test independent use of both old sync and new sync, per client/platform/device
    • Functional test interactions between the two (combined use)(especially if per device syncing is supported), per client/platform/device
    • Test/verify data storage for both sync models per account/device
    • UX/UI interactions and methods, especially for Sign Up and Sign In on both Sync models, and other supported preferences and UI features
    • Old Sync/new Sync verification against V1/Milestone 1 goals
    • General account interaction: old sync, new Sync, Persona
    • Documentation review pertaining to old Sync, new Sync
  • Bug reporting will happen through Bugzilla
  • See Section 6 for the specific migration testing tasks
  • REFs:

TS 16: Migration to "new" Sync [lead: kthiessen]

  • At some point, all old Sync users will either have to
    • Migrate to new Sync
    • Stop using Sync
  • Areas of focus for testing:
    • QA will need to prepare a set of steps, before and after tests, to verify migration
    • QA will need to interrogate/investigate account contents from old and new sync to verify migration
    • All of this QA work can be done in Stage first to verify migration will actually run as expected.
    • Dev and OPs suppport will be needed for the test and verification steps.
    • Documentation review pertaining to old Sync, new Sync and migration steps/strategies
  • Bug reporting will happen through Bugzilla
  • See Migration Strategy for steps involved and timing
  • Notes:
    • From the user end, we want to make sure as much data as possible is migrated over to new sync.
    • Old Sync data will exist in PHX1 datacenter, new Sync data may exist there or in AWS
    • I assume all this work will be done manually by the user (since data is encrypted, OPs can not automatically move data over). Will be a challenge to see how this is handled per account vs.per device (if per device syncing is supported).
  • REFs:

Main References

Wikis and Etherpads

GitHub Repositories

  • FFos specific
    • TBD

Travis CI

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/SyncTestPlanV1&oldid=925223"