TestEngineering/Services/SyncTestPlanV1

From MozillaWiki
Jump to navigation Jump to search

This is the Sync/Picl Test Plan, V1

Intro/Summary/Notes

Intro

  • TBD

Summary

  • A complete snapshot of what it takes to test and release picl/sync: tools, processes, schedules, resources, etc

Notes

  • This is a living document. As such, assume very regular changes, updates, and rewrites as we go through the Dev milestones of v1.
  • This document is designed to accompany the v1 release. Although it will be used as a reference for the various Dev milestones for v1, it is not specific to any one milestone.
  • Milestone-specific test plans could be broken out from here to focus on very specific test sections, as defined below.
  • Requested sync-able collections for V1/M1:
    • Awesomebar/History
    • Passwords
    • Open Tabs
    • Bookmarks
    • (supported per account, but not per device)
  • Requested basic feature set for V1/M1:
    • Device compatibility (desktop/mobile only, not FFox)
    • Setup (Sign Up, Sign In)
    • Account Management
    • Password change/reset(forgot)
    • Detach a device (maintain the account)
    • Better security/encryption
    • Performance/stability

Schedule

QA Team

  • Services QA
    • Edwin Wong - Manager
    • James Bonacci - Sync/Picl Lead
    • John Morrison
    • Karl Thiessen
    • Peter deHaan
  • QA
    • Various members of the Firefox Desktop and Mobile teams will be involved in the testing of V1.
  • QA head count (DRAFT)
    • 1 FT automation/tools lead + QA team support
    • 1 FT android client lead + QA team support
    • 1 FT desktop client lead + QA team support
    • 1 FT FFos lead + QA team support
    • 1 FT Auth/Storage lead + QA team support
    • 1 FT server/security/db lead + QA team support - TBD
    • A subset of the FF and Services QA teams for testing when this goes to FF Nightly.
    • The entire FF and Services QA teams as part of normal testing of FF Aurora, Beta, Release channels
    • Challenges
      • Recruiting QA and SoftVision (or other contractors) for testing Try builds
      • Recruiting QA team for full testing during normal FF releases
      • Recruiting community for testing Firefox channels when they become available (esp. if they have experience with Sync 1.x)

Email Lists and IRC

  • Services QA: services-qa@mozilla.com
  • QA Teams: qa@mozilla.com
  • Sync and Picl: sync-dev@mozilla.org
  • IRC: #picl
    • #sync, #androidsync, #identity

Bug Reporting

Firefox Releases

  • Early stages
    • Desktop try builds, 32bit and 64bit, as required, source: TBD
    • Android Dev build, source: TBD
    • FFos Dev build, source: TBC
  • Nightly
    • Similar to above
  • Three-channel focus with latest features, fixes, etc in Nightly
    • But, basically, at this point, feature set should be frozen

Platforms/OS

  • Mac 10.7.x and 10.8.x
  • Windows XP 32bit, Windows 7 32bit, Windows 7 64bit)
  • Windows 8 (Metro mode and desktop mode). Does this come in a 32bit flavor?
  • Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
    • The Goal is to stay consistent with requirements for FireFox and for FFos.

Test Sections/Suites

TS 1: Authentication/Accounts

  • This is Picl-IdP/KeyServer/FF Accounts Bridge/Scrypt Helper
  • Main parts:
    • PiCL-IdP/KeyServer (client-server)
    • Scrypt-Helper (server-only?)
    • FireFox Account Bridge (server-only?)
  • Areas of focus for testing: primarily server-side
    • Manual testing - unit/functional/API
    • Test automation - unit/functional/API (probably driven by the Loads load test tool)
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Server load testing with Loads tool (with heka for stats collection), monitoring, logging
    • Performance testing with Loads tool (possibly with some logging, monitoring, hekad stuff hooked up)
    • Security (end to end)
    • Scaling - AWS stack (CF) - single-region/multi-region, ELB, instnaces, use of PHX1
    • Documentation review and support for Doc and SUMO teams
    • Auth/API verification against V1/M1 goals (if needed)
    • Env-specific deployments on localhosts and in AWS
    • Build/deploy your own - similar to Sync 1.1 and Persona
  • Track/Test (as needed)
    • Internal dependencies: repos, packages, and libs to track and to test
      • restmail, scrypt-helper, node-kvstore, node-srp, jwcrypto, picl-spec-crypto, firefox-account-bridge, picl-gherkin
    • Loads integration
    • SimplePush integration (has this been defined for v1/milestone 1?)
    • Heka integration
    • External dependencies: Hawk, Hapi, Good, Kibana, etc.
  • Bug reporting will happen through Bugzilla
  • Client manual testing - see client sections below
  • Client automation - see client sections below
  • Notes
    • DB/storage issues and dependencies
      • What level of testing vs. monitoring is required
    • Server-specific notes and information:
      • Stats collection is going to happen with Heka
      • Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite
  • REFs:

TS 2: UX/UI

TS 3: Desktop client (Firefox channels)

  • This is possibly old Sync plus new client auth
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of desktop client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional with Selenium/Mozmill
      • To be coordinated with the FF desktop/client QA team
    • Firefox integration (try builds only)
    • Firefox integration (4 channels)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Desktop-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all desktop-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Desktop integration - all supported platforms and OS
      • Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
        • Match to FF support (including FF ESR)
    • FireFox command line for testing, setting up sync, authing, profiles, who knows?
  • Bug reporting will happen through Bugzilla
  • Tools
  • See Section 5 for Sync 1.1 and new Sync integration/interaction
  • See Section 6 for Sync 1.1 migration to new Sync
  • Notes
    • What about users new to sync?
  • REFs:

TS 4: Mobile client (Android)

  • This is possibly old Sync plus new client auth
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of mobile client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional - do we have anything here besides phone/os emulation?
      • To be coordinated with the FF Mobile QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Mobile-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all mobile-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
      • Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
        • Match to FF support
  • Bug reporting will happen through Bugzilla
  • Tools
    • Use of Android specific adk and debug tools/methods
  • See Section 5 for Sync 1.1 and new Sync integration/interaction
  • See Section 6 for Sync 1.1 migration to new Sync
  • Notes
    • What about users who only have mobile devices. How do we make sure they can sync across those devices?
    • How do they do first setup if they never used sync on a desktop device
    • What about users new to sync?
  • REFs:

TS 5: Old Sync/New Sync Interaction

  • There will be some amount of time where
    • Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
    • A "migration" will start to happen where users voluntarily replace old sync with new sync
    • A "forced migration" or "shutdown period" where users will have to move from old sync to new sync
    • Only the first item will be tested in this section
  • Areas of focus for testing:
    • Functional test independent use of both old sync and new sync, per client/platform/device
    • Functional test interactions between the two (combined use)(especially if per device syncing is supported), per client/platform/device
    • Test/verify data storage for both sync models per account/device
    • UX/UI interactions and methods, especially for Sign Up and Sign In on both Sync models, and other supported preferences and UI features
    • Old Sync/new Sync verification against V1/Milestone 1 goals
    • General account interaction: old sync, new Sync, Persona
    • Documentation review pertaining to old Sync, new Sync
  • Bug reporting will happen through Bugzilla
  • See Section 6 for the specific migration testing tasks
  • REFs:

TS 6: Migration to "new" Sync

  • At some point, all old Sync users will either have to
    • Migrate to new Sync
    • Stop using Sync
  • Areas of focus for testing:
    • QA will need to prepare a set of steps, before and after tests, to verify migration
    • QA will need to interrogate/investigate account contents from old and new sync to verify migration
    • All of this QA work can be done in Stage first to verify migration will actually run as expected.
    • Dev and OPs suppport will be needed for the test and verification steps.
    • Documentation review pertaining to old Sync, new Sync and migration steps/strategies
  • Bug reporting will happen through Bugzilla
  • See Migration Strategy for steps involved and timing
  • Notes:
    • From the user end, we want to make sure as much data as possible is migrated over to new sync.
    • Old Sync data will exist in PHX1 datacenter, new Sync data may exist there or in AWS
    • I assume all this work will be done manually by the user (since data is encrypted, OPs can not automatically move data over). Will be a challenge to see how this is handled per account vs.per device (if per device syncing is supported).
  • REFs:

TS 7: Documentation

Why? Because current documentation (all of it really) is a pain point

  • Note: For QA, this is mostly a tracking responsibility
  • Updates to the old/current Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Creation of new Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Dev and QA can help define, write, and review documentation
    • UX/UI Demos and training
    • Usage demos and training (through SUMO?)
    • Documentation team support is needed
    • SUMO team support is needed
    • REFs: TBD

TS 8: Localization

  • Note: For QA, this is mostly a tracking responsibility
    • across UX/UI/clients
  • Mobile and Desktop clients - full UX/UI experience, wording, etc.
  • Dev and QA can help with string verfication before and after localization
  • Will need support from the FireFox Localization team and community
  • Bug reporting will happen through Bugzilla
  • REFs: TBD

TS 9: Firefox OS

  • TBD

TS 10: Metro (Win8/tablet) and Sync support

  • TBD

TS 11: Server/Backend Data Storage

  • TBD
    • Note this is the Sync Server portion of the product.
    • The Auth data store is covered in Section 1: Authentication.

Main References

Wikis and Etherpads

GitHub Repositories

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/SyncTestPlanV1&oldid=695826"