TestEngineering/Services/SyncTestPlanV1

This is the Test Plan for FxA, Tokenserver, and Sync 1.5

Intro/Summary/Notes

Intro

• The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.5.
• It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.

Summary

• A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc...

Notes

• FxA is scheduled to go out with Firefox 29. This will replace old sync account functionality and use.
• Sync users from Firefox 28 and earlier will need to create new FxA accounts when they upgrade to Fx29.
• Once that happens, devices hooked to sync for Fx28 and older will have to be unlinked and then added to the new FxA account in order to sync.
• If a user wants to hold all devices at old sync, he/she must not upgrade to Fx29.
• A user can not have accounts with devices across versions of Fx older than 29.

User Stories

• Start here for Android and desktop/web:
  • https://wiki.mozilla.org/User_Services/Sync/Relaunch#Product_Requirements_.26_User_Stories
• For FFos: TBD

Building Test Cases

• QA needs at least one resource, probably from SoftVision, to convert the current Android, desktop/web, and FFos user stories into client-side test cases.
• Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
• Scheduling: TBD

Schedule

• NEWER: https://wiki.mozilla.org/User_Services/Sync/Relaunch#Burndown_Worksheet
• NEW: https://wiki.mozilla.org/Identity/Roadmap
• OLD: https://wiki.mozilla.org/User_Services/Sync/v1

QA Team

• Services QA
  • Edwin Wong - Manager
  • James Bonacci
  • John Morrison
  • Karl Thiessen
  • Peter deHaan
  • Richard Pappalardo

• QA
  • Various members of the Firefox OS, Desktop, and Mobile teams will be involved in the testing.

• Note: FxA and Sync Teams information can be found here:
  • https://wiki.mozilla.org/User_Services/Sync/Relaunch/DRIs
  • https://wiki.mozilla.org/Identity/FirefoxAccounts#Team

OPs Information

• OPs has a significant number of really detailed Mana pages for deploying, testing, maintaining, and monitoring our Stage and Production environments.
• Talk to a member of the OPs team for more information and for the Mana links to get you started.

Email Lists and IRC

• Services QA: services-qa@mozilla.com
• QA Teams: qa@mozilla.com
• Sync and FxA:
  • NEW: dev-fxacct@mozila.org, https://mail.mozilla.org/listinfo/dev-fxacct
  • OLD: sync-dev@mozilla.org, https://mail.mozilla.org/listinfo/sync-dev
• IRC:
  • #fxa
  • #fxosqa
  • #native_identity
  • #identity
  • #sync
  • #androidsync
  • #sync-tactics

Bug Lists and Reporting

• Dev-related issues will be mixed within Bugzilla and GitHub (but see trees/meta info below)
• Product issues will all live in Bugzilla

• FxA/FFos Program Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941078&hide_resolved=0
• OLD: FFos Dev Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=920135&hide_resolved=0
• NEW: Fxa/FFos Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941723&hide_resolved=0

• FxA/Sync Relaunch Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=909322&hide_resolved=0

• Mobile (Android) Dev Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=799726&hide_resolved=0
• Desktop Dev Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=905997&hide_resolved=0
  • Note there is also the following trackers:
    • Bug 969593 to track account-based Sync bugs that we intend to address in Firefox 29.
    • Bug 969626 to track Firefox 29-relevant Sync issues that affect desktop, but that are not client issues.
• Server-side Dev Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=907475&hide_resolved=0
  • Fx Accounts Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907494
  • Sync Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907479

• UI Bugs for Aurora
  • https://bugzilla.mozilla.org/showdependencytree.cgi?id=964920&hide_resolved=1
• Enable FxA Sync on Nightly
  • https://bugzilla.mozilla.org/showdependencytree.cgi?id=964922&hide_resolved=1

• WheresMyFox Client Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=938896&hide_resolved=0

• User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
  • Classification: TBD
  • Products: TBD
  • Components: TBD
• Note: We may need to cross link Bugzilla bugs with GitHub issues

Firefox Releases

• Early stages
  • ELM builds for Android and Desktop (see below)
  • FFos unrelease builds: TBD
• Nightly
  • http://nightly.mozilla.org/
  • Similar requirements to above
• Three-channel focus with latest features, fixes, etc.
  • Aurora: Index of /pub/mozilla.org/firefox/nightly/latest-mozilla-aurora
  • Beta: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/latest-beta
  • Release: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases
  • Android-specific: http://ftp.mozilla.org/pub/mozilla.org/mobile

Platforms/OS

• Mac 10.7.x and 10.8.x
• Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
• Windows 8 (Metro mode and desktop mode) 32bit/64bit
• Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
• Android 2.x - 4.x

• The Goal is to stay consistent with requirements for FireFox and Marketplace.
• For more information, see the following issue: https://github.com/mozilla/fxa-content-server/issues/105

ELM Nightly Builds

• Desktop
  • http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-elm/
• Android
  • http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-elm-android/
  • http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-elm-android-x86/
  • http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-elm-android-armv6/

Dev Test Environments

• See the following wiki:
  • https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Development_Environments_-_partial
• Note: You will need Dev/QA AWS credentials to access these servers.

LoadTest Environment

• OLD:
  • https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Load_Test_Environment
• Note: You will need Dev/QA AWS credentials to access these servers.

• NEW:
  • https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Loads_Cluster_Environment
• Note: You will need special AWS credentials to access these servers.

Stage Environments

• See the following wiki sections:
  • FxA: https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Stage_Environment
  • TokenServer: https://wiki.mozilla.org/QA/Services/FxATestEnvironments#TokenServer_Stage_Environment
  • Sync: https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Sync_1.5_Stage_Environment

Test Sections/Suites

TS 1: Authentication and Accounts - Backend

• This is Authentication Server/fxa-auth (IdP), Account Server/fxa-content
• Main parts:
  • fxa-auth service
  • fxa-content service

• Areas of focus for testing: primarily server-side
  • Manual testing - unit/functional/API
  • Test automation - unit/functional/API - an API Test Suite, for example
  • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
  • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
  • Server load testing with Loads tool (with heka for stats collection), monitoring, logging
  • Performance testing with Loads tool (possibly with some logging, monitoring, hekad stuff hooked up)
  • Security (end to end)
  • Scaling - AWS stack (CF) - single-region/multi-region, ELB, instances, use of PHX1
  • Documentation review and support for Doc and SUMO teams
  • Env-specific deployments on localhosts and in AWS
  • Build/deploy your own - similar to Sync 1.5 and Persona

• Track/Test (as needed)
  • Internal dependencies: repos, packages, and libs to track and to test
    • restmail, node-kvstore, node-srp, jwcrypto, picl-spec-crypto
  • Loads integration
  • SimplePush integration (has this been defined for v1/milestone 1?)
  • Heka integration
  • External dependencies: Hawk, Hapi, Good, Kibana, etc.

• Notes
  • DB/storage issues and dependencies
    • What level of testing vs. monitoring is required
  • Server-specific notes and information:
    • Stats collection is going to happen with Heka
    • Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite

TS 2: UX/UI (device/platform-specfic)

• This is across product (desktop vs. mobile client vs. FFos vs. web)
• See also TS 12: Localization

• Areas of focus for testing:
  • Flow, functionality, feel per/on each client
  • Consistency, repeatability, stability between devices/platforms
  • Functional tests against all important features - Sign Up, Sign In, Password change/reset, etc.
  • Functional tests cases for all UX/UI related user stories
  • Functional tests specific to mobile clients
  • Functional tests specific to desktop clients
  • UX/UI verification against V1/Milestone 1 goals and user stories (see REFs below...)
  • QA will help or at least support the localization testing of the UI
  • Documentation review and support for Doc and SUMO teams
  • Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)

TS 3: Desktop/Web Client and Sync 1.5

• This is possibly old Sync plus new client auth
• NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.

• NOTE: Add in Master Password testing

• Areas of focus for testing:
  • Unit testing on localhost/remote host
  • API testing - TBD
  • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
  • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
  • Manual testing of desktop client - per account vs. per device syncing (TBD)
    • Functional testing of all supported features:
      • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
    • Use of Moztrap to create/maintain test cases
  • Client automation - unit/functional with Selenium/Mozmill
    • To be coordinated with the Fx desktop/client QA team
  • Firefox integration (Try/Nightly from ELM)
  • Firefox integration (Aurora, Beta, Release)
  • Security and privacy
  • Documentation review and support for Doc and SUMO teams
  • Desktop-specific verification against V1/Milestone 1 goals
    • https://wiki.mozilla.org/User_Services/Sync/v1#Goals
  • Functional tests cases for all desktop-client-related user stories
    • https://wiki.mozilla.org/User_Services/Sync/v1#User_stories
  • Interaction with about:config for logging, debugging
  • Interaction with add-ons, extensions, themes, personas, etc.
  • Desktop integration - all supported platforms and OS
    • Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
      • Match to Fx support (including Fx ESR)
  • FireFox command line for testing, setting up sync, authing, profiles, who knows?

TS 4: Mobile Client (Android) and Sync 1.5

• This is possibly old Sync plus new client auth
• NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.

• NOTE: Add in Master Password testing

• Areas of focus for testing:
  • Unit testing on localhost/remote host
  • API testing - TBD
  • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
  • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
  • Manual testing of mobile client - per account vs. per device syncing (TBD)
    • Functional testing of all supported features:
      • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
    • Use of Moztrap to create/maintain test cases
  • Client automation - unit/functional - do we have anything here besides phone/os emulation?
    • To be coordinated with the Fx Mobile QA team
  • Firefox integration (Try/Nightly from ELM)
  • Firefox integration (Aurora, Beta, Release)
  • Security and privacy
  • Documentation review and support for Doc and SUMO teams
  • Mobile-specific verification against V1/Milestone 1 goals
    • https://wiki.mozilla.org/User_Services/Sync/v1#Goals
  • Functional tests cases for all mobile-client-related user stories
    • https://wiki.mozilla.org/User_Services/Sync/v1#User_stories
  • Interaction with about:config for logging, debugging
  • Interaction with add-ons, extensions, themes, personas, etc.
  • Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
    • Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
      • Match to Fx support

• Tools
  • Use of Android specific adk and debug tools/methods
• See Section 5 for Sync 1.5 and new Sync integration/interaction
• See Section 6 for Sync 1.5 migration to new Sync

TS 5: Firefox OS, FTU/FTE, and Sync 1.5

• TBD - requirements here, especially for Q4/2013 are still being worked out.
• TBD - this could cover the basics of FTU/FTE, Marketplace, and WhereIsMyFox
• NOTE: this is (or could be) Sign in to FFos as well as Sign in to Marketplace (as a separate path).
• NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.

TS 6: Server/Backend Data Storage

• TBD, but for v1/m1, this will be Sync 1.5, currently living in PHX1
  • Note this is the Sync Server portion of the product.
  • The Auth data store is covered in Section 1: Authentication.

TS 7: Sync 1.5 and "new" Sync Interaction [lead: kthiessen]

• There will be some amount of time where
  • Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
  • A "migration" will start to happen where users voluntarily replace old sync with new sync
  • A "forced migration" or "shutdown period" where users will have to move from old sync to new sync
  • Only the first item will be tested in this section

• Areas of focus for testing:
  • Functional test independent use of both old sync and new sync, per client/platform/device
  • Functional test interactions between the two (combined use)(especially if per device syncing is supported), per client/platform/device
  • Test/verify data storage for both sync models per account/device
  • UX/UI interactions and methods, especially for Sign Up and Sign In on both Sync models, and other supported preferences and UI features
  • Old Sync/new Sync verification against V1/Milestone 1 goals
    • https://wiki.mozilla.org/User_Services/Sync/v1#Goals
  • General account interaction: old sync, new Sync, Persona
  • Documentation review pertaining to old Sync, new Sync

TS 8: Migration to "new" Sync [lead: kthiessen]

• At some point, all old Sync users will either have to
  • Migrate to new Sync
  • Stop using Sync

• Areas of focus for testing:
  • QA will need to prepare a set of steps, before and after tests, to verify migration
  • QA will need to interrogate/investigate account contents from old and new sync to verify migration
  • All of this QA work can be done in Stage first to verify migration will actually run as expected.
  • Dev and OPs suppport will be needed for the test and verification steps.
  • Documentation review pertaining to old Sync, new Sync and migration steps/strategies

TS 9: FxA Documentation and SUMO

• Why? Because current documentation (all of it really) is a pain point
• Note: For QA, this is mostly a tracking responsibility
• Updates to the old/current Sync information across locations
  • internal, app, wiki, SUMO, MDN, etc
• Creation of new Sync information across locations
  • internal, app, wiki, SUMO, MDN, etc
• Dev and QA can help define, write, and review documentation
  • UX/UI Demos and training
  • Usage demos and training (through SUMO?)
  • Documentation team support is needed
  • SUMO team support is needed
  • REFs: TBD

TS 10: FxA Localization

• Note: For QA, this is mostly a tracking responsibility
• across UX/UI/clients
• See also TS 4: UX/UI

• Mobile and Desktop clients - full UX/UI experience, wording, etc.
• Dev and QA can help with string verfication before and after localization
• Will need support from the FireFox Localization team and community
• Bug reporting will happen through Bugzilla
• REFs: TBD

TS 11: Marketplace Support

• Test FxA account activity:
  • On FFos
  • From a Firefox browser (desktop/android)
  • From a non-Fx browser using the web interface
• See the Marketplace Test Plan for more information: TBD

TS 12: Where Is My Fox/Find My Device Support

• Test FxA account activity:
  • On FFos
  • From a Firefox browser (desktop/android)
  • From a non-Fx browser using the web interface
• See the Find My Device Test Plan for more information: TBD

TS 13: Metro (Win8/tablet) and Sync support

• FxA/Sync/Metro integration - TBD

Main References

Wikis and Etherpads

• Main FxA Dev wiki: https://wiki.mozilla.org/Identity/FirefoxAccounts
• Roadmap: https://wiki.mozilla.org/Identity/Roadmap
• Architecture: https://wiki.mozilla.org/Identity/Firefox-Accounts#Architecture
• Main FxA PM wiki: https://wiki.mozilla.org/User_Services/Sync
• Main FxA UX/UI wiki: https://wiki.mozilla.org/Identity/UX
• FxA-Auth-Server (API): https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md

• Older sites for FxA v1:
  • Main: https://wiki.mozilla.org/User_Services/Sync/v1
  • Goals site: https://wiki.mozilla.org/User_Services/Sync/v1#Goals
  • User Stories site: https://wiki.mozilla.org/User_Services/Sync/v1#User_stories
  • UX/UI: https://wiki.mozilla.org/User_Services/Sync/v1#UX_design
  • UX/UI: https://wiki.mozilla.org/User_Services/Sync#UX_Designs
  • UX/UI: https://wiki.mozilla.org/User_Services/Sync#User_Research
  • Sync Migration: https://wiki.mozilla.org/User_Services/Sync/v1#Migration_strategy
  • Technical Materials and Links: https://wiki.mozilla.org/User_Services/Sync#Technical_Materials

• FxA Product: https://id.etherpad.mozilla.org/FxA-Product
• FxA Product Coordination: https://id.etherpad.mozilla.org/picl-product-coordination
• FxA Engineering Coordination: https://id.etherpad.mozilla.org/picl-engineering-coordination

GitHub Repositories

• https://github.com/mozilla/fxa-auth-server
• https://github.com/mozilla/fxa-scrypt-helper
• https://github.com/mozilla/fxa-content-server
• https://github.com/mozilla/fxa-js-client
• https://github.com/mozilla/fxa-js-client-old

• https://github.com/warner/picl-spec-crypto
• https://github.com/mozilla/node-kvstore
• https://github.com/mozilla/node-srp/
• https://github.com/mozilla/jwcrypto
• https://github.com/mozilla/restmail.net
• https://github.com/mozilla/picl-mocks

• FxA for the Web (TBD)
  • https://id.etherpad.mozilla.org/gherkin-web
  • https://wiki.mozilla.org/Identity/Firefox-Accounts#Architecture

• Dev and Loadtest Deployments
  • https://github.com/mozilla/fxa-deployment

• Loadtest (for fxa-auth-server)
  • https://github.com/mozilla-services/loads
  • https://github.com/mozilla-services/loads.js

• FFos specific
  • TBD

• Server-side:
  • TBD

Travis CI

• https://travis-ci.org/mozilla/fxa-auth-server
• https://travis-ci.org/mozilla/fxa-content-server
• https://travis-ci.org/mozilla/fxa-scrypt-helper
• https://travis-ci.org/mozilla/fxa-js-client
• https://travis-ci.org/mozilla/fxa-js-client-old