Latest revision as of 20:05, 26 August 2016

This is the Test Plan for FxA, Tokenserver, and Sync 1.5

Intro/Summary/Notes

Intro and Summary

The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.5.
A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc.
It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.

Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases for each of the main areas of testing: Server-side, Desktop/Web, Android, FFos.

Notes

FxA is scheduled to go out with Firefox 29. This will replace old sync account functionality and use.
Sync users from Firefox 28 and earlier will need to create new FxA accounts when they upgrade to Fx29.
Once that happens, devices hooked to sync for Fx28 and older will have to be unlinked and then added to the new FxA account in order to sync.
If a user wants to hold all devices at old sync, he/she must not upgrade to Fx29.
A user can not have accounts with devices across versions of Fx older than 29.

User Stories

Start here for Android and desktop/web:
- https://wiki.mozilla.org/User_Services/Sync/Relaunch#Product_Requirements_.26_User_Stories
For FFos, start here:
- https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AtVT90hlMtdSdEd4TVVjWXNfU3ctMlVhWFRrWkpweVE#gid=50
- https://mana.mozilla.org/wiki/display/Identity/Firefox+Accounts+%28FxA%29+for+FxOS

NOTE: For all platforms, see also the bug trees below...

Building Test Cases

QA needs at least one resource, probably from SoftVision, to convert the current Android, desktop/web, and FFos user stories into client-side test cases.
Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
Scheduling: TBD

Schedule

QA Team

Services QA
- Edwin Wong - Manager
- James Bonacci
- John Morrison
- Karl Thiessen
- Peter deHaan
- Richard Pappalardo

QA
- Various members of the Firefox OS, Desktop, and Mobile teams will be involved in the testing.

Note: FxA and Sync Teams information can be found here:
- https://wiki.mozilla.org/User_Services/Sync/Relaunch/DRIs
- https://wiki.mozilla.org/Identity/FirefoxAccounts#Team

OPs Information

OPs has a significant number of really detailed Mana pages for deploying, testing, maintaining, and monitoring our Stage and Production environments.
Talk to a member of the OPs team for more information and for the Mana links to get you started.

Email Lists and IRC

Services QA: services-qa@mozilla.com
QA Teams: qa@mozilla.com
Sync and FxA:
- NEW: dev-fxacct@mozila.org, https://mail.mozilla.org/listinfo/dev-fxacct
- OLD: sync-dev@mozilla.org, https://mail.mozilla.org/listinfo/sync-dev
IRC:
- #fxa
- #fxosqa
- #native_identity
- #identity
- #sync
- #androidsync
- #sync-tactics

Bug Lists and Reporting

Dev-related issues will be mixed within Bugzilla and GitHub (but see trees/meta info below)
Product issues will all live in Bugzilla

All FxA/Sync https://bugzilla.mozilla.org/showdependencytree.cgi?id=905997&hide_resolved=0

FxA/FFos Program: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941078&hide_resolved=0
OLD: FFos Dev https://bugzilla.mozilla.org/showdependencytree.cgi?id=920135&hide_resolved=0
NEW: Fxa/FFos: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941723&hide_resolved=0

FxA/Sync Relaunch: https://bugzilla.mozilla.org/showdependencytree.cgi?id=909322&hide_resolved=0

Mobile (Android): https://bugzilla.mozilla.org/showdependencytree.cgi?id=799726&hide_resolved=0
Desktop Dev: https://bugzilla.mozilla.org/showdependencytree.cgi?id=969593&hide_resolved=0
- Note there is also the following trackers:
  - Bug 969626 to track Firefox 29-relevant Sync issues that affect desktop, but that are not client issues.
Server-side Dev: https://bugzilla.mozilla.org/showdependencytree.cgi?id=907475&hide_resolved=0
- Fx Accounts Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907494
- Sync Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907479

UI Bugs for Aurora
- https://bugzilla.mozilla.org/showdependencytree.cgi?id=964920&hide_resolved=1
Enable FxA Sync on Nightly
- https://bugzilla.mozilla.org/showdependencytree.cgi?id=964922&hide_resolved=1

WheresMyFox Client: https://bugzilla.mozilla.org/showdependencytree.cgi?id=938896&hide_resolved=0

Contacts on FFos: https://bugzilla.mozilla.org/showdependencytree.cgi?id=976837&hide_resolved=1

User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
- Classification: TBD
- Products: TBD
- Components: TBD
Note: We may need to cross link Bugzilla bugs with GitHub issues

Firefox Releases

Early stages
- ELM builds for Android and Desktop (see below)
- FFos unrelease builds: TBD
Nightly
- http://nightly.mozilla.org/
- Similar requirements to above
Three-channel focus with latest features, fixes, etc.
- Aurora: Index of /pub/mozilla.org/firefox/nightly/latest-mozilla-aurora
- Beta: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/latest-beta
- Release: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases
- Android-specific: http://ftp.mozilla.org/pub/mozilla.org/mobile

Platforms/OS

Mac 10.7.x and 10.8.x
Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
Windows 8 (Metro mode and desktop mode) 32bit/64bit
Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
Android 2.x - 4.x

The Goal is to stay consistent with requirements for FireFox and Marketplace.
For more information, see the following issue: https://github.com/mozilla/fxa-content-server/issues/105

ELM Nightly Builds

Dev Test Environments

See the following wiki:
- https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Development_Environments_-_partial
Note: You will need Dev/QA AWS credentials to access these servers.

LoadTest Environment

OLD:
- https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Load_Test_Environment
Note: You will need Dev/QA AWS credentials to access these servers.

NEW:
- https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Loads_Cluster_Environment
Note: You will need special AWS credentials to access these servers.

Stage Environments

See the following wiki sections:

Test Sections/Suites

TS 1: Authentication and Accounts - Core

This is Authentication Server/fxa-auth (IdP), Account Server/fxa-content
Main parts:
- fxa-auth service
- fxa-content service

Areas of focus for testing: primarily server-side
- Manual testing - unit/functional/API
- Test automation - unit/functional/API - an API Test Suite, for example
- Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
- Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
- Server load testing with Loads tool (with heka for stats collection), monitoring, logging
- Performance testing with Loads tool (possibly with some logging, monitoring, hekad stuff hooked up)
- Security (end to end)
- Scaling - AWS stack (CF) - single-region/multi-region, ELB, instances, use of PHX1
- Documentation review and support for Doc and SUMO teams
- Env-specific deployments on localhosts and in AWS
- Build/deploy your own - similar to Sync 1.5 and Persona

Track/Test (as needed)
- Internal dependencies: repos, packages, and libs to track and to test
  - restmail, node-kvstore, node-srp, jwcrypto, picl-spec-crypto
- Loads integration
- SimplePush integration (has this been defined for v1/milestone 1?)
- Heka integration
- External dependencies: Hawk, Hapi, Good, Kibana, etc.

Notes
- DB/storage issues and dependencies
  - What level of testing vs. monitoring is required
- Server-specific notes and information:
  - Stats collection is going to happen with Heka
  - Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite

TS 2: UX/UI

This is across product (desktop vs. mobile client vs. FFos vs. web)
See also TS 10: Localization

Areas of focus for testing:
- Flow, functionality, feel per/on each client
- Consistency, repeatability, stability between devices/platforms
- Functional tests against all important features - Sign Up, Sign In, Password change/reset, etc.
- Functional tests cases for all UX/UI related user stories
- Functional tests specific to mobile clients
- Functional tests specific to desktop clients
- UX/UI verification against V1/Milestone 1 goals and user stories (see REFs below...)
- QA will help or at least support the localization testing of the UI
- Documentation review and support for Doc and SUMO teams
- Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)

TS 3: Desktop/Web Client and Sync 1.5

This is possibly old Sync plus new client auth
NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.

NOTE: Add in Master Password testing

Areas of focus for testing:
- Unit testing on localhost/remote host
- API testing - TBD
- Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
- Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
- Manual testing of desktop client - per account vs. per device syncing (TBD)
  - Functional testing of all supported features:
    - account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
  - Use of Moztrap to create/maintain test cases
- Client automation - unit/functional with Selenium/Mozmill
  - To be coordinated with the Fx desktop/client QA team
- Firefox integration (Try/Nightly from ELM)
- Firefox integration (Aurora, Beta, Release)
- Security and privacy
- Documentation review and support for Doc and SUMO teams
- Desktop-specific verification against V1/Milestone 1 goals
  - https://wiki.mozilla.org/User_Services/Sync/v1#Goals
- Functional tests cases for all desktop-client-related user stories
  - https://wiki.mozilla.org/User_Services/Sync/v1#User_stories
- Interaction with about:config for logging, debugging
- Interaction with add-ons, extensions, themes, personas, etc.
- Desktop integration - all supported platforms and OS
  - Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
    - Match to Fx support (including Fx ESR)
- FireFox command line for testing, setting up sync, authing, profiles, who knows?

TS 4: Mobile Client (Android) and Sync 1.5

This is possibly old Sync plus new client auth
NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
NOTE: Add in Master Password testing

Areas of focus for testing:
- Unit testing on localhost/remote host
- API testing - TBD
- Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
- Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
- Manual testing of mobile client - per account vs. per device syncing (TBD)
  - Functional testing of all supported features:
    - account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
  - Use of Moztrap to create/maintain test cases
- Client automation - unit/functional - do we have anything here besides phone/os emulation?
  - To be coordinated with the Fx Mobile QA team
- Firefox integration (Try/Nightly from ELM)
- Firefox integration (Aurora, Beta, Release)
- Security and privacy
- Documentation review and support for Doc and SUMO teams
- Mobile-specific verification against V1/Milestone 1 goals
  - https://wiki.mozilla.org/User_Services/Sync/v1#Goals
- Functional tests cases for all mobile-client-related user stories
  - https://wiki.mozilla.org/User_Services/Sync/v1#User_stories
- Interaction with about:config for logging, debugging
- Interaction with add-ons, extensions, themes, personas, etc.
- Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
  - Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
    - Match to Fx support

Tools
- Use of Android specific adk and debug tools/methods
See Section 5 for Sync 1.5 and new Sync integration/interaction
See Section 6 for Sync 1.5 migration to new Sync

TS 5: Firefox OS and Sync 1.5

TBD - requirements here, especially for Q2 2014 (FFos 1.4) are still being worked out.
TBD - this could cover the basics of FTU/FTE, Marketplace, WhereIsMyFox, and Contacts
NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.

TS 6: Server/Backend Data Storage

TBD, but for v1/m1, this will be Sync 1.5, currently living in PHX1
- Note this is the Sync Server portion of the product.
- The Auth data store is covered in Section 1: Authentication.

TS 7: Sync 1.5 and "new" Sync Interaction [lead: kthiessen]

There will be some amount of time where
- Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
- A "migration" will start to happen where users voluntarily replace old sync with new sync
- A "forced migration" or "shutdown period" where users will have to move from old sync to new sync
- Only the first item will be tested in this section

Areas of focus for testing:
- Functional test independent use of both old sync and new sync, per client/platform/device
- Functional test interactions between the two (combined use)(especially if per device syncing is supported), per client/platform/device
- Test/verify data storage for both sync models per account/device
- UX/UI interactions and methods, especially for Sign Up and Sign In on both Sync models, and other supported preferences and UI features
- Old Sync/new Sync verification against V1/Milestone 1 goals
  - https://wiki.mozilla.org/User_Services/Sync/v1#Goals
- General account interaction: old sync, new Sync, Persona
- Documentation review pertaining to old Sync, new Sync

TS 8: Migration to "new" Sync [lead: kthiessen]

At some point, all old Sync users will either have to
- Migrate to new Sync
- Stop using Sync

Areas of focus for testing:
- QA will need to prepare a set of steps, before and after tests, to verify migration
- QA will need to interrogate/investigate account contents from old and new sync to verify migration
- All of this QA work can be done in Stage first to verify migration will actually run as expected.
- Dev and OPs suppport will be needed for the test and verification steps.
- Documentation review pertaining to old Sync, new Sync and migration steps/strategies

TS 9: FxA Documentation and SUMO

Why? Because current documentation (all of it really) is a pain point
Note: For QA, this is mostly a tracking responsibility
Updates to the old/current Sync information across locations
- internal, app, wiki, SUMO, MDN, etc
Creation of new Sync information across locations
- internal, app, wiki, SUMO, MDN, etc
Dev and QA can help define, write, and review documentation
- UX/UI Demos and training
- Usage demos and training (through SUMO?)
- Documentation team support is needed
- SUMO team support is needed
- REFs: TBD

TS 10: FxA Localization

Note: For QA, this is mostly a tracking responsibility
across UX/UI/clients
See also TS 4: UX/UI

Mobile and Desktop clients - full UX/UI experience, wording, etc.
Dev and QA can help with string verfication before and after localization
Will need support from the FireFox Localization team and community
Bug reporting will happen through Bugzilla
REFs: TBD

TS 11: Marketplace Support

Test FxA account activity:
- On FFos
- From a Firefox browser (desktop/android)
- From a non-Fx browser using the web interface
See the Marketplace Test Plan for more information: TBD

TS 12: Where Is My Fox/Find My Device Support

Test FxA account activity:
- On FFos
- From a Firefox browser (desktop/android)
- From a non-Fx browser using the web interface
See the Find My Device Test Plan for more information: TBD

TS 13: Metro (Win8/tablet) and Sync support

FxA/Sync/Metro integration - TBD

Main References

Wikis and Etherpads

Main FxA wiki: https://wiki.mozilla.org/Identity/FirefoxAccounts
Main FxA wiki: https://wiki.mozilla.org/User_Services/Sync/Relaunch
Roadmap: https://wiki.mozilla.org/Identity/Roadmap
Architecture: https://wiki.mozilla.org/Identity/Firefox-Accounts#Architecture
Main FxA PM wiki: https://wiki.mozilla.org/User_Services/Sync
Main FxA UX/UI wiki: https://wiki.mozilla.org/Identity/UX
Transition: https://wiki.mozilla.org/User_Services/Sync/Transition_To_FxA_Sync
FxA-Auth-Server (API): https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md

Meetings:
FxA Product: https://id.etherpad.mozilla.org/FxA-Product
FxA Engineering Coordination: https://id.etherpad.mozilla.org/fxa-engineering-coordination
FxA Web Coordination: https://id.etherpad.mozilla.org/fxa-web-coordination
Sync Engineering Stand-Up: https://id.etherpad.mozilla.org/sync-eng-standup
FxA Branding: https://id.etherpad.mozilla.org/fxa-branding
FxA Metrics/Monitoring: https://id.etherpad.mozilla.org/fxa-metrics-monitoring
FxA Engagement: https://id.etherpad.mozilla.org/fxa-engagement

GitHub Repositories

Dev and Loadtest Deployments
- https://github.com/mozilla/fxa-deployment

Loadtest (for fxa-auth-server)
- https://github.com/mozilla-services/loads

Android
- https://github.com/mozilla-services/android-sync

Tokenserver and Sync
- https://github.com/mozilla-services/tokenserver
- https://github.com/mozilla-services/server-syncstorage

OPs
- https://github.com/mozilla-services/puppet-config
- https://github.com/mozilla-services/svcops-oompaloompas

TestEngineering/Services/SyncTestPlanV1: Difference between revisions

Latest revision as of 20:05, 26 August 2016

Contents

Intro/Summary/Notes

Intro and Summary

Notes

User Stories

Building Test Cases

Schedule

QA Team

OPs Information

Email Lists and IRC

Bug Lists and Reporting

Firefox Releases

Platforms/OS

ELM Nightly Builds

Dev Test Environments

LoadTest Environment

Stage Environments

Test Sections/Suites

TS 1: Authentication and Accounts - Core

TS 2: UX/UI

TS 3: Desktop/Web Client and Sync 1.5

TS 4: Mobile Client (Android) and Sync 1.5

TS 5: Firefox OS and Sync 1.5

TS 6: Server/Backend Data Storage

TS 7: Sync 1.5 and "new" Sync Interaction [lead: kthiessen]

TS 8: Migration to "new" Sync [lead: kthiessen]

TS 9: FxA Documentation and SUMO

TS 10: FxA Localization

TS 11: Marketplace Support

TS 12: Where Is My Fox/Find My Device Support

TS 13: Metro (Win8/tablet) and Sync support

Main References

Wikis and Etherpads

GitHub Repositories

Travis CI

Navigation menu

TestEngineering/Services/SyncTestPlanV1: Difference between revisions

Latest revision as of 20:05, 26 August 2016

Intro/Summary/Notes

Intro and Summary

Notes

User Stories

Building Test Cases

Schedule

QA Team

OPs Information

Email Lists and IRC

Bug Lists and Reporting

Firefox Releases

Platforms/OS

ELM Nightly Builds

Dev Test Environments

LoadTest Environment

Stage Environments

Test Sections/Suites

TS 1: Authentication and Accounts - Core

TS 2: UX/UI

TS 3: Desktop/Web Client and Sync 1.5

TS 4: Mobile Client (Android) and Sync 1.5

TS 5: Firefox OS and Sync 1.5

TS 6: Server/Backend Data Storage

TS 7: Sync 1.5 and "new" Sync Interaction [lead: kthiessen]

TS 8: Migration to "new" Sync [lead: kthiessen]

TS 9: FxA Documentation and SUMO

TS 10: FxA Localization

TS 11: Marketplace Support

TS 12: Where Is My Fox/Find My Device Support

TS 13: Metro (Win8/tablet) and Sync support

Main References

Wikis and Etherpads

GitHub Repositories

Travis CI

Navigation menu

Search