TestEngineering/Services/SyncTestPlanV1: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(39 intermediate revisions by one other user not shown)
Line 1: Line 1:
This is the Sync/Picl Test Plan, V1
This is the Test Plan for FxA, Tokenserver, and Sync 1.5


= Intro/Summary/Notes =
= Intro/Summary/Notes =
== Intro ==
== Intro and Summary ==
* The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.1.
* The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.5.
* A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc.
* It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.
* It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.


== Summary ==
* Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases for each of the main areas of testing: Server-side, Desktop/Web, Android, FFos.
* A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc...


== Notes ==
== Notes ==
Line 19: Line 19:
* Start here for Android and desktop/web:  
* Start here for Android and desktop/web:  
** https://wiki.mozilla.org/User_Services/Sync/Relaunch#Product_Requirements_.26_User_Stories
** https://wiki.mozilla.org/User_Services/Sync/Relaunch#Product_Requirements_.26_User_Stories
* For FFos: TBD
* For FFos, start here:
** https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AtVT90hlMtdSdEd4TVVjWXNfU3ctMlVhWFRrWkpweVE#gid=50
** https://mana.mozilla.org/wiki/display/Identity/Firefox+Accounts+%28FxA%29+for+FxOS
 
* NOTE: For all platforms, see also the bug trees below...


== Building Test Cases ==
== Building Test Cases ==
* QA needs at least one resource, probably from SoftVision, to convert the current v1/m1 user stories into client-side test cases.
* QA needs at least one resource, probably from SoftVision, to convert the current Android, desktop/web, and FFos user stories into client-side test cases.
* Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
* Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
* Scheduling: TBD
* Scheduling: TBD
== Testing the Major Engineering Milestones ==
How to keep focused while staying sane...
* Pre-release testing
** Unit testing of fxa-auth-server
*** localhost
*** Dev hosts
** Unit testing of fxa-content-server
** Unit testing for fxa-scrypt-helper
** Unit testing of fxa-js-client and fxa-js-client-old
** Review of mocks/UI flows: https://wiki.mozilla.org/Identity/UX
* First try/nightly build on ELM
** See the client-side bugs for each service
** User stories - are they covered?
** Engineering goals - are they covered?
** Functional checks - does this thing work?
** Write test cases
** Write client automation for desktop/android/FFos
* Successive builds on ELM
** Same list as above
* Official Nightly builds
** Full testing, including Documentation, Localization, and SUMO/support information
== Testing the MVP ==
* TBD - this really depends on which platforms are supported for MVP. Initially, this may be just FxA on FFos version 1.4.


= Schedule =
= Schedule =
Line 64: Line 42:
** Karl Thiessen
** Karl Thiessen
** Peter deHaan
** Peter deHaan
** Richard Pappalardo


* QA
* QA
Line 69: Line 48:


* Note: FxA and Sync Teams information can be found here:
* Note: FxA and Sync Teams information can be found here:
** https://wiki.mozilla.org/User_Services/Sync/Relaunch/DRIs
** https://wiki.mozilla.org/Identity/FirefoxAccounts#Team
** https://wiki.mozilla.org/Identity/FirefoxAccounts#Team
* QA head count (DRAFT)
** 1 FT automation/tools lead + QA team support
** 1 FT android client lead + QA team support
** 1 FT desktop client lead + QA team support
** 1 FT web lead + QA team support
** 1 FT FFos lead + QA team support
** 1 FT Auth/Storage lead + QA team support
** 1 FT server/security/db lead + QA team support - TBD
** A subset of the FF and Services QA teams for testing when this goes to FF Nightly.
** The entire FF and Services QA teams as part of normal testing of FF Aurora, Beta, Release channels
** Challenges
*** Recruiting QA and SoftVision (or other contractors) for testing Try builds
*** Recruiting QA team for full testing during normal FF releases
*** Recruiting community for testing Firefox channels when they become available (esp. if they have experience with Sync 1.x)


= OPs Information =
= OPs Information =
Line 103: Line 68:
** #sync
** #sync
** #androidsync
** #androidsync
** #sync-tactics


= Bug Lists and Reporting =
= Bug Lists and Reporting =
Line 108: Line 74:
* Product issues will all live in Bugzilla
* Product issues will all live in Bugzilla


* FxA/FFos Program Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941078&hide_resolved=0
* All FxA/Sync https://bugzilla.mozilla.org/showdependencytree.cgi?id=905997&hide_resolved=0
* OLD: FFos Dev Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=920135&hide_resolved=0
* NEW: Fxa/FFos Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941723&hide_resolved=0


* FxA/Sync Relaunch Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=909322&hide_resolved=0
* FxA/FFos Program: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941078&hide_resolved=0
* OLD: FFos Dev https://bugzilla.mozilla.org/showdependencytree.cgi?id=920135&hide_resolved=0
* NEW: Fxa/FFos: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941723&hide_resolved=0


* Mobile (Android) Dev Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=799726&hide_resolved=0
* FxA/Sync Relaunch: https://bugzilla.mozilla.org/showdependencytree.cgi?id=909322&hide_resolved=0
* Desktop Dev Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=905997&hide_resolved=0
 
* Mobile (Android): https://bugzilla.mozilla.org/showdependencytree.cgi?id=799726&hide_resolved=0
* Desktop Dev: https://bugzilla.mozilla.org/showdependencytree.cgi?id=969593&hide_resolved=0
** Note there is also the following trackers:
** Note there is also the following trackers:
*** Bug 969593 to track account-based Sync bugs that we intend to address in Firefox 29.
*** Bug 969626 to track Firefox 29-relevant Sync issues that affect desktop, but that are not client issues.
*** Bug 969626 to track Firefox 29-relevant Sync issues that affect desktop, but that are not client issues.
* Server-side Dev Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=907475&hide_resolved=0
* Server-side Dev: https://bugzilla.mozilla.org/showdependencytree.cgi?id=907475&hide_resolved=0
** FF Accounts Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907494
** Fx Accounts Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907494
** Sync Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907479
** Sync Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907479


Line 128: Line 95:
** https://bugzilla.mozilla.org/showdependencytree.cgi?id=964922&hide_resolved=1
** https://bugzilla.mozilla.org/showdependencytree.cgi?id=964922&hide_resolved=1


* WheresMyFox Client Tree/Meta: https://bugzilla.mozilla.org/showdependencytree.cgi?id=938896&hide_resolved=0
* WheresMyFox Client: https://bugzilla.mozilla.org/showdependencytree.cgi?id=938896&hide_resolved=0
 
* Contacts on FFos: https://bugzilla.mozilla.org/showdependencytree.cgi?id=976837&hide_resolved=1


* User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
* User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
Line 135: Line 104:
** Components: TBD
** Components: TBD
* Note: We may need to cross link Bugzilla bugs with GitHub issues
* Note: We may need to cross link Bugzilla bugs with GitHub issues
= Bug/Issue Testing and Verification =
* TBD
* Need to add something here to talk about how to track and test fixes/patches/builds in Bugzilla as well as GitHub.
* Also, what to do with regressions in GitHub vs. Bugzilla...


= Firefox Releases =
= Firefox Releases =
Line 157: Line 121:
* Mac 10.7.x and 10.8.x
* Mac 10.7.x and 10.8.x
* Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
* Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
* Windows 8 (Metro mode and desktop mode). Does this come in a 32bit flavor?
* Windows 8 (Metro mode and desktop mode) 32bit/64bit
* Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
* Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
* Android 2.x - 4.x
* Android 2.x - 4.x
Line 173: Line 137:


= Dev Test Environments =
= Dev Test Environments =
* Dev Env #1: non-auto-updating environment
* See the following wiki:
** https://accounts.dev.lcip.org
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Development_Environments_-_partial
** https://api-accounts.dev.lcip.org    (formerly "idp.dev.lcip.org")
** https://scrypt-accounts.dev.lcip.org  (formerly "scrypt.dev.lcip.org")
 
* Dev Env #2: auto-updated environment
** https://accounts-latest.dev.lcip.org
** https://api-accounts-latest.dev.lcip.org
** https://scrypt-accounts-latest.dev.lcip.org
 
*https://restmail.lcip.org
 
* With these:
** https://accounts.dev.lcip.org/flow
** https://accounts.dev.lcip.org/mobile
** https://accounts-latest.dev.lcip.org/flow
** https://accounts-latest.dev.lcip.org/mobile
 
* Note: You will need Dev/QA AWS credentials to access these servers.
* Note: You will need Dev/QA AWS credentials to access these servers.


= LoadTest Environment =
= LoadTest Environment =
* Loads stack:
* OLD:
** LoadsBroker: https://broker.loads.lcip.org (replaces loads.loadtest.lcip.org)
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Load_Test_Environment
** Loads Agents: n/a
 
* Loadtest stack:
** Scrypt-Helper: https://scrypt-accounts.loadtest.lcip.org (replaces scrypt.loadtest.lcip.org)
** IdP: https://api-accounts.loadtest.lcip.org (replaces idp.loadtest.lcip.org)
** Logger: https://logs.loadtest.lcip.org
** Redis Server: https://redis.loadtest.lcip.org
 
* Note: You will need Dev/QA AWS credentials to access these servers.
* Note: You will need Dev/QA AWS credentials to access these servers.


* Kabana Dashboard:
* NEW:
** http://logs.loadtest.lcip.org/kibana/#/dashboard/file/weblogs.json
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Loads_Cluster_Environment
* Note: You will need special AWS credentials to access these servers.


* GitHub Repos
= Stage Environments =
** https://github.com/mozilla-services/loads
* See the following wiki sections:
*** Docs: http://loads.readthedocs.org/en/latest/
** FxA: https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Stage_Environment
** https://github.com/mozilla-services/loads.js
** TokenServer: https://wiki.mozilla.org/QA/Services/FxATestEnvironments#TokenServer_Stage_Environment
*** Docs: TBD
** Sync: https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Sync_1.5_Stage_Environment


= Test Sections/Suites =
= Test Sections/Suites =
== TS 1: Authentication and Accounts - Backend ==
== TS 1: Authentication and Accounts - Core ==
* This is Authentication Server/fxa-auth (IdP), but not device-specific
* This is Authentication Server/fxa-auth (IdP), Account Server/fxa-content
* Main parts:
* Main parts:
** fxa-auth (IdP) authentication service
** fxa-auth service
** MySQL data storage
** fxa-content service
** Heka data/log aggregation


* Areas of focus for testing: primarily server-side
* Areas of focus for testing: primarily server-side
Line 234: Line 174:
** Documentation review and support for Doc and SUMO teams
** Documentation review and support for Doc and SUMO teams
** Env-specific deployments on localhosts and in AWS
** Env-specific deployments on localhosts and in AWS
** Build/deploy your own - similar to Sync 1.1 and Persona
** Build/deploy your own - similar to Sync 1.5 and Persona
 
* Track/Test (as needed)  
* Track/Test (as needed)  
** Internal dependencies: repos, packages, and libs to track and to test
** Internal dependencies: repos, packages, and libs to track and to test
Line 242: Line 183:
** Heka integration
** Heka integration
** External dependencies: Hawk, Hapi, Good, Kibana, etc.
** External dependencies: Hawk, Hapi, Good, Kibana, etc.
* Bug reporting will happen through Bugzilla
 
* Client manual testing - see client sections below
* Client automation - see client sections below
* Notes
* Notes
** DB/storage issues and dependencies
** DB/storage issues and dependencies
Line 252: Line 191:
*** Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite
*** Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite


* REFs:
== TS 2: UX/UI ==
** Auth/KeyServer: https://github.com/mozilla/fxa-auth-server
* This is across product (desktop vs. mobile client vs. FFos vs. web)
** API doc: https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md
* See also TS 10: Localization
** API design: https://wiki.mozilla.org/Identity/AttachedServices/KeyServerProtocol
** Scrypt Helper service: https://github.com/mozilla/fxa-scrypt-helper
** Key Stretching: https://wiki.mozilla.org/Identity/AttachedServices/KeyServerProtocol#Client-Side_Key_Stretching
** Deployment:
*** https://github.com/mozilla/fxa-deployment
*** https://wiki.mozilla.org/Identity/AttachedServices/DeploymentPlanning
** V1/M1 goals (if needed): https://wiki.mozilla.org/User_Services/Sync/v1#Goals


== TS 2: Authentication and Accounts - Frontend/Clients ==
* This is Content Server and gherkin, not device- or platform-specific
* Main parts:
** Content Server (fxa-content-server)
** Scrypt-Helper (fxa-scrypt-helper)
** various gherkin support pieces (gherkin, fxa-js-client, fxa-js-client-old)
** Desktop client - gherkin/about:accounts
** Android client
** Web support - gherkin/about:accounts
** FFos
* Full content: TBD - see TS1
* REFs:
** FFAccounts: https://github.com/mozilla/fxa-content-server
** Client (gherkin):
*** https://github.com/mozilla/fxa-js-client
*** https://github.com/mozilla/fxa-js-client-old
** Deployment:
*** https://github.com/mozilla/fxa-deployment
*** https://wiki.mozilla.org/Identity/AttachedServices/DeploymentPlanning
** V1/M1 goals (if needed): https://wiki.mozilla.org/User_Services/Sync/v1#Goals
== TS 4: UX/UI (device/platform-specfic) ==
* This is across product (desktop vs. mobile client vs. FFos vs. web)
* See also TS 12: Localization
* Areas of focus for testing:
* Areas of focus for testing:
** Flow, functionality, feel per/on each client
** Flow, functionality, feel per/on each client
Line 299: Line 206:
** Documentation review and support for Doc and SUMO teams
** Documentation review and support for Doc and SUMO teams
** Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)
** Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)
* Bug reporting will happen through Bugzilla
* REFs:
** Primary: https://wiki.mozilla.org/Identity/UX


== TS 5: Firefox OS, FTU/FTE, and Sync 1.1 ==
== TS 3: Desktop/Web Client and Sync 1.5 ==
* TBD - requirements here, especially for Q4/2013 are still being worked out.
* This is possibly old Sync plus new client auth
* TBD - this could cover the basics of FTU/FTE, Marketplace, and WhereIsMyFox
* NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
* NOTE: this is (or could be) Sign in to FFos as well as Sign in to Marketplace (as a separate path).
* NOTE: must cover the basic functional/compatibility testing of Sync 1.1 in this suite.
* REFs:
** http://www.mozilla.org/en-US/firefox/os/
** https://id.etherpad.mozilla.org/fxa-on-fxos
** https://id.etherpad.mozilla.org/persona-goes-native
** https://docs.google.com/spreadsheet/ccc?key=0AtVT90hlMtdSdEd4TVVjWXNfU3ctMlVhWFRrWkpweVE#gid=39


== TS 6: Desktop Client and Sync 1.1 ==
* NOTE: Add in Master Password testing
* This is possibly old Sync plus new client auth
* NOTE: must cover the basic functional/compatibility testing of Sync 1.1 in this suite.


* Areas of focus for testing:
* Areas of focus for testing:
Line 328: Line 223:
*** Use of Moztrap to create/maintain test cases
*** Use of Moztrap to create/maintain test cases
** Client automation - unit/functional with Selenium/Mozmill
** Client automation - unit/functional with Selenium/Mozmill
*** To be coordinated with the FF desktop/client QA team
*** To be coordinated with the Fx desktop/client QA team
** Firefox integration (Try/Nightly from ELM)
** Firefox integration (Try/Nightly from ELM)
** Firefox integration (Aurora, Beta, Release)  
** Firefox integration (Aurora, Beta, Release)  
Line 341: Line 236:
** Desktop integration - all supported platforms and OS
** Desktop integration - all supported platforms and OS
*** Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
*** Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
**** Match to FF support (including FF ESR)
**** Match to Fx support (including Fx ESR)
** FireFox command line for testing, setting up sync, authing, profiles, who knows?
** FireFox command line for testing, setting up sync, authing, profiles, who knows?
* Bug reporting will happen through Bugzilla
* Tools
** Use of FF desktop-specific debug tools/methods (Developer tools)
** HTML logging (https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging)
* See Section 5 for Sync 1.1 and new Sync integration/interaction
* See Section 6 for Sync 1.1 migration to new Sync
* Notes
** What about users new to sync?
* REFs:
** https://wiki.mozilla.org/User_Services/Sync#Technical_Materials
** Email thread: ClientImplementation Milestone 1
** more TBD
== TS7: Web-based Client ==
* TBD - to cover gherkin support in non-FF browsers across various platforms/OS
* Pull from desktop section above, since it looks like both "platforms" will share the same client structure/code.
* TBD - sync interaction from plain web front-end


== TS 8: Mobile Client (Android) and Sync 1.1 ==
== TS 4: Mobile Client (Android) and Sync 1.5 ==
* This is possibly old Sync plus new client auth
* This is possibly old Sync plus new client auth
* NOTE: must cover the basic functional/compatibility testing of Sync 1.1 in this suite.
* NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
* NOTE: Add in Master Password testing


* Areas of focus for testing:
* Areas of focus for testing:
Line 375: Line 254:
*** Use of Moztrap to create/maintain test cases
*** Use of Moztrap to create/maintain test cases
** Client automation - unit/functional - do we have anything here besides phone/os emulation?
** Client automation - unit/functional - do we have anything here besides phone/os emulation?
*** To be coordinated with the FF Mobile QA team
*** To be coordinated with the Fx Mobile QA team
** Firefox integration (Try/Nightly from ELM)
** Firefox integration (Try/Nightly from ELM)
** Firefox integration (Aurora, Beta, Release)
** Firefox integration (Aurora, Beta, Release)
Line 388: Line 267:
** Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
** Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
*** Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
*** Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
**** Match to FF support
**** Match to Fx support
* Bug reporting will happen through Bugzilla
 
* Tools
* Tools
** Use of Android specific adk and debug tools/methods
** Use of Android specific adk and debug tools/methods
* See Section 5 for Sync 1.1 and new Sync integration/interaction
* See Section 5 for Sync 1.5 and new Sync integration/interaction
* See Section 6 for Sync 1.1 migration to new Sync
* See Section 6 for Sync 1.5 migration to new Sync
* Notes
** What about users who only have mobile devices. How do we make sure they can sync across those devices?
** How do they do first setup if they never used sync on a desktop device
** What about users new to sync?
* REFs:
** https://wiki.mozilla.org/User_Services/Sync#Technical_Materials
** Email thread: ClientImplementation Milestone 1
** more TBD
 
== TS9: Marketplace Support (all platforms) ==
* TBD - to cover FxA and Marketplace interaction on FFos and all supported platforms
* Test FxA account activity:
** On FFos
** From a Firefox browser (desktop/android)
** From a non-FF browser using gherkin support
 
== TS10: WhereIsMyFox Support (all platforms) ==
* TBD - to cover FxA and WhereIsMyFox interaction on FFos and all supported platforms
* Test FxA account activity:
** On FFos
** From a Firefox browser (desktop/android)
** From a non-FF browser using gherkin support


== TS 11: FxA Documentation and SUMO ==
== TS 5: Firefox OS and Sync 1.5 ==
Why? Because current documentation (all of it really) is a pain point
* TBD - requirements here, especially for Q2 2014 (FFos 1.4) are still being worked out.
* Note: For QA, this is mostly a tracking responsibility
* TBD - this could cover the basics of FTU/FTE, Marketplace, WhereIsMyFox, and Contacts
* Updates to the old/current Sync information across locations
* NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
** internal, app, wiki, SUMO, MDN, etc
* Creation of new Sync information across locations
** internal, app, wiki, SUMO, MDN, etc
* Dev and QA can help define, write, and review documentation
** UX/UI Demos and training
** Usage demos and training (through SUMO?)
** Documentation team support is needed
** SUMO team support is needed
** REFs: TBD


== TS 12: FxA Localization ==
== TS 6: Server/Backend Data Storage ==
* Note: For QA, this is mostly a tracking responsibility
* TBD, but for v1/m1, this will be Sync 1.5, currently living in PHX1
* across UX/UI/clients
* See also TS 4: UX/UI
 
* Mobile and Desktop clients - full UX/UI experience, wording, etc.
* Dev and QA can help with string verfication before and after localization
* Will need support from the FireFox Localization team and community
* Bug reporting will happen through Bugzilla
* REFs: TBD
 
 
== TS 13: Metro (Win8/tablet) and Sync support ==
* FxA/Sync/Metro integration - TBD
 
 
== TS 14: Server/Backend Data Storage ==
* TBD, but for v1/m1, this will be Sync 1.1, currently living in PHX1
** Note this is the Sync Server portion of the product.
** Note this is the Sync Server portion of the product.
** The Auth data store is covered in Section 1: Authentication.
** The Auth data store is covered in Section 1: Authentication.
* REFs
** Sync 1.1: https://docs.services.mozilla.com/storage/apis-1.1.html (primarily)
** Sync 1.1 client Test Plan: n/a
** Sync 1.1 server-side Test Plan: TBD
** Deployments: https://wiki.mozilla.org/Identity/AttachedServices/DeploymentPlanning


== TS 15: Sync 1.1 and "new" Sync Interaction [lead: kthiessen] ==
== TS 7: Sync 1.5 and "new" Sync Interaction [lead: kthiessen] ==
* There will be some amount of time where
* There will be some amount of time where
** Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
** Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
Line 474: Line 300:
** General account interaction: old sync, new Sync, Persona
** General account interaction: old sync, new Sync, Persona
** Documentation review pertaining to old Sync, new Sync
** Documentation review pertaining to old Sync, new Sync
* Bug reporting will happen through Bugzilla
* See Section 6 for the specific migration testing tasks
* REFs:
** https://wiki.mozilla.org/User_Services/Sync/v1#Goals


== TS 16: Migration to "new" Sync [lead: kthiessen] ==
== TS 8: Migration to "new" Sync [lead: kthiessen] ==
* At some point, all old Sync users will either have to
* At some point, all old Sync users will either have to
** Migrate to new Sync
** Migrate to new Sync
Line 490: Line 312:
** Dev and OPs suppport will be needed for the test and verification steps.
** Dev and OPs suppport will be needed for the test and verification steps.
** Documentation review pertaining to old Sync, new Sync and migration steps/strategies
** Documentation review pertaining to old Sync, new Sync and migration steps/strategies
== TS 9: FxA Documentation and SUMO ==
* Why? Because current documentation (all of it really) is a pain point
* Note: For QA, this is mostly a tracking responsibility
* Updates to the old/current Sync information across locations
** internal, app, wiki, SUMO, MDN, etc
* Creation of new Sync information across locations
** internal, app, wiki, SUMO, MDN, etc
* Dev and QA can help define, write, and review documentation
** UX/UI Demos and training
** Usage demos and training (through SUMO?)
** Documentation team support is needed
** SUMO team support is needed
** REFs: TBD
== TS 10: FxA Localization ==
* Note: For QA, this is mostly a tracking responsibility
* across UX/UI/clients
* See also TS 4: UX/UI
* Mobile and Desktop clients - full UX/UI experience, wording, etc.
* Dev and QA can help with string verfication before and after localization
* Will need support from the FireFox Localization team and community
* Bug reporting will happen through Bugzilla
* Bug reporting will happen through Bugzilla
* See Migration Strategy for steps involved and timing
* REFs: TBD
* Notes:  
 
** From the user end, we want to make sure as much data as possible is migrated over to new sync.
== TS 11: Marketplace Support ==
** Old Sync data will exist in PHX1 datacenter, new Sync data may exist there or in AWS
* Test FxA account activity:
** I assume all this work will be done manually by the user (since data is encrypted, OPs can not automatically move data over). Will be a challenge to see how this is handled per account vs.per device (if per device syncing is supported).
** On FFos
* REFs:
** From a Firefox browser (desktop/android)
** https://wiki.mozilla.org/User_Services/Sync/v1#Migration_strategy
** From a non-Fx browser using the web interface
** https://services.etherpad.mozilla.org/sync-migration
* See the Marketplace Test Plan for more information: TBD
 
== TS 12: Where Is My Fox/Find My Device Support ==
* Test FxA account activity:
** On FFos
** From a Firefox browser (desktop/android)
** From a non-Fx browser using the web interface
* See the Find My Device Test Plan for more information: TBD
 
== TS 13: Metro (Win8/tablet) and Sync support ==
* FxA/Sync/Metro integration - TBD


= Main References =
= Main References =


== Wikis and Etherpads ==
== Wikis and Etherpads ==
* Main FxA Dev wiki: https://wiki.mozilla.org/Identity/FirefoxAccounts
* Main FxA wiki: https://wiki.mozilla.org/Identity/FirefoxAccounts
* Main FxA wiki: https://wiki.mozilla.org/User_Services/Sync/Relaunch
* Roadmap: https://wiki.mozilla.org/Identity/Roadmap
* Roadmap: https://wiki.mozilla.org/Identity/Roadmap
* Architecture: https://wiki.mozilla.org/Identity/Firefox-Accounts#Architecture
* Architecture: https://wiki.mozilla.org/Identity/Firefox-Accounts#Architecture
* Main FxA PM wiki: https://wiki.mozilla.org/User_Services/Sync
* Main FxA PM wiki: https://wiki.mozilla.org/User_Services/Sync
* Main FxA UX/UI wiki: https://wiki.mozilla.org/Identity/UX
* Main FxA UX/UI wiki: https://wiki.mozilla.org/Identity/UX
* Transition: https://wiki.mozilla.org/User_Services/Sync/Transition_To_FxA_Sync
* FxA-Auth-Server (API): https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md
* FxA-Auth-Server (API): https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md


* Older sites for FxA v1:
* Meetings:
** Main: https://wiki.mozilla.org/User_Services/Sync/v1
** Goals site: https://wiki.mozilla.org/User_Services/Sync/v1#Goals
** User Stories site: https://wiki.mozilla.org/User_Services/Sync/v1#User_stories
** UX/UI: https://wiki.mozilla.org/User_Services/Sync/v1#UX_design
** UX/UI: https://wiki.mozilla.org/User_Services/Sync#UX_Designs
** UX/UI: https://wiki.mozilla.org/User_Services/Sync#User_Research
** Sync Migration: https://wiki.mozilla.org/User_Services/Sync/v1#Migration_strategy
**  Technical Materials and Links: https://wiki.mozilla.org/User_Services/Sync#Technical_Materials
 
* FxA Product: https://id.etherpad.mozilla.org/FxA-Product
* FxA Product: https://id.etherpad.mozilla.org/FxA-Product
* FxA Product Coordination: https://id.etherpad.mozilla.org/picl-product-coordination
* FxA Engineering Coordination: https://id.etherpad.mozilla.org/fxa-engineering-coordination
* FxA Engineering Coordination: https://id.etherpad.mozilla.org/picl-engineering-coordination
* FxA Web Coordination: https://id.etherpad.mozilla.org/fxa-web-coordination
* Sync Engineering Stand-Up: https://id.etherpad.mozilla.org/sync-eng-standup
* FxA Branding: https://id.etherpad.mozilla.org/fxa-branding
* FxA Metrics/Monitoring: https://id.etherpad.mozilla.org/fxa-metrics-monitoring
* FxA Engagement: https://id.etherpad.mozilla.org/fxa-engagement


== GitHub Repositories ==
== GitHub Repositories ==
* https://github.com/mozilla/fxa-auth-server
* https://github.com/mozilla/fxa-auth-server
* https://github.com/mozilla/fxa-scrypt-helper
* https://github.com/mozilla/fxa-content-server
* https://github.com/mozilla/fxa-content-server
* https://github.com/mozilla/fxa-js-client
* https://github.com/mozilla/fxa-js-client
* https://github.com/mozilla/fxa-js-client-old
* https://github.com/mozilla/browserid-verifier
* https://github.com/mozilla/fxa-profile-server


* https://github.com/warner/picl-spec-crypto
* https://github.com/warner/picl-spec-crypto
Line 536: Line 388:
* https://github.com/mozilla/jwcrypto
* https://github.com/mozilla/jwcrypto
* https://github.com/mozilla/restmail.net
* https://github.com/mozilla/restmail.net
* https://github.com/mozilla/picl-mocks
* FxA for the Web (TBD)
** https://id.etherpad.mozilla.org/gherkin-web
** https://wiki.mozilla.org/Identity/Firefox-Accounts#Architecture


* Dev and Loadtest Deployments
* Dev and Loadtest Deployments
Line 547: Line 394:
* Loadtest (for fxa-auth-server)
* Loadtest (for fxa-auth-server)
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads.js


* FFos specific
* Android
** TBD
** https://github.com/mozilla-services/android-sync
 
* Tokenserver and Sync
** https://github.com/mozilla-services/tokenserver
** https://github.com/mozilla-services/server-syncstorage


* Server-side: OLD
* OPs
** https://github.com/mozilla/picl-server
** https://github.com/mozilla-services/puppet-config
** https://github.com/mozilla-services/svcops-oompaloompas


== Travis CI ==
== Travis CI ==
* https://travis-ci.org/mozilla/picl-server
* https://travis-ci.org/mozilla/fxa-auth-server
* https://travis-ci.org/mozilla/fxa-auth-server
* https://travis-ci.org/mozilla/fxa-content-server
* https://travis-ci.org/mozilla/fxa-content-server
* https://travis-ci.org/mozilla/fxa-scrypt-helper
* https://travis-ci.org/mozilla/fxa-scrypt-helper
* https://travis-ci.org/mozilla/fxa-js-client
* https://travis-ci.org/mozilla/fxa-js-client
* https://travis-ci.org/mozilla/fxa-js-client-old
* https://travis-ci.org/lloyd/restmail.net

Latest revision as of 20:05, 26 August 2016

This is the Test Plan for FxA, Tokenserver, and Sync 1.5

Intro/Summary/Notes

Intro and Summary

  • The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.5.
  • A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc.
  • It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.
  • Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases for each of the main areas of testing: Server-side, Desktop/Web, Android, FFos.

Notes

  • FxA is scheduled to go out with Firefox 29. This will replace old sync account functionality and use.
  • Sync users from Firefox 28 and earlier will need to create new FxA accounts when they upgrade to Fx29.
  • Once that happens, devices hooked to sync for Fx28 and older will have to be unlinked and then added to the new FxA account in order to sync.
  • If a user wants to hold all devices at old sync, he/she must not upgrade to Fx29.
  • A user can not have accounts with devices across versions of Fx older than 29.

User Stories

  • NOTE: For all platforms, see also the bug trees below...

Building Test Cases

  • QA needs at least one resource, probably from SoftVision, to convert the current Android, desktop/web, and FFos user stories into client-side test cases.
  • Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
  • Scheduling: TBD

Schedule

QA Team

  • Services QA
    • Edwin Wong - Manager
    • James Bonacci
    • John Morrison
    • Karl Thiessen
    • Peter deHaan
    • Richard Pappalardo
  • QA
    • Various members of the Firefox OS, Desktop, and Mobile teams will be involved in the testing.

OPs Information

  • OPs has a significant number of really detailed Mana pages for deploying, testing, maintaining, and monitoring our Stage and Production environments.
  • Talk to a member of the OPs team for more information and for the Mana links to get you started.

Email Lists and IRC

Bug Lists and Reporting

  • Dev-related issues will be mixed within Bugzilla and GitHub (but see trees/meta info below)
  • Product issues will all live in Bugzilla
  • User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
    • Classification: TBD
    • Products: TBD
    • Components: TBD
  • Note: We may need to cross link Bugzilla bugs with GitHub issues

Firefox Releases

Platforms/OS

  • Mac 10.7.x and 10.8.x
  • Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
  • Windows 8 (Metro mode and desktop mode) 32bit/64bit
  • Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
  • Android 2.x - 4.x

ELM Nightly Builds

Dev Test Environments

LoadTest Environment

Stage Environments

Test Sections/Suites

TS 1: Authentication and Accounts - Core

  • This is Authentication Server/fxa-auth (IdP), Account Server/fxa-content
  • Main parts:
    • fxa-auth service
    • fxa-content service
  • Areas of focus for testing: primarily server-side
    • Manual testing - unit/functional/API
    • Test automation - unit/functional/API - an API Test Suite, for example
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Server load testing with Loads tool (with heka for stats collection), monitoring, logging
    • Performance testing with Loads tool (possibly with some logging, monitoring, hekad stuff hooked up)
    • Security (end to end)
    • Scaling - AWS stack (CF) - single-region/multi-region, ELB, instances, use of PHX1
    • Documentation review and support for Doc and SUMO teams
    • Env-specific deployments on localhosts and in AWS
    • Build/deploy your own - similar to Sync 1.5 and Persona
  • Track/Test (as needed)
    • Internal dependencies: repos, packages, and libs to track and to test
      • restmail, node-kvstore, node-srp, jwcrypto, picl-spec-crypto
    • Loads integration
    • SimplePush integration (has this been defined for v1/milestone 1?)
    • Heka integration
    • External dependencies: Hawk, Hapi, Good, Kibana, etc.
  • Notes
    • DB/storage issues and dependencies
      • What level of testing vs. monitoring is required
    • Server-specific notes and information:
      • Stats collection is going to happen with Heka
      • Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite

TS 2: UX/UI

  • This is across product (desktop vs. mobile client vs. FFos vs. web)
  • See also TS 10: Localization
  • Areas of focus for testing:
    • Flow, functionality, feel per/on each client
    • Consistency, repeatability, stability between devices/platforms
    • Functional tests against all important features - Sign Up, Sign In, Password change/reset, etc.
    • Functional tests cases for all UX/UI related user stories
    • Functional tests specific to mobile clients
    • Functional tests specific to desktop clients
    • UX/UI verification against V1/Milestone 1 goals and user stories (see REFs below...)
    • QA will help or at least support the localization testing of the UI
    • Documentation review and support for Doc and SUMO teams
    • Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)

TS 3: Desktop/Web Client and Sync 1.5

  • This is possibly old Sync plus new client auth
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
  • NOTE: Add in Master Password testing
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of desktop client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional with Selenium/Mozmill
      • To be coordinated with the Fx desktop/client QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Desktop-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all desktop-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Desktop integration - all supported platforms and OS
      • Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
        • Match to Fx support (including Fx ESR)
    • FireFox command line for testing, setting up sync, authing, profiles, who knows?

TS 4: Mobile Client (Android) and Sync 1.5

  • This is possibly old Sync plus new client auth
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
  • NOTE: Add in Master Password testing
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of mobile client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional - do we have anything here besides phone/os emulation?
      • To be coordinated with the Fx Mobile QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Mobile-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all mobile-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
      • Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
        • Match to Fx support
  • Tools
    • Use of Android specific adk and debug tools/methods
  • See Section 5 for Sync 1.5 and new Sync integration/interaction
  • See Section 6 for Sync 1.5 migration to new Sync

TS 5: Firefox OS and Sync 1.5

  • TBD - requirements here, especially for Q2 2014 (FFos 1.4) are still being worked out.
  • TBD - this could cover the basics of FTU/FTE, Marketplace, WhereIsMyFox, and Contacts
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.

TS 6: Server/Backend Data Storage

  • TBD, but for v1/m1, this will be Sync 1.5, currently living in PHX1
    • Note this is the Sync Server portion of the product.
    • The Auth data store is covered in Section 1: Authentication.

TS 7: Sync 1.5 and "new" Sync Interaction [lead: kthiessen]

  • There will be some amount of time where
    • Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
    • A "migration" will start to happen where users voluntarily replace old sync with new sync
    • A "forced migration" or "shutdown period" where users will have to move from old sync to new sync
    • Only the first item will be tested in this section
  • Areas of focus for testing:
    • Functional test independent use of both old sync and new sync, per client/platform/device
    • Functional test interactions between the two (combined use)(especially if per device syncing is supported), per client/platform/device
    • Test/verify data storage for both sync models per account/device
    • UX/UI interactions and methods, especially for Sign Up and Sign In on both Sync models, and other supported preferences and UI features
    • Old Sync/new Sync verification against V1/Milestone 1 goals
    • General account interaction: old sync, new Sync, Persona
    • Documentation review pertaining to old Sync, new Sync

TS 8: Migration to "new" Sync [lead: kthiessen]

  • At some point, all old Sync users will either have to
    • Migrate to new Sync
    • Stop using Sync
  • Areas of focus for testing:
    • QA will need to prepare a set of steps, before and after tests, to verify migration
    • QA will need to interrogate/investigate account contents from old and new sync to verify migration
    • All of this QA work can be done in Stage first to verify migration will actually run as expected.
    • Dev and OPs suppport will be needed for the test and verification steps.
    • Documentation review pertaining to old Sync, new Sync and migration steps/strategies

TS 9: FxA Documentation and SUMO

  • Why? Because current documentation (all of it really) is a pain point
  • Note: For QA, this is mostly a tracking responsibility
  • Updates to the old/current Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Creation of new Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Dev and QA can help define, write, and review documentation
    • UX/UI Demos and training
    • Usage demos and training (through SUMO?)
    • Documentation team support is needed
    • SUMO team support is needed
    • REFs: TBD

TS 10: FxA Localization

  • Note: For QA, this is mostly a tracking responsibility
  • across UX/UI/clients
  • See also TS 4: UX/UI
  • Mobile and Desktop clients - full UX/UI experience, wording, etc.
  • Dev and QA can help with string verfication before and after localization
  • Will need support from the FireFox Localization team and community
  • Bug reporting will happen through Bugzilla
  • REFs: TBD

TS 11: Marketplace Support

  • Test FxA account activity:
    • On FFos
    • From a Firefox browser (desktop/android)
    • From a non-Fx browser using the web interface
  • See the Marketplace Test Plan for more information: TBD

TS 12: Where Is My Fox/Find My Device Support

  • Test FxA account activity:
    • On FFos
    • From a Firefox browser (desktop/android)
    • From a non-Fx browser using the web interface
  • See the Find My Device Test Plan for more information: TBD

TS 13: Metro (Win8/tablet) and Sync support

  • FxA/Sync/Metro integration - TBD

Main References

Wikis and Etherpads

GitHub Repositories

Travis CI

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/SyncTestPlanV1&oldid=1145611"