TestEngineering/Services/SyncTestPlanV1
From MozillaWiki
< TestEngineering | Services(Redirected from QA/Services/SyncTestPlanV1)
This is the Test Plan for FxA, Tokenserver, and Sync 1.5
Contents
- 1 Intro/Summary/Notes
- 2 Schedule
- 3 QA Team
- 4 OPs Information
- 5 Email Lists and IRC
- 6 Bug Lists and Reporting
- 7 Firefox Releases
- 8 Platforms/OS
- 9 ELM Nightly Builds
- 10 Dev Test Environments
- 11 LoadTest Environment
- 12 Stage Environments
- 13 Test Sections/Suites
- 13.1 TS 1: Authentication and Accounts - Core
- 13.2 TS 2: UX/UI
- 13.3 TS 3: Desktop/Web Client and Sync 1.5
- 13.4 TS 4: Mobile Client (Android) and Sync 1.5
- 13.5 TS 5: Firefox OS and Sync 1.5
- 13.6 TS 6: Server/Backend Data Storage
- 13.7 TS 7: Sync 1.5 and "new" Sync Interaction [lead: kthiessen]
- 13.8 TS 8: Migration to "new" Sync [lead: kthiessen]
- 13.9 TS 9: FxA Documentation and SUMO
- 13.10 TS 10: FxA Localization
- 13.11 TS 11: Marketplace Support
- 13.12 TS 12: Where Is My Fox/Find My Device Support
- 13.13 TS 13: Metro (Win8/tablet) and Sync support
- 14 Main References
Intro/Summary/Notes
Intro and Summary
- The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.5.
- A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc.
- It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.
- Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases for each of the main areas of testing: Server-side, Desktop/Web, Android, FFos.
Notes
- FxA is scheduled to go out with Firefox 29. This will replace old sync account functionality and use.
- Sync users from Firefox 28 and earlier will need to create new FxA accounts when they upgrade to Fx29.
- Once that happens, devices hooked to sync for Fx28 and older will have to be unlinked and then added to the new FxA account in order to sync.
- If a user wants to hold all devices at old sync, he/she must not upgrade to Fx29.
- A user can not have accounts with devices across versions of Fx older than 29.
User Stories
- Start here for Android and desktop/web:
- For FFos, start here:
- NOTE: For all platforms, see also the bug trees below...
Building Test Cases
- QA needs at least one resource, probably from SoftVision, to convert the current Android, desktop/web, and FFos user stories into client-side test cases.
- Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
- Scheduling: TBD
Schedule
- NEWER: https://wiki.mozilla.org/User_Services/Sync/Relaunch#Burndown_Worksheet
- NEW: https://wiki.mozilla.org/Identity/Roadmap
- OLD: https://wiki.mozilla.org/User_Services/Sync/v1
QA Team
- Services QA
- Edwin Wong - Manager
- James Bonacci
- John Morrison
- Karl Thiessen
- Peter deHaan
- Richard Pappalardo
- QA
- Various members of the Firefox OS, Desktop, and Mobile teams will be involved in the testing.
- Note: FxA and Sync Teams information can be found here:
OPs Information
- OPs has a significant number of really detailed Mana pages for deploying, testing, maintaining, and monitoring our Stage and Production environments.
- Talk to a member of the OPs team for more information and for the Mana links to get you started.
Email Lists and IRC
- Services QA: services-qa@mozilla.com
- QA Teams: qa@mozilla.com
- Sync and FxA:
- NEW: dev-fxacct@mozila.org, https://mail.mozilla.org/listinfo/dev-fxacct
- OLD: sync-dev@mozilla.org, https://mail.mozilla.org/listinfo/sync-dev
- IRC:
- #fxa
- #fxosqa
- #native_identity
- #identity
- #sync
- #androidsync
- #sync-tactics
Bug Lists and Reporting
- Dev-related issues will be mixed within Bugzilla and GitHub (but see trees/meta info below)
- Product issues will all live in Bugzilla
- FxA/FFos Program: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941078&hide_resolved=0
- OLD: FFos Dev https://bugzilla.mozilla.org/showdependencytree.cgi?id=920135&hide_resolved=0
- NEW: Fxa/FFos: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941723&hide_resolved=0
- Mobile (Android): https://bugzilla.mozilla.org/showdependencytree.cgi?id=799726&hide_resolved=0
- Desktop Dev: https://bugzilla.mozilla.org/showdependencytree.cgi?id=969593&hide_resolved=0
- Note there is also the following trackers:
- Bug 969626 to track Firefox 29-relevant Sync issues that affect desktop, but that are not client issues.
- Note there is also the following trackers:
- Server-side Dev: https://bugzilla.mozilla.org/showdependencytree.cgi?id=907475&hide_resolved=0
- Fx Accounts Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907494
- Sync Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907479
- UI Bugs for Aurora
- Enable FxA Sync on Nightly
- WheresMyFox Client: https://bugzilla.mozilla.org/showdependencytree.cgi?id=938896&hide_resolved=0
- User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
- Classification: TBD
- Products: TBD
- Components: TBD
- Note: We may need to cross link Bugzilla bugs with GitHub issues
Firefox Releases
- Early stages
- ELM builds for Android and Desktop (see below)
- FFos unrelease builds: TBD
- Nightly
- http://nightly.mozilla.org/
- Similar requirements to above
- Three-channel focus with latest features, fixes, etc.
- Aurora: Index of /pub/mozilla.org/firefox/nightly/latest-mozilla-aurora
- Beta: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/latest-beta
- Release: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases
- Android-specific: http://ftp.mozilla.org/pub/mozilla.org/mobile
Platforms/OS
- Mac 10.7.x and 10.8.x
- Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
- Windows 8 (Metro mode and desktop mode) 32bit/64bit
- Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
- Android 2.x - 4.x
- The Goal is to stay consistent with requirements for FireFox and Marketplace.
- For more information, see the following issue: https://github.com/mozilla/fxa-content-server/issues/105
ELM Nightly Builds
- Desktop
- Android
Dev Test Environments
- See the following wiki:
- Note: You will need Dev/QA AWS credentials to access these servers.
LoadTest Environment
- OLD:
- Note: You will need Dev/QA AWS credentials to access these servers.
- NEW:
- Note: You will need special AWS credentials to access these servers.
Stage Environments
- See the following wiki sections:
Test Sections/Suites
TS 1: Authentication and Accounts - Core
- This is Authentication Server/fxa-auth (IdP), Account Server/fxa-content
- Main parts:
- fxa-auth service
- fxa-content service
- Areas of focus for testing: primarily server-side
- Manual testing - unit/functional/API
- Test automation - unit/functional/API - an API Test Suite, for example
- Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
- Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
- Server load testing with Loads tool (with heka for stats collection), monitoring, logging
- Performance testing with Loads tool (possibly with some logging, monitoring, hekad stuff hooked up)
- Security (end to end)
- Scaling - AWS stack (CF) - single-region/multi-region, ELB, instances, use of PHX1
- Documentation review and support for Doc and SUMO teams
- Env-specific deployments on localhosts and in AWS
- Build/deploy your own - similar to Sync 1.5 and Persona
- Track/Test (as needed)
- Internal dependencies: repos, packages, and libs to track and to test
- restmail, node-kvstore, node-srp, jwcrypto, picl-spec-crypto
- Loads integration
- SimplePush integration (has this been defined for v1/milestone 1?)
- Heka integration
- External dependencies: Hawk, Hapi, Good, Kibana, etc.
- Internal dependencies: repos, packages, and libs to track and to test
- Notes
- DB/storage issues and dependencies
- What level of testing vs. monitoring is required
- Server-specific notes and information:
- Stats collection is going to happen with Heka
- Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite
- DB/storage issues and dependencies
TS 2: UX/UI
- This is across product (desktop vs. mobile client vs. FFos vs. web)
- See also TS 10: Localization
- Areas of focus for testing:
- Flow, functionality, feel per/on each client
- Consistency, repeatability, stability between devices/platforms
- Functional tests against all important features - Sign Up, Sign In, Password change/reset, etc.
- Functional tests cases for all UX/UI related user stories
- Functional tests specific to mobile clients
- Functional tests specific to desktop clients
- UX/UI verification against V1/Milestone 1 goals and user stories (see REFs below...)
- QA will help or at least support the localization testing of the UI
- Documentation review and support for Doc and SUMO teams
- Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)
TS 3: Desktop/Web Client and Sync 1.5
- This is possibly old Sync plus new client auth
- NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
- NOTE: Add in Master Password testing
- Areas of focus for testing:
- Unit testing on localhost/remote host
- API testing - TBD
- Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
- Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
- Manual testing of desktop client - per account vs. per device syncing (TBD)
- Functional testing of all supported features:
- account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
- Use of Moztrap to create/maintain test cases
- Functional testing of all supported features:
- Client automation - unit/functional with Selenium/Mozmill
- To be coordinated with the Fx desktop/client QA team
- Firefox integration (Try/Nightly from ELM)
- Firefox integration (Aurora, Beta, Release)
- Security and privacy
- Documentation review and support for Doc and SUMO teams
- Desktop-specific verification against V1/Milestone 1 goals
- Functional tests cases for all desktop-client-related user stories
- Interaction with about:config for logging, debugging
- Interaction with add-ons, extensions, themes, personas, etc.
- Desktop integration - all supported platforms and OS
- Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
- Match to Fx support (including Fx ESR)
- Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
- FireFox command line for testing, setting up sync, authing, profiles, who knows?
TS 4: Mobile Client (Android) and Sync 1.5
- This is possibly old Sync plus new client auth
- NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
- NOTE: Add in Master Password testing
- Areas of focus for testing:
- Unit testing on localhost/remote host
- API testing - TBD
- Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
- Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
- Manual testing of mobile client - per account vs. per device syncing (TBD)
- Functional testing of all supported features:
- account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
- Use of Moztrap to create/maintain test cases
- Functional testing of all supported features:
- Client automation - unit/functional - do we have anything here besides phone/os emulation?
- To be coordinated with the Fx Mobile QA team
- Firefox integration (Try/Nightly from ELM)
- Firefox integration (Aurora, Beta, Release)
- Security and privacy
- Documentation review and support for Doc and SUMO teams
- Mobile-specific verification against V1/Milestone 1 goals
- Functional tests cases for all mobile-client-related user stories
- Interaction with about:config for logging, debugging
- Interaction with add-ons, extensions, themes, personas, etc.
- Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
- Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
- Match to Fx support
- Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
- Tools
- Use of Android specific adk and debug tools/methods
- See Section 5 for Sync 1.5 and new Sync integration/interaction
- See Section 6 for Sync 1.5 migration to new Sync
TS 5: Firefox OS and Sync 1.5
- TBD - requirements here, especially for Q2 2014 (FFos 1.4) are still being worked out.
- TBD - this could cover the basics of FTU/FTE, Marketplace, WhereIsMyFox, and Contacts
- NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
TS 6: Server/Backend Data Storage
- TBD, but for v1/m1, this will be Sync 1.5, currently living in PHX1
- Note this is the Sync Server portion of the product.
- The Auth data store is covered in Section 1: Authentication.
TS 7: Sync 1.5 and "new" Sync Interaction [lead: kthiessen]
- There will be some amount of time where
- Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
- A "migration" will start to happen where users voluntarily replace old sync with new sync
- A "forced migration" or "shutdown period" where users will have to move from old sync to new sync
- Only the first item will be tested in this section
- Areas of focus for testing:
- Functional test independent use of both old sync and new sync, per client/platform/device
- Functional test interactions between the two (combined use)(especially if per device syncing is supported), per client/platform/device
- Test/verify data storage for both sync models per account/device
- UX/UI interactions and methods, especially for Sign Up and Sign In on both Sync models, and other supported preferences and UI features
- Old Sync/new Sync verification against V1/Milestone 1 goals
- General account interaction: old sync, new Sync, Persona
- Documentation review pertaining to old Sync, new Sync
TS 8: Migration to "new" Sync [lead: kthiessen]
- At some point, all old Sync users will either have to
- Migrate to new Sync
- Stop using Sync
- Areas of focus for testing:
- QA will need to prepare a set of steps, before and after tests, to verify migration
- QA will need to interrogate/investigate account contents from old and new sync to verify migration
- All of this QA work can be done in Stage first to verify migration will actually run as expected.
- Dev and OPs suppport will be needed for the test and verification steps.
- Documentation review pertaining to old Sync, new Sync and migration steps/strategies
TS 9: FxA Documentation and SUMO
- Why? Because current documentation (all of it really) is a pain point
- Note: For QA, this is mostly a tracking responsibility
- Updates to the old/current Sync information across locations
- internal, app, wiki, SUMO, MDN, etc
- Creation of new Sync information across locations
- internal, app, wiki, SUMO, MDN, etc
- Dev and QA can help define, write, and review documentation
- UX/UI Demos and training
- Usage demos and training (through SUMO?)
- Documentation team support is needed
- SUMO team support is needed
- REFs: TBD
TS 10: FxA Localization
- Note: For QA, this is mostly a tracking responsibility
- across UX/UI/clients
- See also TS 4: UX/UI
- Mobile and Desktop clients - full UX/UI experience, wording, etc.
- Dev and QA can help with string verfication before and after localization
- Will need support from the FireFox Localization team and community
- Bug reporting will happen through Bugzilla
- REFs: TBD
TS 11: Marketplace Support
- Test FxA account activity:
- On FFos
- From a Firefox browser (desktop/android)
- From a non-Fx browser using the web interface
- See the Marketplace Test Plan for more information: TBD
TS 12: Where Is My Fox/Find My Device Support
- Test FxA account activity:
- On FFos
- From a Firefox browser (desktop/android)
- From a non-Fx browser using the web interface
- See the Find My Device Test Plan for more information: TBD
TS 13: Metro (Win8/tablet) and Sync support
- FxA/Sync/Metro integration - TBD
Main References
Wikis and Etherpads
- Main FxA wiki: https://wiki.mozilla.org/Identity/FirefoxAccounts
- Main FxA wiki: https://wiki.mozilla.org/User_Services/Sync/Relaunch
- Roadmap: https://wiki.mozilla.org/Identity/Roadmap
- Architecture: https://wiki.mozilla.org/Identity/Firefox-Accounts#Architecture
- Main FxA PM wiki: https://wiki.mozilla.org/User_Services/Sync
- Main FxA UX/UI wiki: https://wiki.mozilla.org/Identity/UX
- Transition: https://wiki.mozilla.org/User_Services/Sync/Transition_To_FxA_Sync
- FxA-Auth-Server (API): https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md
- Meetings:
- FxA Product: https://id.etherpad.mozilla.org/FxA-Product
- FxA Engineering Coordination: https://id.etherpad.mozilla.org/fxa-engineering-coordination
- FxA Web Coordination: https://id.etherpad.mozilla.org/fxa-web-coordination
- Sync Engineering Stand-Up: https://id.etherpad.mozilla.org/sync-eng-standup
- FxA Branding: https://id.etherpad.mozilla.org/fxa-branding
- FxA Metrics/Monitoring: https://id.etherpad.mozilla.org/fxa-metrics-monitoring
- FxA Engagement: https://id.etherpad.mozilla.org/fxa-engagement
GitHub Repositories
- https://github.com/mozilla/fxa-auth-server
- https://github.com/mozilla/fxa-content-server
- https://github.com/mozilla/fxa-js-client
- https://github.com/mozilla/browserid-verifier
- https://github.com/mozilla/fxa-profile-server
- https://github.com/warner/picl-spec-crypto
- https://github.com/mozilla/node-kvstore
- https://github.com/mozilla/node-srp/
- https://github.com/mozilla/jwcrypto
- https://github.com/mozilla/restmail.net
- Dev and Loadtest Deployments
- Loadtest (for fxa-auth-server)
- Tokenserver and Sync
- OPs