TestEngineering/Services/SyncTestPlanV1: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(189 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Sync/Picl Test Plan V1 =
This is the Test Plan for FxA, Tokenserver, and Sync 1.5


== Intro/Summary/Notes ==
= Intro/Summary/Notes =
=== Intro ===
== Intro and Summary ==
* TBD
* The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.5.
* A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc.
* It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.


=== Summary ===
* Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases for each of the main areas of testing: Server-side, Desktop/Web, Android, FFos.
* A complete snapshot of what it takes to test and release picl/sync: tools, processes, schedules,  
resources, etc


=== Notes ===
== Notes ==
* This is a living document. As such, assume very regular changes, updates, and rewrites as we
* FxA is scheduled to go out with Firefox 29. This will replace old sync account functionality and use.
go through the Dev milestones of v1.
* Sync users from Firefox 28 and earlier will need to create new FxA accounts when they upgrade to Fx29.
* This document is designed to accompany the v1 release. Although it will be used as a reference
* Once that happens, devices hooked to sync for Fx28 and older will have to be unlinked and then added to the new FxA account in order to sync.
for the various Dev milestones for v1, it is not specific to any one milestone.
* If a user wants to hold all devices at old sync, he/she must not upgrade to Fx29.
* Milestone-specific test plans could be broken out from here to focus on very specific test sections, as defined below.
* A user can not have accounts with devices across versions of Fx older than 29.


* Requested sync-able collections for V1/M1:
== User Stories ==
** Awesomebar/History
* Start here for Android and desktop/web:  
** Passwords
** https://wiki.mozilla.org/User_Services/Sync/Relaunch#Product_Requirements_.26_User_Stories
** Open Tabs
* For FFos, start here:
** Bookmarks
** https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AtVT90hlMtdSdEd4TVVjWXNfU3ctMlVhWFRrWkpweVE#gid=50
** (supported per account, but not per device)
** https://mana.mozilla.org/wiki/display/Identity/Firefox+Accounts+%28FxA%29+for+FxOS


* Requested basic feature set for V1/M1:
* NOTE: For all platforms, see also the bug trees below...
** Device compatibility (desktop/mobile only, not FFox)
** Setup (Sign Up, Sign In)
** Account Management
** Password change/reset(forgot)
** Detach a device (maintain the account)
** Better security/encryption
** Performance/stability


== Schedule ==
== Building Test Cases ==
* QA needs at least one resource, probably from SoftVision, to convert the current Android, desktop/web, and FFos user stories into client-side test cases.
* Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
* Scheduling: TBD


= Schedule =
* NEWER: https://wiki.mozilla.org/User_Services/Sync/Relaunch#Burndown_Worksheet
* NEW: https://wiki.mozilla.org/Identity/Roadmap
* OLD: https://wiki.mozilla.org/User_Services/Sync/v1


== Team ==
= QA Team =
* Services QA
** Edwin Wong - Manager
** James Bonacci
** John Morrison
** Karl Thiessen
** Peter deHaan
** Richard Pappalardo


* QA
** Various members of the Firefox OS, Desktop, and Mobile teams will be involved in the testing.


== Email Lists ==
* Note: FxA and Sync Teams information can be found here:
** https://wiki.mozilla.org/User_Services/Sync/Relaunch/DRIs
** https://wiki.mozilla.org/Identity/FirefoxAccounts#Team


= OPs Information =
* OPs has a significant number of really detailed Mana pages for deploying, testing, maintaining, and monitoring our Stage and Production environments.
* Talk to a member of the OPs team for more information and for the Mana links to get you started.


== Bug Reporting ==
= Email Lists and IRC =
* Services QA: services-qa@mozilla.com
* QA Teams: qa@mozilla.com
* Sync and FxA:
** NEW: dev-fxacct@mozila.org, https://mail.mozilla.org/listinfo/dev-fxacct
** OLD: sync-dev@mozilla.org, https://mail.mozilla.org/listinfo/sync-dev
* IRC:
** #fxa
** #fxosqa
** #native_identity
** #identity
** #sync
** #androidsync
** #sync-tactics


= Bug Lists and Reporting =
* Dev-related issues will be mixed within Bugzilla and GitHub (but see trees/meta info below)
* Product issues will all live in Bugzilla


* All FxA/Sync https://bugzilla.mozilla.org/showdependencytree.cgi?id=905997&hide_resolved=0


* FxA/FFos Program: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941078&hide_resolved=0
* OLD: FFos Dev https://bugzilla.mozilla.org/showdependencytree.cgi?id=920135&hide_resolved=0
* NEW: Fxa/FFos: https://bugzilla.mozilla.org/showdependencytree.cgi?id=941723&hide_resolved=0


* FxA/Sync Relaunch: https://bugzilla.mozilla.org/showdependencytree.cgi?id=909322&hide_resolved=0


* Mobile (Android): https://bugzilla.mozilla.org/showdependencytree.cgi?id=799726&hide_resolved=0
* Desktop Dev: https://bugzilla.mozilla.org/showdependencytree.cgi?id=969593&hide_resolved=0
** Note there is also the following trackers:
*** Bug 969626 to track Firefox 29-relevant Sync issues that affect desktop, but that are not client issues.
* Server-side Dev: https://bugzilla.mozilla.org/showdependencytree.cgi?id=907475&hide_resolved=0
** Fx Accounts Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907494
** Sync Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=907479


* UI Bugs for Aurora
** https://bugzilla.mozilla.org/showdependencytree.cgi?id=964920&hide_resolved=1
* Enable FxA Sync on Nightly
** https://bugzilla.mozilla.org/showdependencytree.cgi?id=964922&hide_resolved=1


== Main References ==
* WheresMyFox Client: https://bugzilla.mozilla.org/showdependencytree.cgi?id=938896&hide_resolved=0


=== Wikis and Etherpads ===
* Contacts on FFos: https://bugzilla.mozilla.org/showdependencytree.cgi?id=976837&hide_resolved=1
* Main Site: https://wiki.mozilla.org/User_Services/Sync
* Milestone 1: https://etherpad.mozilla.org/SyncMilestone1
* Version 1: https://wiki.mozilla.org/User_Services/Sync/v1
* Goals: https://wiki.mozilla.org/User_Services/Sync/v1#Goals
* User Stories: https://wiki.mozilla.org/User_Services/Sync/v1#User_stories
* UX: https://wiki.mozilla.org/User_Services/Sync/v1#UX_design
** and here: https://wiki.mozilla.org/User_Services/Sync#UX_Designs
** and here: https://wiki.mozilla.org/User_Services/Sync#User_Research
** and here: https://wiki.mozilla.org/Identity/UX
* Migration: https://wiki.mozilla.org/User_Services/Sync/v1#Migration_strategy
** and here: https://services.etherpad.mozilla.org/sync-migration
* Technical Materials and Links: https://wiki.mozilla.org/User_Services/Sync#Technical_Materials


* Numerous other wikis and etherpads, including the following:
* User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
** https://wiki.mozilla.org/Identity/AttachedServices
** Classification: TBD
** https://wiki.mozilla.org/Identity/CryptoIdeas
** Products: TBD
** https://wiki.mozilla.org/Identity/AttachedServices/DeploymentPlanning
** Components: TBD
* Note: We may need to cross link Bugzilla bugs with GitHub issues


= Firefox Releases =
* Early stages
** ELM builds for Android and Desktop (see below)
** FFos unrelease builds: TBD
* Nightly
** http://nightly.mozilla.org/
** Similar requirements to above
* Three-channel focus with latest features, fixes, etc.
** Aurora: Index of /pub/mozilla.org/firefox/nightly/latest-mozilla-aurora
** Beta: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/latest-beta
** Release: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases
** Android-specific: http://ftp.mozilla.org/pub/mozilla.org/mobile


=== GitHub Repositories ===
= Platforms/OS =
* https://github.com/mozilla/picl-idp
* Mac 10.7.x and 10.8.x
* Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
* Windows 8 (Metro mode and desktop mode) 32bit/64bit
* Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
* Android 2.x - 4.x


* https://github.com/mozilla/mozilla-central
* The Goal is to stay consistent with requirements for FireFox and Marketplace.
* For more information, see the following issue: https://github.com/mozilla/fxa-content-server/issues/105
 
= ELM Nightly Builds =
* Desktop
** http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-elm/
* Android
** http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-elm-android/
** http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-elm-android-x86/
** http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-elm-android-armv6/
 
= Dev Test Environments =
* See the following wiki:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Development_Environments_-_partial
* Note: You will need Dev/QA AWS credentials to access these servers.
 
= LoadTest Environment =
* OLD:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Load_Test_Environment
* Note: You will need Dev/QA AWS credentials to access these servers.
 
* NEW:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Loads_Cluster_Environment
* Note: You will need special AWS credentials to access these servers.
 
= Stage Environments =
* See the following wiki sections:
** FxA: https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA_Stage_Environment
** TokenServer: https://wiki.mozilla.org/QA/Services/FxATestEnvironments#TokenServer_Stage_Environment
** Sync: https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Sync_1.5_Stage_Environment
 
= Test Sections/Suites =
== TS 1: Authentication and Accounts - Core ==
* This is Authentication Server/fxa-auth (IdP), Account Server/fxa-content
* Main parts:
** fxa-auth service
** fxa-content service
 
* Areas of focus for testing: primarily server-side
** Manual testing - unit/functional/API
** Test automation - unit/functional/API - an API Test Suite, for example
** Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
** Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
** Server load testing with Loads tool (with heka for stats collection), monitoring, logging
** Performance testing with Loads tool (possibly with some logging, monitoring, hekad stuff hooked up)
** Security (end to end)
** Scaling - AWS stack (CF) - single-region/multi-region, ELB, instances, use of PHX1
** Documentation review and support for Doc and SUMO teams
** Env-specific deployments on localhosts and in AWS
** Build/deploy your own - similar to Sync 1.5 and Persona
 
* Track/Test (as needed)
** Internal dependencies: repos, packages, and libs to track and to test
*** restmail, node-kvstore, node-srp, jwcrypto, picl-spec-crypto
** Loads integration
** SimplePush integration (has this been defined for v1/milestone 1?)
** Heka integration
** External dependencies: Hawk, Hapi, Good, Kibana, etc.
 
* Notes
** DB/storage issues and dependencies
*** What level of testing vs. monitoring is required
** Server-specific notes and information:
*** Stats collection is going to happen with Heka
*** Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite
 
== TS 2: UX/UI ==
* This is across product (desktop vs. mobile client vs. FFos vs. web)
* See also TS 10: Localization
 
* Areas of focus for testing:
** Flow, functionality, feel per/on each client
** Consistency, repeatability, stability between devices/platforms
** Functional tests against all important features - Sign Up, Sign In, Password change/reset, etc.
** Functional tests cases for all UX/UI related user stories
** Functional tests specific to mobile clients
** Functional tests specific to desktop clients
** UX/UI verification against V1/Milestone 1 goals and user stories (see REFs below...)
** QA will help or at least support the localization testing of the UI
** Documentation review and support for Doc and SUMO teams
** Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)
 
== TS 3: Desktop/Web Client and Sync 1.5 ==
* This is possibly old Sync plus new client auth
* NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
 
* NOTE: Add in Master Password testing
 
* Areas of focus for testing:
** Unit testing on localhost/remote host
** API testing - TBD
** Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
** Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
** Manual testing of desktop client - per account vs. per device syncing (TBD)
*** Functional testing of all supported features:
**** account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
*** Use of Moztrap to create/maintain test cases
** Client automation - unit/functional with Selenium/Mozmill
*** To be coordinated with the Fx desktop/client QA team
** Firefox integration (Try/Nightly from ELM)
** Firefox integration (Aurora, Beta, Release)
** Security and privacy
** Documentation review and support for Doc and SUMO teams
** Desktop-specific verification against V1/Milestone 1 goals
*** https://wiki.mozilla.org/User_Services/Sync/v1#Goals
** Functional tests cases for all desktop-client-related user stories
*** https://wiki.mozilla.org/User_Services/Sync/v1#User_stories
** Interaction with about:config for logging, debugging
** Interaction with add-ons, extensions, themes, personas, etc.
** Desktop integration - all supported platforms and OS
*** Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
**** Match to Fx support (including Fx ESR)
** FireFox command line for testing, setting up sync, authing, profiles, who knows?
 
== TS 4: Mobile Client (Android) and Sync 1.5 ==
* This is possibly old Sync plus new client auth
* NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
* NOTE: Add in Master Password testing
 
* Areas of focus for testing:
** Unit testing on localhost/remote host
** API testing - TBD
** Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
** Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
** Manual testing of mobile client - per account vs. per device syncing (TBD)
*** Functional testing of all supported features:
**** account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
*** Use of Moztrap to create/maintain test cases
** Client automation - unit/functional - do we have anything here besides phone/os emulation?
*** To be coordinated with the Fx Mobile QA team
** Firefox integration (Try/Nightly from ELM)
** Firefox integration (Aurora, Beta, Release)
** Security and privacy
** Documentation review and support for Doc and SUMO teams
** Mobile-specific verification against V1/Milestone 1 goals
*** https://wiki.mozilla.org/User_Services/Sync/v1#Goals
** Functional tests cases for all mobile-client-related user stories
*** https://wiki.mozilla.org/User_Services/Sync/v1#User_stories
** Interaction with about:config for logging, debugging
** Interaction with add-ons, extensions, themes, personas, etc.
** Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
*** Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
**** Match to Fx support
 
* Tools
** Use of Android specific adk and debug tools/methods
* See Section 5 for Sync 1.5 and new Sync integration/interaction
* See Section 6 for Sync 1.5 migration to new Sync
 
== TS 5: Firefox OS and Sync 1.5 ==
* TBD - requirements here, especially for Q2 2014 (FFos 1.4) are still being worked out.
* TBD - this could cover the basics of FTU/FTE, Marketplace, WhereIsMyFox, and Contacts
* NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
 
== TS 6: Server/Backend Data Storage ==
* TBD, but for v1/m1, this will be Sync 1.5, currently living in PHX1
** Note this is the Sync Server portion of the product.
** The Auth data store is covered in Section 1: Authentication.
 
== TS 7: Sync 1.5 and "new" Sync Interaction [lead: kthiessen] ==
* There will be some amount of time where
** Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
** A "migration" will start to happen where users voluntarily replace old sync with new sync
** A "forced migration" or "shutdown period" where users will have to move from old sync to new sync
** Only the first item will be tested in this section
 
* Areas of focus for testing:
** Functional test independent use of both old sync and new sync, per client/platform/device
** Functional test interactions between the two (combined use)(especially if per device syncing is supported), per client/platform/device
** Test/verify data storage for both sync models per account/device
** UX/UI interactions and methods, especially for Sign Up and Sign In on both Sync models, and other supported preferences and UI features
** Old Sync/new Sync verification against V1/Milestone 1 goals
*** https://wiki.mozilla.org/User_Services/Sync/v1#Goals
** General account interaction: old sync, new Sync, Persona
** Documentation review pertaining to old Sync, new Sync
 
== TS 8: Migration to "new" Sync [lead: kthiessen] ==
* At some point, all old Sync users will either have to
** Migrate to new Sync
** Stop using Sync
 
* Areas of focus for testing:
** QA will need to prepare a set of steps, before and after tests, to verify migration
** QA will need to interrogate/investigate account contents from old and new sync to verify migration
** All of this QA work can be done in Stage first to verify migration will actually run as expected.
** Dev and OPs suppport will be needed for the test and verification steps.
** Documentation review pertaining to old Sync, new Sync and migration steps/strategies
 
== TS 9: FxA Documentation and SUMO ==
* Why? Because current documentation (all of it really) is a pain point
* Note: For QA, this is mostly a tracking responsibility
* Updates to the old/current Sync information across locations
** internal, app, wiki, SUMO, MDN, etc
* Creation of new Sync information across locations
** internal, app, wiki, SUMO, MDN, etc
* Dev and QA can help define, write, and review documentation
** UX/UI Demos and training
** Usage demos and training (through SUMO?)
** Documentation team support is needed
** SUMO team support is needed
** REFs: TBD
 
== TS 10: FxA Localization ==
* Note: For QA, this is mostly a tracking responsibility
* across UX/UI/clients
* See also TS 4: UX/UI
 
* Mobile and Desktop clients - full UX/UI experience, wording, etc.
* Dev and QA can help with string verfication before and after localization
* Will need support from the FireFox Localization team and community
* Bug reporting will happen through Bugzilla
* REFs: TBD
 
== TS 11: Marketplace Support ==
* Test FxA account activity:
** On FFos
** From a Firefox browser (desktop/android)
** From a non-Fx browser using the web interface
* See the Marketplace Test Plan for more information: TBD
 
== TS 12: Where Is My Fox/Find My Device Support ==
* Test FxA account activity:
** On FFos
** From a Firefox browser (desktop/android)
** From a non-Fx browser using the web interface
* See the Find My Device Test Plan for more information: TBD
 
== TS 13: Metro (Win8/tablet) and Sync support ==
* FxA/Sync/Metro integration - TBD
 
= Main References =
 
== Wikis and Etherpads ==
* Main FxA wiki: https://wiki.mozilla.org/Identity/FirefoxAccounts
* Main FxA wiki: https://wiki.mozilla.org/User_Services/Sync/Relaunch
* Roadmap: https://wiki.mozilla.org/Identity/Roadmap
* Architecture: https://wiki.mozilla.org/Identity/Firefox-Accounts#Architecture
* Main FxA PM wiki: https://wiki.mozilla.org/User_Services/Sync
* Main FxA UX/UI wiki: https://wiki.mozilla.org/Identity/UX
* Transition: https://wiki.mozilla.org/User_Services/Sync/Transition_To_FxA_Sync
* FxA-Auth-Server (API): https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md
 
* Meetings:
* FxA Product: https://id.etherpad.mozilla.org/FxA-Product
* FxA Engineering Coordination: https://id.etherpad.mozilla.org/fxa-engineering-coordination
* FxA Web Coordination: https://id.etherpad.mozilla.org/fxa-web-coordination
* Sync Engineering Stand-Up: https://id.etherpad.mozilla.org/sync-eng-standup
* FxA Branding: https://id.etherpad.mozilla.org/fxa-branding
* FxA Metrics/Monitoring: https://id.etherpad.mozilla.org/fxa-metrics-monitoring
* FxA Engagement: https://id.etherpad.mozilla.org/fxa-engagement
 
== GitHub Repositories ==
* https://github.com/mozilla/fxa-auth-server
* https://github.com/mozilla/fxa-content-server
* https://github.com/mozilla/fxa-js-client
* https://github.com/mozilla/browserid-verifier
* https://github.com/mozilla/fxa-profile-server
 
* https://github.com/warner/picl-spec-crypto
* https://github.com/mozilla/node-kvstore
* https://github.com/mozilla/node-kvstore
* https://github.com/mozilla/node-srp/
* https://github.com/mozilla/node-srp/
* https://github.com/mozilla/jwcrypto
* https://github.com/mozilla/jwcrypto
* https://github.com/mozilla/scrypt-helper
* https://github.com/mozilla/restmail.net
* https://github.com/mozilla/restmail.net
* https://github.com/mozilla/firefox-account-bridge
 
* https://github.com/mozilla/picl-gherkin
* Dev and Loadtest Deployments
** https://github.com/mozilla/fxa-deployment
 
* Loadtest (for fxa-auth-server)
** https://github.com/mozilla-services/loads
 
* Android
** https://github.com/mozilla-services/android-sync
 
* Tokenserver and Sync
** https://github.com/mozilla-services/tokenserver
** https://github.com/mozilla-services/server-syncstorage
 
* OPs
** https://github.com/mozilla-services/puppet-config
** https://github.com/mozilla-services/svcops-oompaloompas
 
== Travis CI ==
* https://travis-ci.org/mozilla/fxa-auth-server
* https://travis-ci.org/mozilla/fxa-content-server
* https://travis-ci.org/mozilla/fxa-scrypt-helper
* https://travis-ci.org/mozilla/fxa-js-client
* https://travis-ci.org/lloyd/restmail.net

Latest revision as of 20:05, 26 August 2016

This is the Test Plan for FxA, Tokenserver, and Sync 1.5

Intro/Summary/Notes

Intro and Summary

  • The following document/wiki outlines the "catch-all" test requirements for FxA (Firefox Accounts), TokenServer, and Sync 1.5 to be released as the first true replacement product for Sync 1.5.
  • A complete snapshot of what it takes to test and release FxA, TokenServer, and Sync 1.5 on Desktop, Android, Web, and FFos: tools, processes, schedules, resources, etc.
  • It also includes information that can be selected and made more specific to the testing needs of the Development milestones m1, m2, m*, etc : FxA functionality, FTU/FTE support , Desktop suupport, Android support, FxA for the Web, Marketplace support, WhereIsMyFirefox support, etc.
  • Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases for each of the main areas of testing: Server-side, Desktop/Web, Android, FFos.

Notes

  • FxA is scheduled to go out with Firefox 29. This will replace old sync account functionality and use.
  • Sync users from Firefox 28 and earlier will need to create new FxA accounts when they upgrade to Fx29.
  • Once that happens, devices hooked to sync for Fx28 and older will have to be unlinked and then added to the new FxA account in order to sync.
  • If a user wants to hold all devices at old sync, he/she must not upgrade to Fx29.
  • A user can not have accounts with devices across versions of Fx older than 29.

User Stories

  • NOTE: For all platforms, see also the bug trees below...

Building Test Cases

  • QA needs at least one resource, probably from SoftVision, to convert the current Android, desktop/web, and FFos user stories into client-side test cases.
  • Additional test cases (manual/automated) are needed for the client-portions of FxA for FFos, desktop, Android, and web.
  • Scheduling: TBD

Schedule

QA Team

  • Services QA
    • Edwin Wong - Manager
    • James Bonacci
    • John Morrison
    • Karl Thiessen
    • Peter deHaan
    • Richard Pappalardo
  • QA
    • Various members of the Firefox OS, Desktop, and Mobile teams will be involved in the testing.

OPs Information

  • OPs has a significant number of really detailed Mana pages for deploying, testing, maintaining, and monitoring our Stage and Production environments.
  • Talk to a member of the OPs team for more information and for the Mana links to get you started.

Email Lists and IRC

Bug Lists and Reporting

  • Dev-related issues will be mixed within Bugzilla and GitHub (but see trees/meta info below)
  • Product issues will all live in Bugzilla
  • User/QA/Testing bugs will live in Bugzilla, per mobile/desktop
    • Classification: TBD
    • Products: TBD
    • Components: TBD
  • Note: We may need to cross link Bugzilla bugs with GitHub issues

Firefox Releases

Platforms/OS

  • Mac 10.7.x and 10.8.x
  • Windows XP 32bit, Windows 7 32bit, Windows 7 64bit
  • Windows 8 (Metro mode and desktop mode) 32bit/64bit
  • Linux 32bit and 64bit - mostly likely one of Ubuntu latest, Mint latest, Fedora latest
  • Android 2.x - 4.x

ELM Nightly Builds

Dev Test Environments

LoadTest Environment

Stage Environments

Test Sections/Suites

TS 1: Authentication and Accounts - Core

  • This is Authentication Server/fxa-auth (IdP), Account Server/fxa-content
  • Main parts:
    • fxa-auth service
    • fxa-content service
  • Areas of focus for testing: primarily server-side
    • Manual testing - unit/functional/API
    • Test automation - unit/functional/API - an API Test Suite, for example
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Server load testing with Loads tool (with heka for stats collection), monitoring, logging
    • Performance testing with Loads tool (possibly with some logging, monitoring, hekad stuff hooked up)
    • Security (end to end)
    • Scaling - AWS stack (CF) - single-region/multi-region, ELB, instances, use of PHX1
    • Documentation review and support for Doc and SUMO teams
    • Env-specific deployments on localhosts and in AWS
    • Build/deploy your own - similar to Sync 1.5 and Persona
  • Track/Test (as needed)
    • Internal dependencies: repos, packages, and libs to track and to test
      • restmail, node-kvstore, node-srp, jwcrypto, picl-spec-crypto
    • Loads integration
    • SimplePush integration (has this been defined for v1/milestone 1?)
    • Heka integration
    • External dependencies: Hawk, Hapi, Good, Kibana, etc.
  • Notes
    • DB/storage issues and dependencies
      • What level of testing vs. monitoring is required
    • Server-specific notes and information:
      • Stats collection is going to happen with Heka
      • Monitoring/Logging/Process views/OPs Stuff to happen with OPsview and/or Graphite

TS 2: UX/UI

  • This is across product (desktop vs. mobile client vs. FFos vs. web)
  • See also TS 10: Localization
  • Areas of focus for testing:
    • Flow, functionality, feel per/on each client
    • Consistency, repeatability, stability between devices/platforms
    • Functional tests against all important features - Sign Up, Sign In, Password change/reset, etc.
    • Functional tests cases for all UX/UI related user stories
    • Functional tests specific to mobile clients
    • Functional tests specific to desktop clients
    • UX/UI verification against V1/Milestone 1 goals and user stories (see REFs below...)
    • QA will help or at least support the localization testing of the UI
    • Documentation review and support for Doc and SUMO teams
    • Accounts interaction: old Sync, new Sync, Persona, SITP/SITB (if supported)

TS 3: Desktop/Web Client and Sync 1.5

  • This is possibly old Sync plus new client auth
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
  • NOTE: Add in Master Password testing
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of desktop client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional with Selenium/Mozmill
      • To be coordinated with the Fx desktop/client QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Desktop-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all desktop-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Desktop integration - all supported platforms and OS
      • Minimal desktop support vs. unsupported list to be documented (Example Win7 but not WinXP)
        • Match to Fx support (including Fx ESR)
    • FireFox command line for testing, setting up sync, authing, profiles, who knows?

TS 4: Mobile Client (Android) and Sync 1.5

  • This is possibly old Sync plus new client auth
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.
  • NOTE: Add in Master Password testing
  • Areas of focus for testing:
    • Unit testing on localhost/remote host
    • API testing - TBD
    • Account/Email/Password usage (Sync vs. Persona vs. third-party etc)
    • Verification of what the user specifies for the account vs. what is actually used for sync (account bridging)
    • Manual testing of mobile client - per account vs. per device syncing (TBD)
      • Functional testing of all supported features:
        • account creation/deletion, account admin, collections, preferences, account page, passwords, etc.
      • Use of Moztrap to create/maintain test cases
    • Client automation - unit/functional - do we have anything here besides phone/os emulation?
      • To be coordinated with the Fx Mobile QA team
    • Firefox integration (Try/Nightly from ELM)
    • Firefox integration (Aurora, Beta, Release)
    • Security and privacy
    • Documentation review and support for Doc and SUMO teams
    • Mobile-specific verification against V1/Milestone 1 goals
    • Functional tests cases for all mobile-client-related user stories
    • Interaction with about:config for logging, debugging
    • Interaction with add-ons, extensions, themes, personas, etc.
    • Android integration: android/ARMv7, ARMv6, x86, emulation (across old to new phones/tablets)
      • Minimal Android support vs. unsupported list to be documented (Example 4.x but not 2.x or 3.x)
        • Match to Fx support
  • Tools
    • Use of Android specific adk and debug tools/methods
  • See Section 5 for Sync 1.5 and new Sync integration/interaction
  • See Section 6 for Sync 1.5 migration to new Sync

TS 5: Firefox OS and Sync 1.5

  • TBD - requirements here, especially for Q2 2014 (FFos 1.4) are still being worked out.
  • TBD - this could cover the basics of FTU/FTE, Marketplace, WhereIsMyFox, and Contacts
  • NOTE: must cover the basic functional/compatibility testing of Sync 1.5 in this suite.

TS 6: Server/Backend Data Storage

  • TBD, but for v1/m1, this will be Sync 1.5, currently living in PHX1
    • Note this is the Sync Server portion of the product.
    • The Auth data store is covered in Section 1: Authentication.

TS 7: Sync 1.5 and "new" Sync Interaction [lead: kthiessen]

  • There will be some amount of time where
    • Old Sync and New Sync will coexist and users will be interacting with neither, one, or both
    • A "migration" will start to happen where users voluntarily replace old sync with new sync
    • A "forced migration" or "shutdown period" where users will have to move from old sync to new sync
    • Only the first item will be tested in this section
  • Areas of focus for testing:
    • Functional test independent use of both old sync and new sync, per client/platform/device
    • Functional test interactions between the two (combined use)(especially if per device syncing is supported), per client/platform/device
    • Test/verify data storage for both sync models per account/device
    • UX/UI interactions and methods, especially for Sign Up and Sign In on both Sync models, and other supported preferences and UI features
    • Old Sync/new Sync verification against V1/Milestone 1 goals
    • General account interaction: old sync, new Sync, Persona
    • Documentation review pertaining to old Sync, new Sync

TS 8: Migration to "new" Sync [lead: kthiessen]

  • At some point, all old Sync users will either have to
    • Migrate to new Sync
    • Stop using Sync
  • Areas of focus for testing:
    • QA will need to prepare a set of steps, before and after tests, to verify migration
    • QA will need to interrogate/investigate account contents from old and new sync to verify migration
    • All of this QA work can be done in Stage first to verify migration will actually run as expected.
    • Dev and OPs suppport will be needed for the test and verification steps.
    • Documentation review pertaining to old Sync, new Sync and migration steps/strategies

TS 9: FxA Documentation and SUMO

  • Why? Because current documentation (all of it really) is a pain point
  • Note: For QA, this is mostly a tracking responsibility
  • Updates to the old/current Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Creation of new Sync information across locations
    • internal, app, wiki, SUMO, MDN, etc
  • Dev and QA can help define, write, and review documentation
    • UX/UI Demos and training
    • Usage demos and training (through SUMO?)
    • Documentation team support is needed
    • SUMO team support is needed
    • REFs: TBD

TS 10: FxA Localization

  • Note: For QA, this is mostly a tracking responsibility
  • across UX/UI/clients
  • See also TS 4: UX/UI
  • Mobile and Desktop clients - full UX/UI experience, wording, etc.
  • Dev and QA can help with string verfication before and after localization
  • Will need support from the FireFox Localization team and community
  • Bug reporting will happen through Bugzilla
  • REFs: TBD

TS 11: Marketplace Support

  • Test FxA account activity:
    • On FFos
    • From a Firefox browser (desktop/android)
    • From a non-Fx browser using the web interface
  • See the Marketplace Test Plan for more information: TBD

TS 12: Where Is My Fox/Find My Device Support

  • Test FxA account activity:
    • On FFos
    • From a Firefox browser (desktop/android)
    • From a non-Fx browser using the web interface
  • See the Find My Device Test Plan for more information: TBD

TS 13: Metro (Win8/tablet) and Sync support

  • FxA/Sync/Metro integration - TBD

Main References

Wikis and Etherpads

GitHub Repositories

Travis CI

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/SyncTestPlanV1&oldid=1145611"