MozillaWiki

Identity/DDT Smoke Test

< Identity
Revision as of 13:59, 8 May 2012 by Stomlinson (talk | contribs) (→‎Sanity/Acceptance/Smoke)

This document is taken from content on QA BrowserID TestPlan - https://wiki.mozilla.org/QA/BrowserID/TestPlan

Purpose

As BrowserID grows and the product becomes increasingly complex, it is a difficult task to introduce new features without also introducing bugs. To help reduce the number of bugs that make it to prod as well as to ease some of the pain on our own QA team, we are introducing a WWE move - the DDT. DDT (developer driven testing) is like jumping off the top rope to smash bugs before they have a chance to cause pain.

QA has a well defined set of tests they perform before qualifying any release as "ready for production." We are going to use the Sanity/Acceptance/Smoke and Support for Primaries portions of their tests as the basis for DDT. Functional and UI tests can also be run to ensure maximal flow coverage. If each developer/contributor is able to run through the tests on a couple of desktop browsers and a couple of mobile browsers, we should be able to completely cover all A-Grade browsers.

Tests should be performed against both dev and staging. 123done.org will be the RP of choice once it is set up to handle all environments. 123done.org currently only supports staging. Until 123done.org is ready, testing will take place using myfavoritebeer.org as the RP.

dev: http://dev.myfavoritebeer.org/ staging: http://beta.myfavoritebeer.org/

Tests

Sanity/Acceptance/Smoke

  • Small, repeatable set of tests with known, good, expected results.
  • See the Test Cases (listed in the "Links and Documentation" section above) for more information.
  • Main Site (https://diresworb.org/)
    • Sign Up with a new account and one email
      • Verify correct email verification sent to correct account (email provider)
    • Sign In with a current account of one email
    • Sign Out from a current account of one email
    • Change the password on a current account with one email
      • Verify correct email verification sent to correct account (email provider)
    • Remove email on account with one email closes account.
    • Delete account
  • Dialog (http://myfavoritebeer.org or http://123done.org using one desktop OS with one browser) - create account using secondary email
    • Sign In using one RP/client to create a new account with one secondary email
      • Ensure set password screen displayed and works as expected
      • Verify correct email verification sent to correct account (email provider)
    • Logout/Sign In/Logout using the same account from a different RP/client
    • Sign In and add one BID email to the same account from a different RP/client
      • Verify correct email verification sent to correct account (email provider)
    • Sign In and add one primary email to the same account
      • Verify user is correctly sent to IdP to verify credentials
    • Sign In and change the password on the same account
      • Verify correct email verification sent to correct account (email provider)
    • Sign In to the same account and "sign out" by using the This is not me... link
    • Delete/Remove the account from the server-side
  • Dialog (http://myfavoritebeer.org or http://123done.org using one desktop OS with one browser) - create account using primary email
    • Sign In using one RP/client to create a new account with one primary email
      • Verify user does not see set password screen but is correctly sent to IdP to verify credentials
    • Logout/Sign In/Logout using the same account from a different RP/client
    • Sign In and add second primary email to the account
      • Verify user is correctly sent to IdP to verify credentials
    • Sign In and add first secondary address to account
      • Ensure set password screen displayed and works as expected
      • Verify correct email verification sent to correct account (email provider)


  • Dialog (http://myfavoritebeer.org or http://123done.org using mobile browser)
    • Sign In using one RP/client to create a new account with one email
      • Verify correct email verification sent to correct account (email provider)
    • Logout/Sign In/Logout using the same account from a different RP/client
    • Sign In and add one BID email to the same account from a different RP/client
      • Verify correct email verification sent to correct account (email provider)
    • Sign In and add one primary email to the same account
    • Sign In and change the password on the same account
      • Verify correct email verification sent to correct account (email provider)
    • Sign In to the same account and "sign out" by using the This is not me... link
    • Delete/Remove the account from the server-side

Support for Primaries

  • Verify basic Primary support through the use of one or more test Primary sites
  • Primary: https://eyedee.me/
  • BrowserID tests (from Server or from RPs)
    • Creating an account with Primary emails
    • Creating an account with mixed emails (Primary/BID)
    • Adding/Deleting a Primary email from a Primary account
    • Adding/Deleting a BID email from a Primary account
    • Adding/Deleting a Primary email from a BID account
    • Adding/Deleting a BID email from a BID account
    • Deleting an account with Primary emails
    • Deleting an account with mixed emails (Primary/BID)
    • Account Manager password changes on accounts with both primary and secondary emails
  • Primary site UI flow
    • General tests for navigating the site
    • Include failures, cancellations, backing out



Basic Functional

Manual and automated testing on the client and the server to verify basic functionality of BrowserID:

  • Accounts and Email Verification
    • Creating an account from the server site using Sign Up
    • Creating an account inline (at first use of an RP/client Sign In)
    • Email notifications for new accounts: verification email through email provider with proper email account listed, live verification link, etc.
      • Test with emails/accounts on various, popular email servers/services
      • See more details in the next bulleted list
    • Creating multiple accounts with one or more emails
    • Deleting one or more accounts (cancellation) from the server site
  • Email Notifications for server or RP/Client
    • On the same OS
      • Email notification and verification using one browser
      • Email notification on one browser, verification on an another
    • Across multiple OS
      • Email notification on one specific browser on one specific OS
      • Verification on the same browser on a different OS
    • Other cases
      • Email notification on one specific browser on one specific OS
      • Verification on a different browser on a different OS
    • Mail Servers
      • Check functionality when the user does not verify by email (skips, forgets)
      • Check functionality when the user can not verify by email (email provider is down or user can not access email account for some reason)
    • BrowserID Server site
      • Check functionality when BrowserID server is unavailable (down or user is off the net)
      • Check functionality when BrowserID server is available but user has slow connection (like a public wifi)
  • Accounts and Emails
    • Adding additional emails to an account
    • Attempt to add an email (that may or may not be yours) from another account
    • Deleting one or more emails (without actually deleting the account)
    • Leaving/returning to sites (while signed in, after signing out)
    • Browser restart after creation of account or access of an account
    • Always logging out from sites vs. never logging out from sites (session timeouts)
    • Shared access to same computer or profiles or accounts with different users
    • Browser settings and preferences, esp. pop-ups, cookies, security, privacy
    • Cancelling accounts
    • Copy/Pasting emails (names) or passwords from other sources
    • Auto-completion of emails and passwords
    • Merging one or more accounts (that may or may not be yours)
  • Email and Password fields
    • Email strings/types/limits
    • Verify all legal combinations of characters for both "local name" and "domain name" parts of the email string
    • Password strings/types/limits
  • More on Emails and Passwords
    • Use of passwords (strict) vs. pass phrases
    • Verify minimum/maximum sizes of emails and passwords (length)
    • Password reset, password remember/restore
    • Unique/unusual/edge case emails and passwords
    • Email and Password character compatibility
    • Valid vs. invalid email formats
    • Valid vs. invalid password formats
    • Different accounts using same email/password combos
    • Try to setup a new account with a password and/or email already in use
    • Copy/Pasting passwords from other sources
    • Verify that passwords are never stored in LocalStore on the user machine
    • Verify whether or not passwords are stored client-side (Stage RP)
    • Verify whether or not passwords are stored on the server (Stage server)
    • Verify proper formatting with very long emails and/or passwords.
  • Cross site activity
    • Creating an account on one RP/client, verifying the account/email on another RP/client
    • Adding an email on one RP/client while logged into another RP/client
    • Deleting an email on one while logged into other RPs/clients
    • Cancel account/delete email from the server while signed into an RP/client
    • Deleting the whole account while logged into one or more RPs/clients
      • This must be done from the server
    • Verify how account information on the server reflects the changes for each of these tests
  • Other Areas
    • Verify that the user cannot log in with an email if he/she did not confirm the used email
    • Log in simultaneously in two different browsers with the same email, then log out from one of the two browsers
    • Log in with different emails for different clients in the same browser/different browser
    • Log in with the same email for different clients in the same browser, then log out from one of the browsers


UI

Manual and automated testing on the RP/client and the BID server to cover all aspects of the current UI.

  • Sign-in UI
  • Email field
  • Password field
  • "forgot your password?" link
  • "learn more" link
  • "This is not me" link
  • "Use a different email" link
  • "Terms of Service" and "Privacy" links off of RP sites
  • Account management: https://www.diresworb.org Account Manager page
  • Sign Up
  • Sign In
  • Edit button
  • Remove button
  • Password button
  • "cancel your account" link
  • UI for new user
    • Email field
    • Verify button
    • Verification email
    • Verification link
    • Password
    • Email selection - radio button vs. email string
  • Confirm your Email UI - verifying accurate "prove" link
    • Confirm email verification from client-side and server-side, as defined
  • Confirm Email verification UI
  • UI to Add new emails ("Use a different email") while still logged into RP/client
  • UI to Change a password ("Forgot your password?") on the RP/client
  • UI to select Terms of Services, Privacy, Learn More on the RP/client
  • UI to Edit the account on the Server from the Account Manager
  • UI to Remove an email on the Server from the Account Manager
  • UI to Change the Password on the Server from the Account Manager
    • Verify functionality with only BID emails
    • Verify functionality with only Primary emails
    • Verify functionality with a mix of email types
  • UI to Cancel an account (all emails) on the Server
  • General UI navigation to cover the following:
    • Error screens and dialogs
    • Email verification pop-ups and in browser (the fade/change to a server page)
    • Closing pop-ups manually rather than with a Cancel, OK, Continue, or other button
  • Various UI scenarios and navigation not covered above...
    • The affects of Stay logged in vs. always logging out
  • General ease of use
    • Mouse/pointer: left (click), right/ctrl click, hover over links, etc.
    • Keyboard: selection, tabbing, arrows, etc.
    • Mobile: touchscreen, keyboard, etc.


Accessibility - Desktop only

  • Verify minimal accessibility in the UI - keyboard only
    • Creating a new BrowserID account
      • Email verification from mail application and from BrowserID
    • Adding a new email to a current BrowserID account
      • Email verification from mail application and from BrowserID
    • Changing/resetting a password - RP flow
      • Email verification from mail application and from BrowserID
    • Changing a password - Account Manager
    • Deleting an email from an account - Account Manager
    • Deleting an account - Account Manager
    • Other UI flows: This is not me, BrowserID links, etc.
  • Note: this testing may require changes at the OS-level (like Mac OS) or changes at the browser level to more fully support keyboard-only access to a site.
Retrieved from "https://wiki.mozilla.org/index.php?title=Identity/DDT_Smoke_Test&oldid=428442"