QA/Fennec/Fx Accounts Functionality

From MozillaWiki
< QA‎ | Fennec
Jump to: navigation, search

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying that user can sign in with a Firefox Account to Firefox for Android

Ownership

Developer contact: [:ADavis]

QA: [:ninu] [:sorina] [Kanchan]

Overall Status

[LANDED] Nightly 54:

Testing Summary

Scope of testing

The testing will be focus on verifying that user can sign in with a Firefox Account to Firefox for Android

Environments

Testing will be performed on following devices:

  • Phone: 4.x, 5.x, 6.x, 7.x
  • Tablet: 4.x, 5.x, 6.x, 7.x

Specifications

  • A Firefox Account is only required for Mozilla Services that require authentication, such as Firefox Sync and advanced features on Firefox Marketplace like purchasing paid apps, adding app reviews etc.
  • Firefox Accounts only stores information that will deliver significant user value across applications or is tightly related to the user's identity. It will not store user data for relying services. Relying Mozilla services can use Firefox Accounts for authentication, but application data storage is the responsibility of the individual applications.

Acceptance Criteria

  • Independently of Persona, Mozilla needs an account database to deliver a fantastic, integrated experience across all its products and on all the user's devices.

Test Cases

Feature Testing

Regression Testing

Date:2017-02-02

  • Platforms used
- Windows 7 (64) and Windows 10 (64)
- Android : Samsung Galaxy tab active (Android 5.1.1),  Motorola Nexus 6 (Android 6.0.1)
- iOS : iPhone 6s (fw 10.2)
  • Newly logged bugs during this test session
ID Summary Priority Status
1336099 "Unexpected error" message is displayed when user changes preferences in Communication area -- VERIFIED

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);


End to End Testing

Date:2017-03-23

- Windows 7 (64) and Windows 10 (64)
- Android (Samsung Galaxy S6 EDGE - Android 6.0 , HTC Desire 820 - Android 6.0.1, Huawei MediaPad M2 - Android 5.1.1)
  • Sign-off mail send on 2017/03/29
  • Newly logged bugs during this test session
ID Summary Priority Status
1351632 [fxa][Staging 83] creating account stuck if email confirmation is triggered before choosing what to sync -- NEW
1351642 [fxa][Staging 83] Verification not always received when creating a new account -- RESOLVED
1351648 [fxa][Staging 83] Confirmation email not expired after an hour -- RESOLVED

3 Total; 1 Open (33.33%); 2 Resolved (66.67%); 0 Verified (0%);


Date:2017-04-06

  • Started End to End testing suite Train 84 - Staging Server on Windows, iOS and Android: https://goo.gl/trVxd5
  • Platforms used:
- Windows 7 (64) and Windows 10 (64)
- Android (Samsung Galaxy S6 EDGE - Android 6.0 , HTC Desire 820 - Android 6.0.1, Huawei MediaPad M2 - Android 5.1.1), Nexus 9 (Android N)
- iOS (iPhone 6Plus Firmware 9.3.5)
  • Sign-off mail send on 2017/04/07
  • Newly logged bugs during this test session
ID Summary Priority Status
1354133 [fxa][Staging 84] User can't change password for account -- NEW
1354481 [fxa][Staging 84] Loging in stuck in "working" even though the log in is performed in the background -- NEW
1354493 [fxa][Staging 84] Sync issues on Staging - Train 84 -- RESOLVED

3 Total; 2 Open (66.67%); 1 Resolved (33.33%); 0 Verified (0%);


  • Known bugs with high impact on testing process
ID Summary Priority Status
1351642 [fxa][Staging 83] Verification not always received when creating a new account -- RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);


Date:2017-04-20

  • Started End to End testing suite Train 85 - Staging Server on Windows, iOS and Android: https://goo.gl/HThwLZ
  • Platforms used:
- Windows 7 (64) and Windows 10 (64)
- Android (Samsung Galaxy S6 EDGE - Android 6.0 , HTC Desire 820 - Android 6.0.1, Huawei MediaPad M2 - Android 5.1.1), Nexus 9 (Android N)
- iOS (iPhone 6S Firmware 9.3.5)
  • Sign-off mail send on 2017/04/24
  • Known bugs with high impact on testing process
ID Summary Priority Status
1354493 [fxa][Staging 84] Sync issues on Staging - Train 84 -- RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);


Date:2017-05-04

  • Started End to End testing suite Train 86 - Staging Server on Windows, iOS and Android: https://goo.gl/9jWZjD
  • Platforms used:
- Windows 7 (64) and Windows 10 (64)
- Android: Honor 8 (Android 6.0), Asus ZenPad 8(Android 6.0.1), Nexus 6P (Android N), Nexus 9 (Android N)
- iOS (iPhone 6S Firmware 9.3.5)
  • Sign-off mail send on 2017/05/08

Date:2017-05-18

  • Started End to End testing suite Train 87 - Staging Server on Windows, iOS and Android: https://goo.gl/d8W9UK
  • Platforms used:
- Windows 7 (64) and Windows 10 (64)	
- Android: (Samsung Galaxy S6 EDGE - (Android 6.0) , Xiaomi mi i4 (Android 5.1.1),  Nexus 9 (Android N), Asus ZenPad 8(Android 6.0.1)
- iOS (iPhone SE Firmware 10.0.2)	
  • Feature testing:
- https://wiki.mozilla.org/QA/Expandable_List_of_Synced_Tabs#Date:2017-05-19 - green on Train 87
- Add a secondary email feature - exploratory testing	
  • Known bugs found on testing process
ID Summary Priority Status
1349547 Don't show Google play to install Firefox while you are in Firefox -- ASSIGNED
1351648 [fxa][Staging 83] Confirmation email not expired after an hour -- RESOLVED

2 Total; 1 Open (50%); 1 Resolved (50%); 0 Verified (0%);


  • Sign-off mail send on 2017/05/19

Date:2017-06-07

  • Started End to End testing suite Train 88 - Staging Server on Windows, iOS and Android: https://goo.gl/ijDsTu
  • Platforms used:
- Windows 10 (64)
- Android: (Samsung Galaxy S6 EDGE - (Android 6.0),  Nexus 9 (Android N)
- iOS (iPhone SE Firmware 10.0.2)
  • Feature SignOff:
- https://wiki.mozilla.org/QA/Expandable_List_of_Synced_Tabs - pre_Release
- Add a secondary email feature - exploratory testing
  • Known bugs found on testing process
ID Summary Priority Status
1349547 Don't show Google play to install Firefox while you are in Firefox -- ASSIGNED
1351648 [fxa][Staging 83] Confirmation email not expired after an hour -- RESOLVED

2 Total; 1 Open (50%); 1 Resolved (50%); 0 Verified (0%);


  • Sign-off mail send on 2017/06/08

Date:2017-06-15

  • Started End to End testing suite Train 89 - Staging Server on Windows, iOS and Android: https://goo.gl/lrb0eZ
  • Platforms used:
- Windows 10 (64)
- Android: (Samsung Galaxy S6 EDGE (Android 7.0),  Nexus 9 (Android N)
- iOS (iPhone SE Firmware 10.3.3)
  • New bugs found on testing process
    • bug 1373224 - [Train 89][Staging server only] User can't access log-in form from synced devices section on Android - HIGH IMPACT
    • bug 1373233 - [Train 89]"Back" button from creating account process is misleading
    • bug 1373616 - [Train 89][Staging server only] "Unexpected error" message displayed in Communications preferences
  • Known bugs found on testing process
ID Summary Priority Status
1349147 Crash in java.lang.IllegalStateException: stateLabelString and stateString must not be null, but: (stateLabelString == null) = true and (stateString == null) = true at org.mozilla.gecko.fxa.authenticator.AndroidFxAccount.getState(AndroidFxAccount.java) P2 NEW
1349547 Don't show Google play to install Firefox while you are in Firefox -- ASSIGNED

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);


  • Partial Sign-off:yellow.
  • Sign-off mail send on 2017/06/16

Date:2017-07-07

  • Started End to End testing suite Train 90 - Staging Server on Windows, iOS and Android: https://goo.gl/cXXB7q
  • Platforms used:
- Windows 10 (64)
- Android: (Samsung Galaxy S6 EDGE (Android N),  Asus ZenPad 8 (Android 6.0.1)
- iOS iPad Air 2 (Firmware 10.3.2 - 14F89)
  • New bug found on testing process - qablocker (still an issue for Android O)
    • bug 1377838 - [Train 90] Sync doesn't work on the staging server
  • Known bugs found on testing process
ID Summary Priority Status
1349547 Don't show Google play to install Firefox while you are in Firefox -- ASSIGNED

1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);


  • Sign-off mail send on 2017/07/10: STATUS: GREEN

Date:2017-07-14

  • Started End to End testing suite Train 91 - Staging Server on Windows, iOS and Android: https://goo.gl/ATn7jw
  • Platforms used:
- Windows 10 (64)
- Android: (Motorola Nexus 6 (Android N),  Asus ZenPad 8 (Android 6.0.1)
- iOS (iphone 7 ios 10.3.2)
  • Known bugs found on testing process
ID Summary Priority Status
1349547 Don't show Google play to install Firefox while you are in Firefox -- ASSIGNED

1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);


  • Sign-off mail send on 2017/07/19: STATUS: GREEN

Date:2017-07-31

  • Started End to End testing suite Train 92 - Staging Server on Windows, iOS and Android: https://goo.gl/uoPgHf
  • Platforms used:
- Windows 10 (64)
- Android (Nexus 6P - Android 8.0 , Motorola Nexus 6 (Android 7.0) si Lg G4 (Android 6)
- iOS (iPhone SE Firmware 10.0.2)
  • Known bugs found on testing process
ID Summary Priority Status
1349547 Don't show Google play to install Firefox while you are in Firefox -- ASSIGNED

1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);


  • Sign-off mail send on 2017/07/31: STATUS: GREEN

Date:2017-08-10

  • Started End to End testing suite Train 93 - Staging Server on Windows, iOS and Android: https://goo.gl/xADFMb
  • Platforms used:
- Windows 10 (64)
- Android (Nexus 6P - Android 8.0 , Motorola Nexus 6 (Android 7.0) , Samsung Galaxy S6 EDGE (Android 7.0)
- iOS (iPhone SE Firmware 10.0.2)
  • Known bugs found on testing process
ID Summary Priority Status
1349547 Don't show Google play to install Firefox while you are in Firefox -- ASSIGNED

1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);


  • Sign-off mail send on 2017/08/11:STATUS: GREEN

Date:2017-09-08

  • Started End to End testing suite Train 95 - Staging Server on Windows, iOS and Android: https://goo.gl/ueGRKo
  • Platforms used:
- Windows 10 (64)
- Android (Samsung Galaxy Note 4 (Android 5.0.1), Asus ZenPad 8 (Android 6.0.1), Samsung Galaxy S6 EDGE (Android 7.0)
- iOS (iPhone SE Firmware 10.0.2)
  • Known bugs found on testing process
ID Summary Priority Status
1338165 Add search engine option is being displayed in email/username fields -- NEW
1349547 Don't show Google play to install Firefox while you are in Firefox -- ASSIGNED

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);


  • Sign-off mail to be send on 2017/09/11

Date:2017-09-22

  • Started End to End testing suite Train 96 - Staging Server on Windows, iOS and Android: https://goo.gl/dGw2hV
  • Since we're in the merging phase, the testing was performed pointing to the Staging server (as usual) but now we tested using the RC build (56.0 build 1) and the latest beta build (57.0b3)
  • Platforms used:
- Windows 10 (64)
- Android Android (Samsung Galaxy Tab S3 (Android 7.0) , Samsung Galaxy S6 EDGE (Android 7.0), Sony Xperia Z5 (Android 6.0.1), Motorola Nexus 6 (Android 7.1.1), Huawei MediaPad M2 (Android 5.1.1)  and Asus ZenPad 8 (Android 6.0.1)
- iOS (iPhone SE Firmware 10.0.2)
  • Known bugs found on testing process
ID Summary Priority Status
1338165 Add search engine option is being displayed in email/username fields -- NEW
1349547 Don't show Google play to install Firefox while you are in Firefox -- ASSIGNED

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);


FEATURES:

  • Change Email Address on Account - Status YELLOW
    • bug 1403534 - Primary email not changed in settings on Android devices
    • bug 1403524 - Primary email changed on desktop not displayed on mobile
  • [NINU]These two issues were encountered on both Stage and prod server, on a few devices. More investigating will be performed the next days by Sorina to determine exactly why this happens since I was not able to reproduce by following the same repro steps.
  • Comparing local device vs server clocks in order to determine if collection has new data available - Status Green on both Nightly and Beta (57.0b3) - tested on the Prod server

Date:2017-10-05

  • Started End to End testing suite Train 97 - Staging Server on Windows, iOS and Android: https://goo.gl/zdw5FB
  • Sign off on Manual Testing for the staging server to track the release of train 97 and details are available below:
    • MANUAL TESTING PROGRESS: DONE
    • OVERALL STATUS: GREEN
    • MANUAL QA'S RECOMMENDATION: Ship It
  • No new issues logged during this test session.
  • MANUAL TESTING SUMMARY:
  • Platforms used:
- Windows 10 (64)
- Android (Sony Xperia Z5 (Android 6.0.1), LG (Android 7.0), Nexus 6P (Android 8.0.0) and Pixel C (Android 7.1.1))
- iOS (iPhone S Firmware 10.3.3)
  • Known bugs found on testing process
ID Summary Priority Status
1338165 Add search engine option is being displayed in email/username fields -- NEW
1349547 Don't show Google play to install Firefox while you are in Firefox -- ASSIGNED

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);


FEATURES:

  • Change Email Address on Account - Status Green (In Beta - 57.0b5)