|
|
| Line 1: |
Line 1: |
| = Beta = | | = Beta = |
| * [https://moztrap.mozilla.org/runtests/environment/5626/ Smoketests] | | * [https://moztrap.mozilla.org/runtests/environment/5626/ Smoketests] |
|
| |
| = Aurora =
| |
| == Checklist ==
| |
| * [daily] Review dogfooding bugs to escalate new critical concerns and flag fixed bugs for verification
| |
| * [daily] Review uplift bugs to flag fixed bugs for verification
| |
| * [daily] Review results of Moztrap testruns for potential bugs
| |
| * Create a smoketest run for community testing
| |
| * Create a regression testrun for release sign-off
| |
| * Create a report of individual component readiness for Beta
| |
|
| |
| == Pre-uplift Sign-offs ==
| |
| The following bugs have patches landed on the [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/rjesup@wgate.com-f9eb2cbac352 Fig branch] for testing prior to uplift to Aurora.
| |
| * bugs with ''[fig:verifyme]'' in the whiteboard will be verified before uplift
| |
| * bugs with ''[fig:wontverify]'' will not be verified before uplift by may still be verified after uplift
| |
| * bugs with ''[fig:verified]'' in the whiteboard have been tested and verified
| |
| There should be 0 bugs with ''[fig:verifyme]'' in the whiteboard before the uplift.
| |
| <bugzilla>
| |
| {
| |
| "id":"1073047,1073027,1072279,1071835,1070065,1070045,1069965,1069962,1069178,1068580,1068178,1067937,1067845,1067519,1066816,1066609,1066509,1066506,1066502,1066219,1065777,1065608,1065591,1065275,1065155,1065153,1065144,1065052,1062835,1062126,1061756,1060809,1059754,1059186,1055567,1055239,1055139,1053774,1049565,1048938,1047667,1047181,1047146,1045690,1044411,1042060,1035655,1035846,1035348,1032700,1017257,1008990,1002416,1000766,1000237,1000112,974873",
| |
| "f1":"flagtypes.name",
| |
| "o1":"equals",
| |
| "v1":"qe-verify+",
| |
| "include_fields":"id,summary,qa_contact,whiteboard,status"
| |
| }
| |
| </bugzilla>
| |
|
| |
| == Uplifts ==
| |
| === Pending Triage ===
| |
| <bugzilla>
| |
| {
| |
| "include_fields":"id,summary,status",
| |
| "product":"Loop",
| |
| "f1":"status_whiteboard",
| |
| "o1":"substring",
| |
| "v1":"loop-uplift",
| |
| "f2":"flagtypes.name",
| |
| "o2":"notequals",
| |
| "v2":"qe-verify+",
| |
| "f3":"flagtypes.name",
| |
| "o3":"notequals",
| |
| "v3":"qe-verify-"
| |
| }
| |
| </bugzilla>
| |
|
| |
| === Flagged for Verification ===
| |
| <bugzilla>
| |
| {
| |
| "include_fields":"id,summary,status,cf_status_firefox35,cf_status_firefox34",
| |
| "product":"Loop",
| |
| "f1":"status_whiteboard",
| |
| "o1":"substring",
| |
| "v1":"[loop-uplift]",
| |
| "f2":"flagtypes.name",
| |
| "o2":"equals",
| |
| "v2":"qe-verify+"
| |
| }
| |
| </bugzilla>
| |
|
| |
| == Dogfooding ==
| |
| Once the final uplifts occur on Aurora we'll start having Mozillians use Firefox Hello for their 1:1 meetings. Any bugs reported will be listed below.
| |
| <bugzilla>
| |
| {
| |
| "include_fields":"id,summary,status,last_change_time",
| |
| "product":"Loop",
| |
| "chfield":"[Bug creation]",
| |
| "chfieldto":"Now",
| |
| "chfieldfrom":"2014-10-06"
| |
| }
| |
| </bugzilla>
| |
|
| |
| == Smoketesting ==
| |
| We need a Moztrap testrun for community to run, including the following tests:
| |
| * Verify you can view ToS/Privacy/Support docs
| |
| * Make a call in guest mode via copy
| |
| * Make a call in guest mode via email
| |
| * Make a voice only call
| |
| * Pop-out the conversation window during a call
| |
| * Mute/unmute audio/video during the call
| |
| * Unshare/reshare your devices during a call
| |
| * End a call and submit feedback
| |
| * Register a Firefox Account and sign in
| |
| * Manage your contacts
| |
| * Call someone in your contacts
| |
|
| |
| Criteria:
| |
| * Call host should be using Firefox 34 on any of our Desktop platforms
| |
| * Caller should be using Firefox or Chrome on any of our Desktop platforms
| |
| * Tester should report the desktop platforms, browsers, and network environments of both host and caller when reporting a test failure
| |
|
| |
| == Regression Testing ==
| |
| === Guest Mode Tests ===
| |
| * Conduct a 1:1 call via Copy
| |
| * Conduct a 1:1 call via E-mail with Thunderbird as the default email handler
| |
| * Conduct a 1:1 call via E-mail with Outlook as the default email handler
| |
| * Conduct a 1:1 call via E-mail with Mail.app as the default email handler
| |
| * Conduct a 1:1 call via E-mail with Gmail as the default email handler
| |
| * Conduct a 1:1 call via E-mail with Yahoo Mail as the default email handler
| |
| * Verify entering do-not-disturb mode changes the button appearance and prevents receiving a call
| |
| * Verify entering available mode changes the button appearance and resumes the ability to receive a call
| |
| * Verify the conversation window can be popped out and in without affecting the call
| |
| * Verify the conversation window can be minimized and maximized without affecting the call
| |
| * Verify you can view the Terms of Service via the link in the panel on first run
| |
| * Verify you can view the Privacy Policy via the link in the panel on first run
| |
| * Verify you can view the Support documentation via a link
| |
| * Verify you can sign in to a Firefox Account via the link and/or the gear menu, username should appear in the panel and the icon should turn blue momentarily
| |
| * Verify you can view your account information via the gear icon
| |
| * Verify you can sign out of your account via the gear icon
| |
| * Verify navigating away from the call page should be treated as a disconnect
| |
| * Verify you can submit various types of feedback when the call ends
| |
| * Verify you see an error notification when the network is disconnected
| |
| * Verify you can generate a call URL and have a call when the network is reconnected
| |
| * Move the button into the customize panel and make sure the panel renders properly and is functional
| |
| * Verify notifications of various failure states
| |
| * Try conducting a call in private browsing mode
| |
|
| |
| === Standalone Page Tests ===
| |
| * Loading a call URL using an unsupported browser (for example Internet Explorer and Safari) should show an "Incompatible Browser" page with a link to download Firefox
| |
| * Loading a call URL using a supported browser should load the call page and check the following:
| |
| ** The URL displayed on page should match the URL you loaded
| |
| ** You should be able to initiate a video call
| |
| ** You should be able to initiate a voice-only call
| |
| ** You should be able to mute and unmute the video during the call
| |
| ** You should be able to mute and unmute the audio during the call
| |
| ** You should be able to submit feedback about the call once the call is terminated
| |
| ** If the build is localized, so too should the page
| |
| * Trying to start a call for an invalid call URL like https://call.mozilla.com/#call/invalid results in an error page
| |
| * Verify notifications of various failure states
| |
| * Try initiating a call in private browsing mode
| |
| * Verify you can view the Privacy/Support/ToS documentation via a link
| |
|
| |
| === Authenticated Mode Tests ===
| |
|
| |
| === Platform Coverage ===
| |
| ==== Call Hosts ====
| |
| * Firefox 34 en-US on Windows XP on ethernet
| |
| * Firefox 34 de on Windows Vista on wifi
| |
| * Firefox 34 es-ES on Windows 7 on ethernet
| |
| * Firefox 34 zh-TW on Windows 8 on wifi
| |
| * Firefox 34 ru on Windows 10 on ethernet
| |
| * Firefox 34 en-US on Mac OS X 10.9 on wifi
| |
| * Firefox 34 ja-JP on Mac OS X 10.10 on ethernet
| |
| * Firefox 34 en-US on Ubuntu 14.04 on wifi
| |
| * Firefox 34 he on Ubuntu 12.04 on ethernet
| |
|
| |
| ==== Callers ====
| |
| Firefox
| |
| * Firefox 32 en-US on Windows XP on ethernet
| |
| * Firefox 33 de on Windows 8 on wifi
| |
| * Firefox 34 ja-JP Mac OS X 10.9 on ethernet
| |
| * Firefox 35 zh-TW Ubuntu 14.04 on wifi
| |
| * Firefox 32 en-US on Android 4.4 on 3g
| |
| * Firefox 35 en-US on Android 4.3 on wifi
| |
| * B2G 2.0 en-US on wifi
| |
| * B2G 2.0 en-US on 3g
| |
|
| |
| Chrome
| |
| * Windows XP on ethernet
| |
| * Windows 7 on wifi
| |
| * Mac OS X 10.7 on ethernet
| |
| * Ubuntu 14.04 on wifi
| |
| * Android 4.4 on wifi
| |
| * Android 4.4 on 3g
| |
|
| |
| Internet Explorer
| |
| * Windows XP on WiFi
| |
| * Windows Vista on Ethernet
| |
| * Windows 8.1 on WiFi
| |
| * Windows 10 Preview on Ethernet
| |
|
| |
| Safari
| |
| * Mac OS X 10.9 on WiFi
| |
| * Mac OS X 10.10 on Ethernet
| |
| * iOS 8 on 3g
| |
|
| |
| Opera
| |
| * Windows 7 on ethernet
| |
| * Windows 8.1 on wifi
| |
| * Mac OS X 10.8 on ethernet
| |
| * Ubuntu 14.04 on wifi
| |
| * Android 4.4 on 3g
| |
|
| |
| Android Browser
| |
| * Android 4.4 on wifi
| |
| * Android 4.4 on 3g
| |
|
| |
| == Exploratory Testing ==
| |
| The following should be tested on all platforms (Windows, Mac, and Linux):
| |
| * Try crashing the browser during a call and see if you can restart the call using the same URL
| |
| * Try moving the call tab to another browser window
| |
| * Try moving the call window to another screen
| |
| * Try changing the screen resolution during a call
| |
| * Try switching between speakers and headphones during a call
| |
| * Try switching between an integrated camera to a USB camera during a call
| |
| * Try testing in networks with different firewall applications installed
| |
| * Test primary use cases in e10s mode
| |
| * Test primary use cases in win64 builds
| |
|
| |
| == End-to-End Automation ==
| |
| * {{ok|}} Automated test is running locally (2014-09-29: ewong)
| |
|
| |
| == Bug Triage ==
| |
| * Need to verify all high-risk fixes are flagged for verification
| |
| * Need to verify all flagged fixes are verified
| |
| * Need to verify that there are no blocking unresolved bugs remaining
| |
|
| |
| == Health ==
| |
| * Desktop
| |
| * Mobile
| |
| * Platform
| |
| * Automation
| |
| * Server
| |
| * Services
| |