Summary

This page documents the test strategy of verifying Firefox Hello is working end-to-end.

Meetings

Discussions of end-to-end test coverage are on-going in the weekly Loop QA meeting, Thursdays at 13:00 Pacific. As this meeting may involve partner specific information it is not always publicly accessible. Please contact Anthony Hughes to request an invitation.

Coverage

Desktop Client

Owner: Anthony Hughes

Testing specific to server deployments is handled by Services QA. On the client-side we do daily manual testing and bug triage which should be sufficient in detecting serious regressions caused by faulty server deployments. Additionally, we assist the Services QA team with testing when requested.

Tests

• Moztrap: Fx34
• in-testsuite: 19 bugs have coverage
• in-qa-testsuite: 0 bugs have coverage

Outstanding

• We need to investigate automated coverage of the Standalone Page with Stephen Donner (WebQA)
• We need to investigate broadening our Loop test coverage once platform dependencies are in place

Firefox OS

Owner: Tony Chung

Firefox OS client is under the responsibility of our partner. Whenever there's an issue with the platform it is escalated to Mozilla via bugzilla.mozilla.org. In terms of end-to-end test coverage during server deployments there's not much we can do here. We'll need to monitor weekly status reports from the partner and via meetings to ensure no regressions are slipping through the cracks.

Tests

• Desktop -> Firefox OS testing is in progress as of October 20, 2014
• Flame/Fire-E device testing was completed on September 29, 2014
• Weekly status reports

Server

Owner: Richard Pappalardo

Current coverage of server deployments includes verification that what was deployed does not regress 1:1 calling, nor does it regress being able to view the Terms of Service and Privacy Policy. In addition, there is some verification that the files on the server matches what was intended to be pushed.

Server Components:

• Simplepush - see bug 1055139 for more information
• Firefox Accounts
• Scalability
• Client
• Server

Platform

Owner: Nils Ohlmeier

• Subset of WebRTC platform tests running across different OS’s, build versions, network scenarios

Automation

Owner: Nils Ohlmeier

• [ON TRACK] 1:1 calling via marionette (trello)
• Integrating platform WebRTC tests into treeherder
• Investigate getting Softvision/A-team resources to help with test development
• Investigate getting Stephen Donner's support in on-device testing
• Set up a process for report monitoring and failure escalation

Next Steps

Anthony Hughes

• document a list of what needs to be checked for each server/client/product deployment
• develop a list of what needs to be automated for each server/client/product deployment
• [DONE] clarify roles, expectations, division of labour for Firefox OS
• [DONE] create a weekly status update page for Firefox OS - Tony will continue to update weekly
• [DONE] Find a Softvision resource to assist with test development - working with Henrik on this

Nils Ohlmeier

• [DONE] Review dmose's trello dashboard to make sure it’s a complete list of work to be completed

Richard Pappalardo

• [DONE] Review dmose's trello dashboard to make sure it’s a complete list of work to be completed
• [ON TRACK] Investigate feasibility of developing a middleware test using a fake Loop client to test server APIs

Syd Polk

• [DONE] review and escalate gaps in current coverage to Anthony Hughes

Tony Chung

• [DONE] Document on-going status of Firefox OS coverage