Firefox Social Integration QA Plan
Notes from Clint Talbert:
I think we are likely going to need a few different frameworks here before it is all said and done. But focusing on the smaller problem of something that is very developer friendly so that people can run it quickly to write unit tests is the right place to start.
The Test Server
We have a newer webserver that we are looking at replacing it with. This is a simpler server called mozhttpd and it is part of our wider array of mozbase test harness framework base classes. ,  This server is faster, and is more suited to general purpose web-serving and is unable to do all the configuration stuff that httpd.js can do (although we have a contributor working on those capabilities so we can retire httpd.js one day).
How do you plan to implement the communication between your server and the test itself? That's the part I kept getting hung up on. Most of our testing is quite stateless. The closest thing we have to a stateful testing is TPS (the test system for sync), which runs outside of the main automation trees and uses real servers. It sounds like this is going to have quite a bit of in-browser integration so something that runs outside of the main automation trees while necessary for QA testing, is probably a non-starter if you have to clear Firefox Team review hurdles.
We will want to wire in your test server so that the browser-chrome test framework will start it up for you and keep it running the entire duration of the test run. My team can help wire that into the harness.
The other thing I worried about is that with these tests that are essentially using the browser itself as the verification mechanism (i.e. did the button glow blue to indicate server state X?) we run a chance for a whole lot of timing related intermittent issues. It would be awesome if the tests had two modes of verification: 1. Simple Verification Mode: Captures the event that is going to make the button turn blue (this is easier to verify) 2. Complex Verification Mode: Not only does it capture the event, it also ensures that the button did in fact turn blue (this is a little harder and can be prone to causing intermittent orange tests)
If the test is data driven, then the test object can have an attribute stating the type of verification mode it will be using.
Furthermore, this will help streamline the test verification steps into a set of concrete APIs that any given test can use. And that will further simplify test authoring and debugging. (I'm totally making these up, but here are some possible examples of APIs):
- "bool VerifyGoSocialNoticeReceived(int verificationMode)"
- "bool VerifySideBarActivated(int verificationMode)"
- and so on...it would be fine for these verify APIs to write their mochitest pass/fail notice (i.e. they would do the assertion that things are ok or not), so their return value could simply indicate whether we should abort the test at that point or not.
Aborting the test
I think that any long lived test that could cause compounding failures (if we aren't in state 1 we will never make it to state 2, etc) needs to abort early. This way when people go to debugging it, it is clear where we failed and when. So an early abort system should be built into these from the start.
I think that browser chrome tests are probably the best avenue that we have available for you to use right now. Ideally, Marionette  tests would be better simply because then you could drive your test from python scripts so you could ostensibly manage your server (assuming the server can be written quickly and simply in python) and your test from the same code, handling the state checking between test and server in a simple fashion. However, Marionette is not in production yet for desktop firefox (it is for B2G) and we won't have deployed in desktop until sometime in Q3.
: https://developer.mozilla.org/en/HTTP_server_for_unit_tests : Mozbase's location in m-c: http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/ : Mozbase's canonical location: https://github.com/mozilla/mozbase : https://developer.mozilla.org/en/TPS_Tests : https://developer.mozilla.org/en/Browser_chrome_tests : https://developer.mozilla.org/en/Marionette