Thunderbird:Unit Test Ideas

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

The following is a culmination of efforts to come up with a unit test framework for Thunderbird. It is still not complete though, but please feel free to add your own opinions. I will be adding more information the next week or the following one.

Unit tests that may be relevant to Thunderbird

Crashtests

• Always the number 1 priority as they determine whether Thunderbird is actually usable for that certain function that is defined in the crashtest.
• Standard8: So from a quick glance, Firefox loads in html (and other files?) into the browser and hence it shouldn't crash. How do we apply this to mailnews? If we crash on loading an email into the preview window due to a layout issue, then that is core. Crashes within mailnews code I think can normally be covered by xpcshell/make check level tests. Maybe this needs thinking about a bit more.
• dmose: Given that Tinderbox already turns orange when the browser crashes while running one of the other frameworks, it'd be interesting to understand the motivations behind having a separate framework here.

xpcshell tests

• There is some work on this ongoing at xpcshell tests; they require no UI and are written in Javascript.
• Standard8: We mainly now just need to encourage devs to write tests.
• dmose: Agreed.

Fakeserver tests

• These tests include but are not limited to POP, IMAP, SMTP, news, etc. There is some current traction in getting some of these in place, the ones for SMTP and News are already implemented - see Standard8's blog.
• Standard8: These are currently a subset of xpcshell tests, though I expect can be used for other testing architectures. Possibly need a bit more documentation about these on the wiki.
• dmose: Right; the fake server stuff is not it's own framework; it's simply a set of functionality that is useful within various other frameworks.

Chrometests

• These tests are relevant, albeit in Thunderbird chrome instead of Firefox chrome or browser/chrome. Chrometests are basically Mochitests with chrome privileges. Similar to crashtests, I do not believe we have any Thunderbird-chrome tests in place at this point in time. Mail chrome tests are more advanced and powerful but they contain a high learning curve, as they require knowledge of the internals of Thunderbird in intricate detail. See Gristmill notes below.
• Standard8: So in Thunderbird terms, what is the difference between a chrome test and a browser chrome test?
  • dmose: My somewhat fuzzy recollection is that there may be more than two types of chrome tests, even. This probably a good thing to talk to Gavin about.
• Standard8: What support environment do you get?
  • dmose: Different for each framework, I think.
• Standard8: Would this be for testing items like the address book results window pane, popup menu contents etc?
  • dmose: Yes. Or anything that requires driving the UI in some way.
• Standard8: In Firefox terms, are these all in /browser, or are some of these tests core?
  • dmose: Some are in toolkit, I suspect, since there's a bunch of shared UI implemented there.

Sidenote: Gristmill

• Gristmill, a project that Clint and Mikael (from MoCo) are currently working on, can potentially replace manual litmus tests and possibly some browser/chrome tests (its relevance to Firefox), and is found here: http://wiki.mozilla.org/QA/TDAI/MozMillTestTool
• In demonstrations, Gristmill can allow simple functions, e.g. mouse-clicks, to be automated. An action to open up a browser window and click on the Home button, after which a new tab is added and a search in the search box is executed, all of which are automated.
• This may potentially be useful when we apply Gristmill to Thunderbird, to get it to open a Write window, type something in the subject and contents, then send it out. Upon successful sending, the fake server will then report the results.
• Gristmill makes use of ids found in the XUL files of Fx / Tb's frontend. The actions are defined in a separate javascript file that can either be manually defined (this is the current situation) or automatically generated (in the future, hopefully).
• We can then tell if a certain action causes a crash, or even an assertion, and in addition to that, fake servers can tell if the message was sent successfully.
• By randomizing the actions, which are selected from pre-defined lists of objects (e.g. Write button is found on the main UI but not in the message compose window), depending on the names of ids found in the XUL files, and letting Gristmill run continuously.
• Looking at the logs will enable us to see which exact actions led to the crash / assertion. Human intervention can verify the suspects.
  • Standard8: This sounds good, though obviously we'll have to wait a while to get it.