Calendar:QA Chat:2008-01-10:Log

[18:32] <mschroeder> TADA... it's QA Chat time!
[18:33] <mschroeder> Our agenda for today's QA Chat: http://wiki.mozilla.org/Calendar:QA_Chat:2008-01-10
[18:35] <nth10sd> mschroeder: what's eggplant? (and the agenda looks similar to the prev week? (which had only u and Andrea)
[18:36] <mschroeder> nth10sd: Eggplant is software for automated UI testing (on Mac).
[18:36] <nth10sd> i see
[18:36] <mschroeder> Andreas: ping
[18:36] <ctalbert> Eggplant is actually cross platform, but it is closed source and requires a license.
[18:37] <Andreas> mschroeder: pong
[18:37] <ctalbert> thetux: you're welcome.  That's probably one of the worst bits of UI in the entire bugzilla product.
[18:37] <mschroeder> Andreas: just wanted to check you're here for the QA chat.
[18:37] <mschroeder> ;)
[18:37] <ctalbert> but unfortunately a very important one
[18:38] <Andreas> Yes, i'm back.
[18:38] <nth10sd> mschroeder: ulf's leaving calendar?
[18:38] <mschroeder> nth10sd: Has already left.
[18:38] <nth10sd> mschroeder: i see. that's one less.
[18:39] <mschroeder> First topic: Test day today!!!
[18:39] <mschroeder> We had _no_ Litmus tests today until now.
[18:40] <mschroeder> So, if you have time to run especially the drag&drop-conversion tests, do it. :)
[18:42] <Andreas> It would be great someone with a mac could run this testcases.
[18:42] <skuribay> I'm in d-d testcase...
[18:42] <mschroeder> skuribay: perfect
[18:44] <mschroeder> Next topic: Refurbishing Litmus testcases
[18:45] <mschroeder> I'll try to contact damian again. Hopefully he'll have some spare time.
[18:46] <mschroeder> If anyone is interested to help in cleaning up and reworking our existing Litmus testcases, feel free to raise your hand. :)
[18:46] <Andreas> I will try to rework some of the litmustest written by Ulf and me.
[18:47] <mschroeder> That's a good start.
[18:48] <thetux> i would help, but i'm not sure if i'm the right person for this, since i just have an litmus account for a few minutes...
[18:49] <mschroeder> thetux: Try do run some Litmus testcases from Calendar group. If you find a testcase that isn't exercisable anymore (e.g. because a dialog or menu changed), add a comment, so we can find it-
[18:50] <thetux> ok :)
[18:51] <mschroeder> thetux: That already helps a lot.
[18:51] <mschroeder> Next topic: QA sessions: qawanted, qa discussion needed, verifying fixed bugs
[18:52] <mschroeder> I think we should concentrate on 0.8 testing and verifying fixed bugs at the moment.
[18:52] <ctalbert> And ad-hoc testing too
[18:52] <ctalbert> oh wait you said that
[18:53] <ctalbert> +1 :-)
[18:53] <mschroeder> Andreas does a good job verifying bugs. :)
[18:55] <mschroeder> Next topic: AI: "in-testsuite" flag for Calendar component in Bugzilla (mschroeder)
[18:56] * mschroeder talked to daniel, and I'll file a bug after the QA chat.
[18:57] <mschroeder> Next topic: AI: Enable unit tests on tinderboxen (mschroeder)
[18:57] <mschroeder> I had no time to make any progress.
[18:58] <Sebo> isn't that pending review?
[18:59] <mschroeder> Sebo: no, I updated the Makefile on Trunk, but enabling tests on our production tinderboxen is complicated.
[19:00] <Sebo> ah, I thought the Makefile will do the trick...
[19:01] <mschroeder> no, I have to talk to robcee, who should know what to do next
[19:02] <mschroeder> Next topic: Test Automation Sumit
[19:02] <thetux> short question: When the steps to perform in litmus are not right, where to write this down?
[19:02] <mschroeder> thetux: mark it unclear and add a comment
[19:03] <thetux> k
[19:04] <mschroeder> The effort has stalled because ctalbert and I have too much on our plates at the moment.
[19:04] <ctalbert> mschroeder: well, it's a little different than that
[19:05] <ctalbert> I talked with the folks in the Mozilla Automation team about the idea, and they are all for it.
[19:05] <mschroeder> sorry
[19:05] <ctalbert> that's ok
[19:05] <ctalbert> I need to blog abotu this
[19:05] <ctalbert> And what I'm starting to think is that it would be more beneficial to every project, if we did a Mozilla-wide automation summit.
[19:06] <ctalbert> And if we did that, we would have more help in putting it on than just those of us in the calendar project
[19:06] <ctalbert> We could draw several people in that are interested in thunderbird, firefox, camino, seamonkey etc
[19:06] * mschroeder agrees
[19:06] <ctalbert> And we could do more with more of the automation tools that mozilla offers if we do it that way too - we could cover the reftests, litmus, QAX, mochikit, XPCShell etc
[19:07] <ctalbert> So, I think the original idea has changed slightly - it has gotten bigger.  But, it's still on the table.  I'm just not exactly sure when we're going to actually *do* it.  If we hire someone that we just interviewed, then he and I will probably make it our task to spearhead this effort and get it done.
[19:08] <mschroeder> cool
[19:08] <ctalbert> I'll post something to the calendar blog about this to update folks.
[19:08] <mschroeder> ctalbert: Thanks for the update. :)
[19:09] <ctalbert> Sure, np
[19:09] <mschroeder> Last topic: Automated UI testing: Eggplant & ...
[19:10] <Sebo> Is firefox doing UI testing?
[19:11] <mschroeder> I talked to celina63 last week, and she had used some tool from Hewlett-Packard at work. She also gave my a link to some tools, but I haven't look at them.
[19:11] <ctalbert> Sebo: yes.
[19:11] <mschroeder> Sebo: I think, they use Eggplant for Firefox testing.
[19:12] <ctalbert> Firefox uses Eggplant and has an entire team dedicated to UI testing.
[19:12] <Sebo> wow
[19:12] <ctalbert> You've probably heard a mention of the "QA Execution Team" -- that's who they are
[19:13] <Sebo> but we would need a license?
[19:13] <ctalbert> Since Eggplant is closed source, it's a bit difficult for the Mozilla license to be used by community projects (because MoCo bought a certain number of seats for that license).
[19:14] <ctalbert> There has been a lot of talk about this, and people are trying to find a way to do that, though.  Tracy probably knows where that effort is at.
[19:14] <ctalbert> Sebo: yes, is the short answer :-)
[19:14] <Sebo> hehe
[19:14] <Sebo> thanks
[19:14] <ctalbert> If we could find some other tool that could work for us, that'd be awesome.
[19:15] <ctalbert> Many of them are OS specific, but if we could even get coverage on one OS with a tool, that would be OK since it would free us up to look at other OS's.
[19:16] <Sebo> what sort of things would you actually test that we can't test otherwise?
[19:18] <ctalbert> GUI automation would test stuff that you could test by hand.
[19:18] <ctalbert> In fact, it will only find glaring errors.
[19:18] <mschroeder> The Calendar views, menus and dialogs.
[19:18] <ctalbert> ususally
[19:18] <ctalbert> But, it will be able to run through all that stuff a lot faster than you could by hand.
[19:19] <ctalbert> And that's why it makes a good "smoketest" tool.  Especially useful for l10N testing.
[19:19] <ctalbert> When we have to go through all the builds for all the locales right before release.
[19:19] <Sebo> yeah, l10n is a good argument
[19:19] <ctalbert> that's probably where it is most beneficial.
[19:19] <Sebo> regarding the rest I would say - we don't have so much to cover that its worth it
[19:19] <ctalbert> but if the UI is funny-looking and unusable, the GUI automation tool won't capture that.
[19:20] <ctalbert> Only a human can do that.
[19:20] <ctalbert> Sebo: you are probably right
[19:22] <mschroeder> Okay, any question, suggestions etc.?
[19:23] <mschroeder> You can set the in-testsuite flag now, if you find a bug that needs a unit test.
[19:24] <mschroeder> reed created it for me one minutes after filing the bug. :)
[19:24] <ctalbert> he tends to do that. 
[19:25] <mschroeder> That's the end of the QA chat, thanks for participating.