Calendar:QA Chat 2006-06-29:ircLog
From MozillaWiki
#calendar-qa <ctalbert> Howdy folks <emose> Good morning <ctalbert> Are we waiting for anyone? <jminta> is there an agenda anywhere i can add an item to? <ctalbert> s /anyone/anyone else <ctalbert> http://wiki.mozilla.org/Calendar:QA_Chats <emose> I talked to Marcia yesterday, and she said that she wanted to try and make this meeting, but that she had some sort of conflict, so it wasn't entirely clear she would be able to <ctalbert> I talked with her as well. She's attending a conference today I believe. She said that she'd like to attend a future QA chat, if she couldn't make it today. <ctalbert> From the agenda on Calendar:QA_Chats, anyone else want to add anything? I see jminta's addition -->| jmacminta (chatzilla@moz-46636307.office.mozilla.org) has joined #calendar-qa <emose> It's unfortunate that neither sipaq nor ssitter is here -->| refu (chatzilla@moz-87B74317.adsl.alicedsl.de) has joined #calendar-qa <ctalbert> Can anyone ping either sipaq or ssitter? <jminta> i'll send sipaq a wuick email, but that's all i think we can do <jminta> quick* <ulf> has anyone spoke to davel? <jminta> yeah, he didn't think he could make it either, but he gave me a run-down of unit-testing bullet points <ctalbert> Nice <ulf> interesting -->| ssitter (chatzilla@moz-AAC453E0.dip0.t-ipconnect.de) has joined #calendar-qa <emose> woot! <ctalbert> Great. Let's go ahead and get started <ssitter> hi folks <ctalbert> I think we all agreed that the goals as stated in Ulf's Newsgroup post are the right way to go. <ctalbert> So, it seems the first thing to discuss is how to start the process of structuring the calendar QA effort <ulf> agreed <ctalbert> I had a couple of ideas on the agenda (http://wiki.mozilla.org/Calendar:QA_Chats) <ctalbert> testcase creation, investigating regression/unit test systems, are there others? <jminta>release-prep? <--| refu has left #calendar-qa -->| ssa (ssa@moz-4811E34E.staroffice.de) has joined #calendar-qa <ctalbert> You mean getting ready to QA the 0.3 release? Or defining some more general procedure? <jminta> right <jminta> there are some tests that we want to run more of less continuously <jminta> and then there's more rigorous testing before certifying a release <ulf> can you name a few of those tests? <jminta> well, things like regression testing are in the "constant" category <jminta> things like website-link checks, download integrity, etc are really only necessary before releases <jminta> branding checks too would be in that category <lilmatt> Yup <ulf> I like the idea of test case categories <ulf> how do others feel about this? <jminta> can you elaborate on what you mean by "categories"? <ulf> well, what you proposed; have categories for e.g. continously tests, some perhaps in a 2 weeks cycle and others right before a release <ctalbert> It sounds like creating release-prep procedures can be largely addressed by formalizing (categorizing, prioritizing, scheduling, tracking, etc) the testcases. <jminta> dmose says "that seems slightly towards the side of over-managing" <jminta> "making a document outlining release procedures might be sufficient there" <ctalbert> It might be. <ulf> ok <ctalbert> I'd be all for that too <jminta> i'm also a bit concerned about getting stuck into a framework of "this is what we do for QA and nothing more" <ctalbert> I don't want to ever do that. <jminta> and our builds will pass those tests, but fail in the real world <ssitter> make a what-to-do-for-release-checklist <jminta> i know the FF-team has been really fighting to make releases more repeatable <ctalbert> It's always good to have a set of testcases so that you know all the features work. <jminta> right -->| refu (chatzilla@moz-87B74317.adsl.alicedsl.de) has joined #calendar-qa <lilmatt> That sounds like litmus <ctalbert> But that does not ever subsititute for just creative testing by the QA folks, hunting for edge cases and bugs. <jminta> right <ctalbert> We will be putting our testcases in litmus <jminta> dmose suggests "go ahead and categorize/test-case, just don't work about prioritizing and scheduling as much" <ctalbert> The main reason to create and formalize the testcases to some degree is to have a place where we can point new people who are getting involved, so that they have a concrete place to get started. <jminta> ok, i like that <ctalbert> And so that we can measure each build against some defined yardstick, which are those testcases. <jminta> dmose agrees <ctalbert> I agree with dmose as well. <ulf> @ctalbert: what needs to be done to get testcases into litmus? <ctalbert> We have to put them into a standard XML format so that coop can run a perl script on them and import them into the Litmus database. <ctalbert> Creating testcases in litmus is still a manual process, at the moment. <ulf> ok, thanks |<-- refu has left irc.mozilla.org (Quit: ChatZilla 0.9.67+ [SeaMonkey 1.0.2/2006051612]) <jminta>i talked with ispiked and he said there is now a GUI for creating some tests <ctalbert>Awesome! <ulf> cool <ctalbert> I looked at it this morning, and I was noticing some changes since I last checked it out... <jminta> (you need to have sufficient privs to see the GUI) <ctalbert> Ah <ctalbert> I saw they fixed one of my major complaints with it...but back to the agenda... <ulf>no problem, let's stay at litmus for a while <ctalbert> Well, the bug that I wanted fixed on it was the ability to search for testcases. It makes the system far more user friendly. <ctalbert>And that is now completed. <ctalbert>What else about litmus? <ulf>may be I didn't tried hard enough, but is there any docu available to get started an input our test cases? <ssitter>are the testcases stored in cvs so it is easy to edit/add new testcases that are automatically send to litmus? <ctalbert>http://wiki.mozilla.org/Litmus:Adding_Testcases_to_Litmus Was the page that coop had directed me to. It now says that it is "out of date" <ulf> thx -->| ispiked (ispiked@moz-1EFCA1F7.trilug.org) has joined #calendar-qa <jminta> dmose says "the page does give a link to the right way to do it" <jminta> ispiked: we're trying to figure out what needs to be done for us to have our own testcases in litmus <ctalbert> It sure does <ispiked> jminta: you probably need to ask zach to add calednar as a product, then ask to get admin access so you can add testgropus and testcases, etc. <jminta> ok <ispiked> jminta: right now, adminstration for adding testgroups isn't too great, so you'd probably have to ask either zach or coop to do that. <jminta>and there's a GUI now, so we don't need to fuss with XML for the actual tests? <ispiked>jminta: right. <ulf>who is willing to take the action item? <jminta>zach says "EOD is a reasonable timeframe for gettins Calendar set up in litmus" <ctalbert>Great! <ctalbert>I can do that (help get calendar set up in Litmus) <ulf> nice, thank you! <ctalbert>np <ctalbert>What about the investigation into regression/unit test systems for the calendar projects? <ulf> talking about test cases, what areas people think should be addressed first? <jminta> that was what i added to the agenda <jminta> i have an xpcshell test for ics-roundtripping <jminta> that can be made to run for arbitrary VEVENTs <jminta> (since dataloss seems to be one of our priorities) <ctalbert> Its a great place to get started <jminta> davel has asked that all unit-tests (from all products) be sent to him, so he can abstract common elements and come up with a reasonable framework <jminta>he also said that fundamentally, this needs to begin with the developers <jminta> they need to write tests for the code they touch * ctalbert agrees <ulf> what do others think about this suggestion? <emose> I agree also, however <ulf> is that feasable? <jminta> emose says "i think we need to have some framework in place to point devs at before they get started" <ulf> e.g. write a unit test for every new (bigger) feature? <ssitter> if we have a tool like litmus set up it could be handled like fix the bug + create test case <ulf> ahh, great <jminta> emose says "there are 2 pieces here, both bugs and features" <ssitter> automatically tests is another topic as dmose said <lilmatt> Well, we could start by enabling things like codesighs and Ts on the tbxn <jminta> "bugs map easily to litmus, features are more complicated" <jminta> emose suggests "we should worry about bugs first, since they're easier" <ulf> are you referring to GUI test, when speaking of automated tests? <jminta> emose says "and then, once we're more familiar with litmus, we can address the feature case" <ispiked> jminta: am I free to leave now? ;) <jminta> ispiked: yeah, thanks! <ctalbert> thanks ispiked <lilmatt> thanks ispsiked <ispiked> no problem. <--| ispiked has left #calendar-qa <ssitter> not only ui tests <ssitter> there could be automated (js) tests for backend, ... <jminta> emose says "let's start with the low-hanging fruit, and that's not automated GUI testing" <ulf> right! <jminta> i agree with him, let's use Litmus for the UI, and xpcshell for the backend * lilmatt puts down his "adding Automator actions to your app" book <ctalbert> we need to start with things that will give us the most return for our time, I think jminta's on the right track <ctalbert> Since we're talking about it, jminta: is your xpcshell test for dataloss checked into calendar/test <jminta> no <ctalbert> Could we get you to do that? calendar/test is not a built directory last time I checked, so it would be pretty safe. <ssitter> i have never used xpcshell, does this can do something like: run testxyz ... passed! <ctalbert> ssitter: It does sort of work like that. <emose> XPCshell is a good thing to use as a core for our testing framework but it is not a test driver <jminta> yeah, through use of dump() <ctalbert> xpcshell is a way to run javascript outside the application. You can create the XPCOM interfaces and perform functions with them and then report the results <emose> which we also need <ctalbert> I don't want to run out of time, so allow me to try to sum up where we're at (briefly) -->| zach (zach@moz-46636307.office.mozilla.org) has joined #calendar-qa <zach> hi folks <ctalbert> We'll get started with the testcase creation, ctalbert will look into getting us into litmus. Davel is looking into unit tests/regression tests for the entire suite. We'll put together a release-procedure checklist and we will start the testcase creation around the dataloss scenarios that we currently are working on. <ctalbert> Does that kind of sum up where we are so far? <zach> ctalbert: the looking into getting you into litmus step just got somewhat easier :-) <ulf> yeah, agreed <ctalbert> zach: thanks <ssitter> sounds good <ctalbert> I want to move to the next item. <ctalbert> jminta asked me yesterday how we should go about alerting the calendar QA folks when a new feature is landing. <jminta> not even a new feature, but just a substantial change <ulf> very good point <ctalbert> I spoke with some people on #qa about it <ctalbert> Essentially, they use the general qa@mozilla.org email list to alert the entire team, and offered to allow us to use the same list. They also make use of the newsgroup, and they keep a close watch on the bugzilla states of the bugs that are slated as blocking whatever release they're interested in. <jminta> yeah, they have the fixed1.8.1 keyword <ctalbert> Exactly <jminta> and that gets replaced with the verified1.8.1 keyword when QA has looked at it <ulf> great <ctalbert> We could do the same <ulf> sounds good to me, what do others think about it? <ctalbert> Since we're still fairly small, I don't know if we want to add ourselves to qa@mozilla.org. That might just cause more headaches than its worth because of filtering through the emails <ctalbert> s /emails/emails for the other products <lilmatt> prob not <zach> ctalbert: who was offering to add you to qa@mozilla.org? <ctalbert> jay had mentioned that we could email that list in the interim with calendar issues. <ctalbert> No one offered to put me on it, say tomorrow or anything <jminta> i'd suggest just using the RESOLVED and VERIFIED states, the only reason the keywords exist is because there's a 1_8 branch <emose> How about just using our newsgroup? <ctalbert> I like the idea of using m.d.a.calendar <jminta> and use the newsgroup (like emose says) for when wider publicity is needed <ctalbert> RESOLVED and VERIFIED seem far more natural to me as well <ctalbert> Anything else on that one? <ssitter> crosspost to blog, m.d.a.calendar and mozillazine should reach most testers and nightly users <ctalbert> good idea <ctalbert> And that leads us to the next item of how to build a mozilla calendar QA community <ctalbert> We need to be very public with everything we're doing. Perhaps doing some QA specific blogs on the mozillazine to widen the net <ctalbert> And of course continuing to use the newsgroup. Are there other ideas out there? <jminta> test-days? <ulf> where would you anounce those? <jminta> Calendar-blog, QA-blog, newsgroup, maybe the forum? <ulf> sounds good to me <ctalbert> Probably also mention it in m.d.quality perhaps <ctalbert> just for the first one, to get some visibility <ulf> good point <ctalbert> It is noon by my clock, and that means the hour is up. I hope I didn't drive this too fast. <jminta> nope, i think it was very productive <ctalbert> I will put this log onto the wiki QA_Chats page, just like we do for the calendar_Status meetings <ctalbert> jminta: good! I'm glad <emose> Ctalbert: I think it went well. <jminta> can I make a first QA announcement to this group now? <ctalbert> Go for it <jminta> I'd love it if I could have attached to all ics-dataloss bugs, since VEVENT testcases <jminta> so that I can get this test-group up and running meaningful data <ctalbert> So, you want the ICS attached to the corresponding dataloss bug? <jminta> yeah, just a single file with a single VEVENT showing the bug <ctalbert> Ok. <jminta> (this would include FIXED bugs) <ctalbert> Makes sense. That way we can build a regression tool <jminta> right <ctalbert> Anyone else have anything? <ssitter> at the moment dataloss is used for dataloss and for not displayed in ui - should this be a different keyword in future? * ctalbert looks to jminta and emose <ssitter> so one can differentiate? <ulf> I'd vote for a different keyword, but that's a different topic <emose> hang on <jminta> emose says "dataloss there is not just confined to actual ics files, but dataloss between the user and the computer" * jminta isn't sure what we gain from differentiating <ulf> could we put that keyword discussion to next meetings agenda? <ctalbert> ulf: sure <ssitter> np <ctalbert> Ok then, I think we're done here unless anyone else has something. Thanks everyone. <ulf> thanks guys, great session! <emose> thanks all' <--| jminta has left #calendar-qa (Ex-Chat) <--| emose has left #calendar-qa (Leaving) <ssitter> cu <--| ssitter has left #calendar-qa
