Calendar:QA Chat:2006-08-24 Log

celina63: everyone here and ready to go?
celina63: Hello all and welcome to the Calendar QA Chat
celina63: please turn, if you will, to the Meeting Agenda:
http://wiki.mozilla.org/Calendar:Current_QA_Chat
celina63: First order of business: Notetaker to post the chat afterwards...
any volunteers?
celina63: ok, I'll do it  :)  Next, lets discuss how Community Test Day went...
celina63: any specific discussion points anyone wants to bring up... ?
celina63: I unfortunately had a "real-job" build to test, so was unable to
participate on the actual day... did most testing yesterday.  For people who
*did* participate (and there were alot, judging by the number of tests run!)
what was the overall impression?
celina63: How can we make it smoother next time?
Sebo: Well, maybe there schould be a more specific topic next time?
*** ulf has joined #calendar-qa.
celina63: Topic for what, Sebo?
Sebo: Not like: Just test everything
ulf: hi
celina63: well, we kinda *did* want to test everything
celina63: Hi ulf!  We're having the Calendar QA Chat right now
Sebo: ok, but then maybe a list of known issues would help
ulf: sorry, I'm a bit late
celina63: and discussing what went well and what can improve during the next
Community Test Day
ssitter: once everything is tested future tests can focus on new added features
etc.
celina63: We tried to narrow the focus a bit by stating the 3 things that
needed to be done on the TestDay wiki... Bugzilla bugs (and a link to the bugs
to be tested), Litmus tests (with a link to the Calendar tests), and then
instructions for Ad-hoc/Useablilty tests
celina63: could those sections have been phrased better?
Sebo: No, it was clear...
rogerk: Perhaps the prizes for best testers could be upped a bit, to attract
even more testers? Don't you think it would be possible to find sponsors if you
made it easy to contribute? Perhaps it is and I just missed it.
*** Andreas has joined #calendar-qa.
celina63: Andreas, Welcome to the Calendar QA Chat
Andreas: hi celina63
Sebo: prizes are fine, but do you think that testers are really after that?
celina63: rogerk, that would be a question for the "regulars" (ctalbert, etc),
so can be addressed before next test day
celina63: I know *I* test because... well, it's what I do  :)
celina63: prizes are cool, but I do it because it's fun
celina63: it also *seemed* to me that we actually had quite a few testers... 
rogerk: Sebo: no, of course not when the prize is worth $25 =)
celina63: I wasn't able to pay very close attention thanks to the "real job"
interferring
Sebo: any numbers as how many came by?
rogerk: I'm just saying perhaps you could attract even more testers by upping
the prizes just a little. My company wants to help, and we got a dialogue going
with ctalbert, so... :)
celina63: Sebo, *I* don't have numbers... ctalbert will announce those when he
gets back
celina63: rogerk, we also were hoping to get Slashdotted, but it didn't get
picked up
celina63: I remember ctalbert saying over 600 tests had been run though
ssitter: the only numbers I see says that 661 tests were done in Litmus
ssitter: http://wiki.mozilla.org/Calendar:Status_Meetings:2006-08-23)
Sebo: Can you actually mark tests in litmus that somebody has checked them?
celina63: I thought there was
Sebo: So it is clear that they all have been looked at?
Sebo: Probably a question for ctalbert
rogerk: celina63: ok.. try digg as well next time? :)
celina63: you can select a result of your test
celina63: rogerk, good suggestion  :)
rogerk: and get people to digg it.
celina63: and there must be a way of pulling out how many times the tests were
run, since i *know* there aren't 661 tests  :D
celina63: ok, I see now... so everytime someone runs a test, it logs their
individual results underneath the test
karnesky: I thougt this did hit digg (through the linux.com/newsforge article)
celina63: so for instance, the Alarms test was run 6 times, on 3 platforms,
with varying results
karnesky: can't seem to find an article on digg now, though--I could be wrong.
rogerk: I couldn't find it either, that's why I suggested it.
celina63: ok, so we'll make a note to attempt to Slashdot and Digg the date for
next time
karnesky: http://digg.com/software/Mozilla_Calendar_Test_Day_August_22
celina63: any other suggestions on improving TestDay?
celina63: ah cool
celina63: so far we have 1.) improve prizes and 2.) Slashdot/Digg
celina63: I know we tried to get it Slashdotted but it wasn't picked up... but,
try try again!
celina63: I'd like to expand on Sebo's suggestion of more specific testing
Sebo: There is this list what 0.3 is aiming for
celina63: Sebo, do you mean perhaps to assign people to different areas as they
come into the chat?  Also perhaps an update on the wiki to let people know what
has been tested and what still needs to be tested?
celina63: ok, so hit up the 0.3 goal functionality first, then the other stuff? 
Sebo: Yes, but this needs to be more specific, of course.
Sebo: But your ideas are quite good, I think.
Sebo: ctalbert did make some suggestions, what needs more attention...
Sebo: and what has been covered a lot already.
celina63: that's a good suggestion... someone to act as a sort of "traffic cop"
to direct people to tests/areas that haven't been hit as much
celina63: I was going to update the wiki when all the bugs had been tested,
etc... but that "real job" thing happened  :p
celina63: anything else on that subject?
karnesky: The forums served that purpose in a casual way--that's the primary
reason I tested Sunbird on the Mac.  With Bugzilla, it was easy to see what had
been verified & reopened, but it'd be nice to have some sort of table showing
how much of the Litmus tests had been performed for each platform.
celina63: right
karnesky: When you started tests for a particular platform, you saw that.
celina63: but having one place to go without having to open up actual tests
would be helpful?
karnesky: yup
celina63: sounds good
celina63: about 15 minutes to go, so we'll move on
celina63: any other Litmus issues encountered, other than duplicate tests?
 ctalbert is planning on cleaning them up
*** Fallen|away is now known as xFallenAngel.
celina63: Hi Fallen!
Sebo: also, if a bug has been found and confirmed, there is no need for
following testers to test this again.
xFallenAngel: Hello. Sorry I'm late :)
celina63: no worries
celina63: Sebo, exactly why I wanted to update the wiki when all bugs had been
tested/verified
karnesky: "invalid" tests--ones that tested features which were intentionally
removed or known to be broken.
lilmatt: Sebo: True, except in the case where we want to ensure it's not
platform-specific
lilmatt: kernesky: Yes, a list of "known issues" tests would be useful, so you
know if we already expect to fail a particular test
lilmatt: ...like all the help ones.
karnesky: The phrasing of the tests could also be edited so that they are less
platform specific
lilmatt: "Tools > Options" vs. "Sunbird (Calendar)  > Preferences" sort of
thing?
karnesky: Yup.  And the keyboard shortcuts and revising tests such as the
windows custom installation so that similar functions could be tested on other
platforms
celina63: that would be hard to do on some tests
lilmatt: Some tests must be platform specific. That's a given.
celina63: like the keyboard shortcut ones... how could you make that
non-platform-specific when each platform does it differently?
lilmatt: Note what each platform should be in the test:
lilmatt: "Select All works"
lilmatt: Mac: Cmd + A
lilmatt: Win/Lin: Ctrl + A
celina63: you would have to have seperate tests... ones for Windows shortcuts,
Mac shortcuts, and Linux
karnesky: I think lilmatt's approach is fine & was acutally used in some tests.
lilmatt: Not necessarily if you spelled out the differences in the test case
itself
celina63: lilmatt, that would be fine to do *after* the tests are written,
considering there are probably very few people who are familiar with all 3
platforms
lilmatt: :)
lilmatt: True.
celina63: so the tests could be written individually, and then combined later
lilmatt: Or tests that need this "genericizing" could be listed, and edited
celina63: and then if you're testing on Windows only, you just add your results
for that platform
lilmatt: prior to the next test day
celina63: right
celina63: ok, anything else before we move on?
celina63: 10 minutes to go
celina63: ok cool... next on the agenda: TestDay times
celina63: ctalbert has suggested starting the next Testday at 1200AM UTC so
that people outside the Americas can get involved
Sebo: If johannes is volunteering to moderate in the early stage
Sebo: then it should be fine to do it
celina63: but for that to be successful, we need more volunteers to man the
channel during those early times, until us Lazy Americans get outta bed and
come play  :)
xFallenAngel: I volunteer too, I'm in the same timezone
xFallenAngel: but I think that is known
celina63: ok great, Fallen!
Sebo: I will be here, too, but can never promise...
celina63: how about this:  Once I post the Chat, how about any volunteers edit
that page in that section, giving your name and times you can be in the chat
 (in UTC of course)
celina63: that way we can ensure coverage
celina63: does that sound good?
celina63: and we can firm up our volunteer mods the day before the next Testday
Sebo: sounds good.
celina63: cool
celina63: ok, I have to be honest and say that I'm not sure what we need to
discuss regarding the next two agenda items... jminta, lilmatt, can you shed
some light on this?
celina63: and I think you guys will have to hit the Release status as well
* lilmatt looks
lilmatt: "Unconfirmed bugs"
celina63: and Verified bugs
celina63: I suppose it's a reminder for everyone to check out the bugs from
yesterday and confirm them
lilmatt: Any help we can get in confirming the unconfirmed bugs would be
wonderful. Detailed STR (steps to reproduce)
lilmatt: are key
jminta: yeah, a bunch of stuff landed yesterday
lilmatt: Verified Bugs: What jminta said. 
celina63: so after bugs are verified, what should we do about those?
lilmatt: A bunch of fixes landed yesterday. We need verifying that they fixed
what we say they fixed, and didn't regress anything else
celina63: oh, fixes
celina63: gotcha
lilmatt: (Note that I didnt make the agenda)
lilmatt: Release Status is jminta's world:
celina63: ok gang, so if we can confirm the unconfirmed bugs, and verify the
fixed ones
celina63: jminta, you get to close it out. How are we looking for release and
what else needs to be done?
jminta: celina63: i'd say verifying bugs is a big one
jminta: also, keep an eye out for qawanted bugs
lilmatt: Yes!
jminta: those are bugs that are likely interesting for the release, if QA can
say more about them
celina63: is there a release date yet?
celina63: a "drop dead" date for these things to have to happen by?
jminta: that depends...
jminta: in my mind, there will be a drop dead date, but i think i'm in the
minority there
jminta: in any case, it's not for at least 2-3 weeks
celina63: ok good, that gives us some time
rogerk: will there be a log of this chat anywhere, tomorrow?
Sebo: jminta: so qawanted is not just confirming?
jminta: keep an eye on dev.apps.calendar for more news there
jminta: Sebo: not always, but it oftentimes is
jminta: usually there will be a comment in the bug saying what is needed from
qa
Sebo: thanks, I see.
celina63: ok, ran over time, sorry about that
jminta: no worries :)
* lilmatt heads to coffee shop
celina63: OK, to wrap up   
1.) celina63 to post the log 
2.) ctalbert to clean up Litmus 
3.) Johannes, Sebo, and xFallenAngel volunteer to moderate chat so that TestDay
can start at 1200AM UTC 
4.) Everyone to hit Bugzilla and confirm/verify bugs
celina63: is that about it?
lilmatt: I can be around for 3, since I'm on the east coast
celina63: I'll also go through the log and break out all the suggestions for
improvement
celina63: lilmatt, awesome
lilmatt: Actually jminta might be by then also
lilmatt: Ok
*** lilmatt has signed off IRC (Quit: driving).
celina63: ok, and after I get the chat posted, y'all can go in and specify the
times you'll be here so we can make sure we'll have coverage
celina63: ok, if that's everything... ?
celina63: any last minute stuff?
celina63: Awesome!  Thank you all for participating!
celina63: now go hit some bugs  :D
jminta: woot