Calendar:QA TestDay:2006-08-22

From MozillaWiki
Jump to: navigation, search

<< Back to Calendar QA TestDays

RESULTS!

  • 661 Tests Ran in Litmus!
  • 16 New Bugs!
  • 3 new Blocking Defects Discovered!

Many thanks to everyone involved. I'd like to thank several people who went above and beyond the call to help out: Sebo confirmed bugs, helped to ensure one of our blockers got noticed, and put together a CalDav server for testing. xFallenAngel took the lead with Litmus and sorted out the inconsistancies in our testcases. He also set up a WebDav server for the test day. Johannes worked with him to sort out the Litmus test case issues. Thanks also to our channel moderators, dmose, lilmatt, and jminta. I appreciate everyone's help, and I hope to see you all at our next Calendar Test Day Event.

The Stars:

Johannes lilmatt jminta
Andreas atrogu cpm
demo-n LisaH mschroder
rogerk sebo SirBlackheart
Vali_Systm VenomousGecko Vamplr3
Taleisin xFallenAngel LonelyBob
karnesky ahz jeizer_a
tbodner excessory Damian
bbbrowning torusturtle celina63

In all, the Test Day was a resounding success. I look forward to doing this again with everyone. We will start having regular test days in as we near the 0.3 release date.

Remember that two rewards were given for the day. One reward for the most tests run/bugs verified. And one for the best bug found. With that in mind, the winners are....

  • Karnesky takes the first prize with a whopping 278 tests run in Litmus!
  • Sebo takes the second prize for confirming two blocking defects, setting up calendar servers, and working hard to make the test day a success.


Where do I start?


OK, I am in the channel, now what?

Let the moderators know if you have any questions or encounter any problems. If you are not familiar with our bugzilla bug tracking system, feel free to report your problems to a moderator and they can log any bugs for you.

Who will be moderating the #calendar-qa channel on the Test Days?

The moderator's nicks for this test day will be:

  • ctalbert
  • jminta
  • lilmatt
  • dmose

What build should I test?

We will be testing the Sunbird and Lightning nightly candidate builds:

Sunbird:

ftp://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/latest-trunk

Lightning (for Thunderbird 1.5.x and 2.0 candidates):

ftp://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-mozilla1.8

Lightning (for Thunderbird 3.0 candidates):

ftp://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-trunk (On Windows and Linux use the Thunderbird builds next to Lightning because of bug 321244, bug 343053)

What specifically will we be testing?

In a word, everything.

There are three main areas that we need to concentrate on. Here they are in order of importance.

Bugzilla Testing

Verify that the fixed bugs are addressed by these nightly builds.

Litmus Testing

There are four test suites in Litmus for Calendar. We want to run as many of these Litmus tests as possible for each product.

  • Sunbird Basic Functional Tests - basic tests for Sunbird
  • Sunbird Full Functional Tests - in depth tests for Sunbird
  • Lightning Basic Functional Tests - basic tests for Lightning
  • Lightning Full Functional Tests - in depth tests for Lightning
Usability Testing

From the release overview and the Calendar Charter, we have several high level goals that we want the 0.3 Calendar release to achieve. We need your help and creativity in ensuring that the nightly candidates for 0.3 move the Calendar project toward meeting these goals.


BEFORE YOU START--BACKUP YOUR PROFILE

Once you have downloaded the version of the application you want to test, but before you start it, you MUST back up your profile and create a testing profile, otherwise, you run a very good chance of losing your data.

Backing up a profile is easy:
1. Find your profile directory:

OS Thunderbird Profile Sunbird Profile
Windows  %AppData%\Thunderbird  %AppData%\Mozilla\Sunbird
Linux ~/.thunderbird ~/.mozilla/sunbird
MacOSX ~/Library/Thunderbird ~/Library/Application Support/Sunbird

2. Copy the root of it somewhere else, for example copy the Thunderbird or Sunbird directory and everything below it.

Creating a testing profile is also easy:

  1. Launch your application with a -P option
  2. Create a New Profile by clicking the Create a New Profile button
  3. Click Next, Give it a name like "Testing Profile", click Finish
  4. Select the testing profile and click the "Start Calendar" (Or Start Mail in Thunderbird) button.

Remember that you will have to start the application with the -P option during testing so you can pick your Testing profile

Once you've finished testing, you can delete the extra profiles and you'll be back to normal.

If something goes terribly wrong (although we don't expect it to), close the application and just replace the profile with the backed up copy. That will reset the application to the way it was before you started testing.

Testing Instructions

  1. Download Sunbird or Lightning from the above URLs
  2. Create your testing profile (see above)

THEN - Verify Bugzilla Fixed Bugs

  1. Click on the above Bugzilla bug list link and verify Fixed bugs. Read how the bug was produced and try to recreate it to verify the error doesn't happen anymore.
  2. If the error is no longer occurring, select the "Mark bug as Verified" radio button and click the Commit button. If the error still occurs, select Reopen Bug and enter comments as to why the bug is being reopened (i.e. what is still failing), then click the Commit button.

OR - Run Litmus Test Cases 

buildIds for August 22 Test Day
--------------------------------------------
                     Lightning/   Lightning/
OS       Sunbird     Tbird 1.5/2  Tbird 3.0
Linux    2006082206  2006082206   2006082206
Mac      2006082206  2006082009   2006082108
Windows  2006082207  2006082206   2006082206
  1. Go to Litmus.
    1. Select Run Tests, the Product is Calendar. Your buildId should match the one in the table above. Verify that it does. Sunbird's buildId is in the "About Sunbird" window. Lightning's buildId is in its description in the Add-ons Manager.
    2. Choose a Test group and subgroup to test on (choose one that has a low percentage of Community Coverage).
    3. Follow the test steps and verify the function under test works as described in Expected Results.
    4. If the test case performs as expected, mark it as a "Pass". If not, mark it as a "Fail" and then enter a bug (see below).
    5. When you've finished one test, go to the next test by clicking the test case name.
    6. Once you've completed all the tests in a group, click the "Submit All Results" button at the bottom.

OR - Perform Usability Testing

  1. Use Sunbird and Lightning like you normally would. Try to do everything an end-user might do, ranging from simple calendar operations to power user features. Try to think of actions that are not typical since you will most likely find a bug there.

BUG FOUND!

  1. When you think you've found a bug, try to reproduce it. Restart Sunbird/Lightning and do the same steps again. The most valuable bugs are those that can be reproduced.
  2. Once the bug has been reproduced, search for it or something similar on Bugzilla. This can easily be done by searching for the error message, window name, or some over identifying feature of the bug. If you found an existing bug that is just like the bug you found, read the comments. Maybe your situation is slightly different and your input could help developers find the origin of the bug.
  3. If you can't find the bug in Bugzilla, talk about it in the channel. Maybe others can confirm your bug or find a similar bug in bugzilla. This also gives moderators a chance to guide you to either provide more details, update an existing bug or file a new bug.
  4. If you are certain you found a unique bug and have read the Bug Writing Guidelines and Bugzilla Etiquette, go ahead and file the bug in Bugzilla. Remember to be as specific as possible.

Some other ways you can help:

  • Think of some test cases that aren't already included in Litmus (we love suggestions!).
  • Ask your friends to join you. The more people help the better!

Where do I find Litmus and Bugzilla?

  • Bugzilla can be found at http://bugzilla.mozilla.org/
  • A list of Calendar specific bugs (without enhancements) can be found here
  • A list of already filed enhancement requests can be found here

Tips:

  • Make sure you enter the correct information for your platform, version of Thunderbird, Lightning etc. Please include the full build identifier (i.e Thunderbird 1.5.0.5 (20060719) with Lightning 0.1+ (2006080907)).
  • Be sure to read the Testing Instructions above before filing a bug.
  • Feel free to ask on IRC if you need help or have any more questions.

How do I send feedback?

Please use email us at dev-apps-calendar at lists.mozilla.org to send feedback about how the testday went, any overall thoughts about Sunbird and Lightning, and to follow up on issues found and bugs filed. We really appreciate your help today and your feedback is very valuable.

Rewards!

There will be two prizes given. Both prizes will be $25 gift certificates to the Mozilla Store. There are two categories for winners.

  • Prize #1 goes to the person that completes the most tests. This means that they verify the most Bugzilla fixes and/or complete the most Litmus test cases.
  • Prize #2 goes to the person that finds the best set of bugs. These bugs will be high-profile ones as deemed by the calendar QA team and the 0.3 release drivers as high profile, possibly blocking bugs.
  • As always, know that we appreciate everyone's contribution and you all have our heart felt gratitude. Thank you.
Retrieved from "https://wiki.mozilla.org/index.php?title=Calendar:QA_TestDay:2006-08-22&oldid=34982"