Calendar:QA TestDay:2006-10-03

From MozillaWiki
Jump to: navigation, search

<< Back to Calendar QA TestDays

RESULTS!

  • 1 ShowStopper Bug
  • 159 Tests Ran in Litmus!
  • 5 New Bugs!
  • 7 Fixes Verified!

Many, many thanks to everyone involved. The test day was a success--we did a ton of good testing with far fewer people than our last test day. Special thanks go to the good people at Linpro AS (rogerk and jamesb) for sponsoring our test day awards. Thanks to everyone who stepped up to help out with moderating this test day: ssitter, xFallenAngel, lilmatt, dmose, jminta. Thanks to Celina63 for architecting our Test Day wiki page.

Thanks also to all the new contributors we had this test day: TedBullock, omar_public, Mc, matthaeus123, karora.

I continue to be impressed by the expertise and professionalism exhibited by the Calendar Community. You should all pat yourselves on the back. We hope to see you all soon for our next test day. And don't forget to check out our weekly Calendar-QA ToDo list

Our Stars:

jminta lilmatt dmose
celina63 xFallenAngel Mc
bbbrowning karora mschroder
rogerk sebo lonelybob
nightrat Taliesin ssitter
omar_public matthaeus123 excessory
TedBullock worzel Damian

And now...the winners!

  • 1st Prize goes to Sebo for finding the show stopper bug 355304!
  • 2nd Prize for running 45 test cases in Litmus goes to karora!

Thanks again everyone!

Where do I start?

OK, I am in the channel, now what?

Let the moderators know if you have any questions or encounter any problems. If you are not familiar with our bugzilla bug tracking system, feel free to report your problems to a moderator and they can log any bugs for you.

Who will be moderating the #calendar-qa channel on the Test Days?

The moderator's nicks for this test day will be:

  • ctalbert
  • ssitter
  • lilmatt
  • dmose
  • xFallenAngel

What build should I test?

We will be testing the Sunbird 0.3 and Lightning 0.3 Release Candidate builds:

Sunbird:

http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/releases/0.3rc1/

Lightning (for Thunderbird 1.5.x and 2.0 candidates):

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/releases/0.3rc1/

What specifically will we be testing?

In a word, everything.

There are three main areas that we need to concentrate on. Here they are in order of importance.

Bugzilla Testing

Verify that the fixed bugs are addressed by these nightly builds.

Litmus Testing

There are four test suites in Litmus for Calendar. We want to run as many of these Litmus tests as possible for each product.

  • Sunbird Basic Functional Tests - basic tests for Sunbird
  • Sunbird Full Functional Tests - in depth tests for Sunbird
  • Lightning Basic Functional Tests - basic tests for Lightning
  • Lightning Full Functional Tests - in depth tests for Lightning
Usability Testing

From the release overview and the Calendar Charter, we have several high level goals that we want the 0.3 Calendar release to achieve. We need your help and creativity in ensuring that the nightly candidates for 0.3 move the Calendar project toward meeting these goals.


BEFORE YOU START--BACKUP YOUR PROFILE

Once you have downloaded the version of the application you want to test, but before you start it, you MUST back up your profile and create a testing profile, otherwise, you run a very good chance of losing your data.

Backing up a profile is easy:
1. Find your profile directory:

OS Thunderbird Profile Sunbird Profile
Windows  %AppData%\Thunderbird  %AppData%\Mozilla\Sunbird
Linux ~/.thunderbird ~/.mozilla/sunbird
MacOSX ~/Library/Thunderbird ~/Library/Application Support/Sunbird

2. Copy the root of it somewhere else, for example copy the Thunderbird or Sunbird directory and everything below it.

Creating a testing profile is also easy:

  1. Launch your application with a -P option
  2. Create a New Profile by clicking the Create a New Profile button
  3. Click Next, Give it a name like "Testing Profile", click Finish
  4. Select the testing profile and click the "Start Calendar" (Or Start Mail in Thunderbird) button.

Remember that you will have to start the application with the -P option during testing so you can pick your Testing profile

Once you've finished testing, you can delete the extra profiles and you'll be back to normal.

If something goes terribly wrong (although we don't expect it to), close the application and just replace the profile with the backed up copy. That will reset the application to the way it was before you started testing.

Testing Instructions

  1. Download Sunbird or Lightning from the above URLs
  2. Create your testing profile (see above)

THEN - Verify Bugzilla Fixed Bugs

  1. Click on the above Bugzilla bug list link and verify Fixed bugs. Read how the bug was produced and try to recreate it to verify the error doesn't happen anymore.
  2. If the error is no longer occurring, select the "Mark bug as Verified" radio button and click the Commit button. If the error still occurs, select Reopen Bug and enter comments as to why the bug is being reopened (i.e. what is still failing), then click the Commit button.

OR - Run Litmus Test Cases 

Build ID for October 3 Test Day
--------------------------------
OS      | Sunbird    | Lightning
Linux   | 2006100207 | 2006100206
Mac     | 2006100207 | 2006100209
Windows | 2006100207 | 2006100206
  1. Go to Litmus.
    1. Select Run Tests, the Product is Calendar. Your buildId should match the one in the table above. Verify that it does. Sunbird's buildId is in the "About Sunbird" window. Lightning's buildId is in its description in the Add-ons Manager.
    2. Choose a Test group and subgroup to test on (choose one that has a low percentage of Community Coverage).
    3. Follow the test steps and verify the function under test works as described in Expected Results.
    4. If the test case performs as expected, mark it as a "Pass". If not, mark it as a "Fail" and then enter a bug (see below).
    5. When you've finished one test, go to the next test by clicking the test case name.
    6. Once you've completed all the tests in a group, click the "Submit All Results" button at the bottom.

OR - Perform Usability Testing

  1. Use Sunbird and Lightning like you normally would. Try to do everything an end-user might do, ranging from simple calendar operations to power user features. Try to think of actions that are not typical since you will most likely find a bug there.

BUG FOUND!

  1. When you think you've found a bug, try to reproduce it. Restart Sunbird/Lightning and do the same steps again. The most valuable bugs are those that can be reproduced.
  2. Once the bug has been reproduced, search for it or something similar on Bugzilla. This can easily be done by searching for the error message, window name, or some other identifying feature of the bug. If you found an existing bug that is just like the bug you found, read the comments. Maybe your situation is slightly different and your input could help developers find the origin of the bug.
  3. If you can't find the bug in Bugzilla, talk about it in the channel. Maybe others can confirm your bug or find a similar bug in bugzilla. This also gives moderators a chance to guide you to either provide more details, update an existing bug or file a new bug.
  4. If you are certain you found a unique bug and have read the Bug Writing Guidelines and Bugzilla Etiquette, go ahead and file the bug in Bugzilla. Remember to be as specific as possible.

Some other ways you can help:

  • Think of some test cases that aren't already included in Litmus (we love suggestions!).
  • Ask your friends to join you. The more people help the better!

Where do I find Litmus and Bugzilla?

  • Bugzilla can be found at http://bugzilla.mozilla.org/
  • A list of Calendar specific bugs (without enhancements) can be found here
  • A list of already filed enhancement requests can be found here

Tips:

  • Make sure you enter the correct information for your platform, version of Thunderbird, Lightning etc. Please include the full build identifier (i.e Thunderbird 1.5.0.7 (20060909) with Lightning 0.3 (2006100206)).
  • Be sure to read the Testing Instructions above before filing a bug.
  • Feel free to ask on IRC if you need help or have any more questions.

How do I send feedback?

Please use email us at dev-apps-calendar at lists.mozilla.org to send feedback about how the testday went, any overall thoughts about Sunbird and Lightning, and to follow up on issues found and bugs filed. We really appreciate your help today and your feedback is very valuable.

Rewards!

There will be one prize available for the person that makes the largest contribution to the test day. That can be done by verifying bugs, finding bugs, and/or running Litmus testcases. Each of these endeavors will be scored equally, so the more you do, the more likely you are to win!

Rewards are provided by Linpro!

I only know most of you by your IRC nickname, so to ensure your bugzilla activity gets counted, be certain to email cmtalbert at gmail dot com with your bugzilla user ID. Please use "TEST DAY" in the subject line of the mail.

  • As always, know that we appreciate everyone's contribution and you all have our heart felt gratitude. Thank you.
Retrieved from "https://wiki.mozilla.org/index.php?title=Calendar:QA_TestDay:2006-10-03&oldid=35566"