From MozillaWiki
Jump to: navigation, search

in-litmus? Bug Burn

Date Sept 13 - 17, 2010
Time 1800Z - 2400Z (11am - 6pm PDT)
Place #bugday
What We will be triaging the in-litmus? bug queue and writing test cases for these bugs. This will be a great opportunity to hone or learn bugzilla and litmus skills, test case writing skills, and investigative skills.
Goal The goal will be to resolve ALL of the bugs, meaning no bugs with in-litmus? at the end of the week.
Artifacts Bug List, Spreadsheet, Scorecard, Pre-bugday Survey, Post-bugday Survey, Testcase Writing Documentation


All times are in Z-time (GMT). Subtract 7 hours for PDT.

Monday Tuesday Wednesday Thursday Friday
1800Z WORKSHOP: Learning to Write Testcases STRIKE TEAM: Tabbed Browsing STRIKE TEAM: Session Store STRIKE TEAM: Panorama SPRINT & TESTDAY

Prizes & Tiers

Prizes will be awarded based on individual contributions. In the case of teams, each contributor gets equal credit for the team effort. Mozilla QA employees do not qualify for prizes

Level Achievement Prize
1 10 points Mozilla wristband, entry into MozillaQA draw
2 20 points T-shirt of your choosing
3 50 points $25 USD credit for the Mozilla Store
Top Tester
$100 USD gift card (store TBD)
  • SCORECARD: link
  • WORKSHOP: Educators give score based on lessons accomplished
  • STRIKE: Team Leads track team members and score; awarding each member with the team score at the end of the day
  • SPRINT: Moderator gives score based on tasks complete
  • TESTDAY: Moderator gives score based on tasks complete
  • See individual testplans for details on scoring for each day

Sign Up

Please put your name (or IRC nick) and the times you will be willing to help out. The roles are moderator, assistant, and contributor (see below for a more detailed description).


During the first day we will be running workshops to help community members gain the skills needed to successfully triage bugs, navigate Litmus, and write good test cases. Please sign up as an assistant if you would like to help me with moderator duties. Please sign up as an educator if you would like to help train community members.

Strike Team

During the next three days, we will apply the skills learned in the first day. This will give the community the hands on experience with resolving in-litmus bugs and writing test cases. Each day we will be focusing on a particular feature are of Firefox. Please sign up as an assistant if you would like to help me with moderator duties. Please sign up as a striker if you would like to be part of a strike team.

Each day we will be focusing on a particular feature area of Firefox. The following is a synopsis of each feature area:

Tuesday Tabbed Browsing
LEAD juanb
SUMMARY Tabbed Browsing is a long standing feature in Firefox which allows users to browse multiple web pages in a single window by placing each page in its own tab. Today we will be vetting the current set of testcases, creating testcases for any in-litmus? bugs, and writing exploratory testcases for any fixed bugs.
LINKS Litmus Testcases, FIXED bugs, in-litmus? bugs
Wednesday Session Store
LEAD ashughes
SUMMARY Session Store is a long standing feature in Firefox which allows users to recover the browser from a crash. Today we will be vetting the current set of Litmus testcases and creating exploratory testcases against any fixed bugs.
LINKS Litmus testcases, FIXED bugs
Thursday Panorama
LEAD tracy
SUMMARY Panorama (aka Tab Candy) is a new feature in Firefox which allows users to group and organize their tabs. Today we will be creating testcases for any in-litmus? bugs, and creating exploratory testcases against any open bugs.
LINKS in-litmus? Bugs, Open Bugs, Litmus Testcases, Feature Summary
Friday Test Day
MODERATORS whimboo, juanb, ashughes, marcia, ...
SUMMARY We will hold a Testday on Friday, Sept 17. By then we should have candidate builds available for beta 6. There have been a ton of landings in several areas, but it has been suggested we concentrated on the Add-ons Manager.
LINKS Testday Details, Bugs

During the final day, we will have a 4-hour contest. We will let the community members work on their own at their own pace. This will be the community's opportunity to work toward prizes. Please sign up as an assistant if you would like to help me with moderator duties. Please sign up as a sprinter if you will just be around to work on test cases and answer some community questions.

Please sign up for a timeslot and role:

Monday Moderator Assistant Educators
1800Z - 2000Z ashughes krupa
2000Z - 2200Z ashughes mw22 tracy
2200Z - 2400Z ashughes tracy
Tabbed Browsing
Moderator Assistant Strikers
Lead: juanb
1800Z - 2000Z ashughes Cameron Dawson
2000Z - 2200Z ashughes rbillings
2200Z - 2400Z ashughes
Session Store
Moderator Assistant Strikers
Lead: ashughes
1800Z - 2000Z tracy
2000Z - 2200Z tracy
Moderator Assistant Strikers
Lead: tracy
1800Z - 2000Z ashughes AaronMT
2000Z - 2200Z ashughes
2200Z - 2400Z ashughes tchung
Friday Moderator Assistant Sprinters
1800Z - 2000Z ashughes stephend
2000Z - 2200Z ashughes

Roles-specific Tasks

Moderating the Channel
  • Great people as they join
  • Make sure they have filled out the pre-bugday survey
  • Make sure they have read the documentation and this test plan
  • Be proactive and ask them if they know what they want to do. If not, give them something to do
  • If they ask questions, answer them. If they don't ask questions, ask if they need help
  • If they are working with a specific component, introduce them to the component owner
  • Don't forget to broadcast the hourly advertisement in other IRC channels:
    • QA is running a week-long bugzilla event to burn through bugs needing test cases. Want to help write test cases or learn how? Join us in #bugday
    • Channels: #firefox, #developers, #qa
Assisting the Moderator
  • Be sure to greet newcomers and point them to the survey if they haven't already filled it out
  • Be prepared to complete any of the above tasks
  • Be proactive and ask if the moderator needs help
  • If the moderator seems idle, be proactive and assist community members
  • Cover for the moderator if a break is needed
Assisting Contributors
  • If someone asks a questions, help them as best you can
  • If they don't ask questions, ask if they need help
  • Review contributor test cases and give them advice on good writing techniques
  • Help contributors navigate Litmus and Bugzilla by sharing your personal tips, tricks, and techniques
  • Make sure contributors are aware of and familiar with the documentation
  • Make sure they know how to use the spreadsheet
  • Answer questions
  • Possible topics to teach to community:
    • Finding a testcase in Litmus
    • Anatomy of a testcase
    • Understanding how to reproduce a bug
    • Determining if a bug needs a testcase
Strike Team
  • Make sure teammates are familiar with the documentation, spreadsheet and bug list
  • Help teammates work through the bug list
  • Help teammates navigate Litmus to find existing test cases
  • Review test cases added to the spreadsheet and give teammates feedback
  • Make changes in Litmus on behalf of teammates and give them credit in the spreadsheet
  • Answer any questions from contributors
  • Work through the spreadsheet adding/revising test cases in Litmus for contributors
  • Answer any questions from contributors
Component Owners
  • Answer questions people may have specific to your component areas

Common Tasks

Identify Steps to Reproduce
  • Sometimes a bug will have clearly defined steps to reproduce it
  • If the steps are not well defined, read through the comments to see if you can formulate your own steps
  • If you are finding it difficult, ask for help from QA on IRC
  • If all else fails, comment on the bug to get a clear set of steps from the developer:
    • I'm having a hard time understanding the steps to reproduce this bug. Could someone please indicate how this bug was originally reproducible?
Identify Expected Results
  • Sometimes a bug will have a clearly defined result
  • If the expected result (ie. the fix) is not well defined, read through the comments to see if you can formulate your own assumptions of what should happen
  • If the bug has a well defined set of steps to reproduce, run through those steps and note the behaviour of the browser
  • Ask someone from QA on IRC if they can help you verify what the expected result should be
  • Ask a developer on the bug if
    • I'm having a hard time understanding what the desired behaviour is now that this is fixed. Could someone please indicate more clearly what the expected result is?
    • I ran through the steps and got this result, is that expected?
Find existing test in Litmus
  • Firstly, check with the component owner. She may know if a test case already exists from memory. If not, she will be able to direct you to the testgroup in Litmus.
  • You could also check by going to the Firefox 3.6 test run or the Firefox 4 test run and looking for adequate test cases in the subgroup matching your component
  • Otherwise, you'll have to do a manual search on litmus by going to view/search tests and use the following criteria:
    • Product: Firefox
    • Branch: 3.6 or 4.0 (you can get the version from the bug -- if unsure, assume 4.0)
    • Testgroup: Select Full Functional Tests
    • Subgroup: Select the subgroup matching the component of the bug
Revising a test
  • If a test case already exists but the behaviour has slightly changed, simply note in the spreadsheet the changes that need to be made
Disabling a test
  • If a test case already exists but is obsolete or no longer needed, simply note in the spreadsheet the test should be disabled
Write a new test
  • If the test case cannot be found and you think a new one needs to be written, simply write your steps and expected results in the spreadsheet
  • Make sure you notify the component owner of the entry so that he can vet it and add it to Litmus
Using the Spreadsheet
  • Link
  • Component Owners
    • work using the Overview sheet
    • track progress as you vet test cases and add/revise/disable in Litmus
    • create a new sheet for you component for contributors to add test cases
    • don't forget to flip the in-litmus flag
  • Contributors
    • add test cases to the specific component sheet
    • if it's a revision, simply note the changes that need to be made
    • if it's a test case that needs to be disabled, simply state the test no longer applies and should be disabled
    • if no test case is needed for the bug, flip the in-litmus flag to in-litmus- (or ask the component owner to do it)


Before writing any test cases you should familiarize yourself with our Test Case Writing Primer. It will tell you everything you need to know about writing good test cases.

The document is pretty extensive so try not to be overwhelmed or intimidated. Just try your best to understand and remember some of the key points it tries to convey. The following is a quick set of important guidelines you should take away from reading this document:

1. Be descriptive and concise
2. Make the steps easy to follow
3. Make the expected results clear to understand
4. Don’t get hung up on terminology
5. Write your test cases as if the reader has never used Firefox
6. Test case summaries should include a component header and briefly but clearly describe the test:

  • [component:subgroup] Summary
  • Example: [addons:themes] Install a theme using the Add-on Manager

7. Test case steps should be a numbered-list and expected results should be a bullet-list

Example Test Case
[toolbar:customize] Add a Print icon to the toolbar

1. Right click on the toolbar, and select “Customize” from the dropdown menu.
2. A “Customize Toolbar” dialog box containing icons/spaces should be shown.
3. Select the Print icon using your mouse.
4. Drag the Print icon to the toolbar and place it to the right of the URL bar (between the end of the URL bar and the beginning of the search box).
5. Click “Done” to close the Customize Toolbar dialog box.
6. Restart Firefox.

Expected Results:
  • The Print icon should be added to the toolbar in the place that you specified.
  • Clicking on the Print icon should launch the Print dialog box.

The above test could probably stand to be a bit more concise, but it is a "good" test case, in general.

Components & Owners

Component Owner IRC Nick
Add-ons Henrik Skupin whimboo
Audio & Video Anthony Hughes ashughes
Awesomebar Aakash Desai aakashd
Content Handling Martijn Wargers mw22
Cookies Marcia Knous marcia
Downloading Al Billings abillings
Find in Page Juan Becerra juanb
Form Manager Juan Becerra juanb
General Tomcat tomcat
Geolocation Marcia Knous marcia
Import Marcia Knous marcia
Install/Migration Marcia Knous marcia
Layout Martijn Wargers mw22
Library (History & Bookmarks) Tracy Walker tracy
Lightweight Themes Aravind aravind
Menu Bar Henrik Skupin whimboo
Microsummaries Aravind aravind
Online/Offline Events Juan Becerra juanb
Options Aakash Desai aakashd
Password Manager Juan Becerra juanb
Plugins Juan Becerra juanb
Popup & Annoyance Blocking Al Billings abillings
Printing Matt Evans mevans
Private Browsing Marcia Knous marcia
RSS Tomcat tomcat
Search Henrik Skupin whimboo
Security Anthony Hughes ashughes
Session Store Anthony Hughes ashughes
Sync Tony Chung tchung
Tabbed Browsing Matt Evans mevans
Technical Tools Aravind aravind
Toolbar Customization Juan Becerra juanb