QA/Talkback/Topcrash Analysis

Introduction

How often does Firefox or Thunderbird crash? In what particular features (or during which tasks) do we see frequent crashes? Talkback, our crash reporting tool, can show us data to help answer these questions.

Publically accessible Talkback database: http://talkback-public.mozilla.org/talkback/fastfind.jsp

What needs to be done?

Mozilla QA periodically analyzes the Talkback data to see if new topcrash bugs need filing (and ultimately developer attention). Ideally this would occur on a regular basis:

• For the trunk products (Firefox and Thunderbird), check Talkback data approximately 3x a week.
• For branch Firefox and Thunderbird products, check Talkback during the candidate process and immediately after release. Then about once or twice a month thereafter.
• For branch (1.7.x) Mozilla, , check Talkback immediately after release. Then about once or twice a month thereafter.

How to view topcrash reports

1. Go to http://talkback-public.mozilla.org/reports/
2. Click on the product: Firefox, Thunderbird, Mozilla or Camino.
3. Choose the branch, trunk, release, etc. from the links under these three or two columns:
   • Development: trunk builds are here.
   • Major releases (Firefox and Thunderbird only): 1.0, 1.0.x, etc.
   • Milestone releases: also includes final Mozilla releases.
   • Beta releases: Mozilla and Camino only.
4. In the left side, a list of the reports available should appear, such as:
   • Topcrashers by Build: this is the main report to start with.
   • Simple Reports: this groups crashes by signature, and skimming this helps with getting additional comments/feedback.
   • Smart Analysis: also grouped by signature, but also lists bug number info (where available). You can add info found here to new topcrash bugs filed, especially if it helps with finding a reproducible test. Note that you should skim first to see that there's no sensitive data there (eg, user accidentally typed in a passwd!) and remove that before pasting into a bug report.

How to file a topcrash bug

1. Look at the Topcrash Reports page, and scan the top 10-15 ranked Stack Signatures.
2. Scan the Open Bugs column for stack signatures lacking bugs (empty cells).
3. Also look at the Total column which shows the total crash count per stack signature, for the range of build ID's displayed in the table.
   • Note that during the early stages of trunk development, the total crash number might be rather low (1-5), hence might not be significant enough to be filed as a topcrash bug (yet!). These shouldn't be completely ignored, however, so might deserve their own bug filed if Smart Analysis displays user comments with enough information to reproduce.
4. To figure out the best Bugzilla product and component for the topcrash bug, click (best open in a new tab) the Smart Analysis: All Platforms link on the left side of the page.
5. While filing the bug in Bugzilla, make sure the summary has the following format: branch [@ Stack:Signature()][...] - bug_summary
   • branch is the specific branch (e.g., FX102, TB10x, Trunk, M176, etc.).
   • [@ Stack:Signature()] is each stack signature from Talkback, enclosed in [] with @ followed by a whitespace preceding the signature. You can also have multiple signatures in the summary. Note that you need not include the following in the signature:
     • parenthesis in functions ()
     • call line numbers from functions
     • + signs
     • 0x00000000 or other memory offsets
   • bug_summary is, where applicable, the actual summary of the bug. If the bug was filed before topcrash analysis found it, it's fine to have this part of the summary at the beginning.
   • Example: Crash in aviary builds using Linkification extension - FF10X [@ js_GC][@ js_LinkFunctionObject]
6. The bug should also include the crash and topcrash keywords. If any of the comments have reproducible test steps, go ahead and use topcrash+.
7. Go through either the Smart Analysis and/or Simple Reports pages to gather the bug content:
   1. Paste a link to a list of the Talkback incidents; simply copy the link for the corresponding Total number from the Topcrash Report page.
   2. Where applicable, enter a summary of users' comments; if there are reproducible steps, do add them here.
   3. Paste the first frame for the most commonly reported crashes for the given signature. This will make it easier to query Bugzilla for the bug.
8. Choose the platform/OS based on the frequency of crashes listed under Lin (Linux), Win (Windows) and Mac —again, this in the Topcrash Reports page, currently on the right side of the table.
9. Copy and paste the entire Smart Analysis content for the given signature into a separate text file, then attach it to the bug. This is useful if the stacks are quite long (or varying) or if there are a lot of user comments.
10. Repeat steps 7-9 for other signatures which are similar or seem related to the bug.