QA/Talkback/Topcrash Analysis
From MozillaWiki
Contents
Introduction
How often does Firefox or Thunderbird crash? In what particular features (or during which tasks) do we see frequent crashes? Talkback, our crash reporting tool, can show us data to help answer these questions.
Publically accessible Talkback database: http://talkback-public.mozilla.org/search/start.jsp
What needs to be done?
Mozilla QA periodically analyzes the Talkback data to see if new topcrash bugs need filing (and ultimately developer attention). Ideally this would occur on a regular basis:
- For the trunk products (Firefox and Thunderbird), and for branch products with active development (such as a release branch before the .0 release on that branch), check Talkback data approximately 3x a week.
- For branch Firefox and Thunderbird products being maintained for security/stability releases, check Talkback during the candidate process and immediately after release. Then about once or twice a month thereafter.
How to view topcrash reports
- Go to http://talkback-public.mozilla.org/reports/
- Click on the product: Firefox, Thunderbird, Mozilla or Camino.
- Choose the branch, trunk, release, etc. from the links under these three or two columns:
- Development: trunk builds are here.
- Major releases (Firefox and Thunderbird only): 1.0, 1.0.x, etc.
- Milestone releases: also includes final Mozilla releases.
- Beta releases: Mozilla and Camino only.
- In the left side, a list of the reports available should appear, such as:
- Topcrashers by Build: this is the main report to start with.
- Simple Reports: this groups crashes by signature, and skimming this helps with getting additional comments/feedback.
- Smart Analysis: also grouped by signature, but also lists bug number info (where available). You can add info found here to new topcrash bugs filed, especially if it helps with finding a reproducible test. Note that you should skim first to see that there's no sensitive data there (eg, user accidentally typed in a passwd!) and remove that before pasting into a bug report.
Topcrash status
These pages track known status of the signatures reported in talkback for nightly builds and for the most recent releases of the following:
These pages are intended to track information like:
- which of the crashes reported under a particular signature are covered by a particular bug report
- when the crashes for a signature were last examined to verify that they were mostly covered by existing bug reports
- when recent bug fixes have landed that should be verified by tracking data
- when top crashes in the most recent release have been fixed in nightly builds since that release (which is important to track since some crashes tend to show up in releases but not nightlies, due to differing usage patterns)
These status pages often list parts of stacks. Note that some frames of a stack are sometimes omitted depending on how the crash happens and what platforms it happens on. When stacks are listed, frames that can sometimes appear omitted are parenthesized.
How to file a topcrash bug
- Look at the Topcrash Reports page, and scan the top 10-15 ranked Stack Signatures.
- Scan the Open Bugs column for stack signatures lacking bugs (empty cells).
- Also look at the Total column which shows the total crash count per stack signature, for the range of build ID's displayed in the table.
- Note that during the early stages of trunk development, the total crash number might be rather low (1-5), hence might not be significant enough to be filed as a topcrash bug (yet!). These shouldn't be completely ignored, however, so might deserve their own bug filed if Smart Analysis displays user comments with enough information to reproduce.
- To figure out the best Bugzilla product and component for the topcrash bug, click (best open in a new tab) the Smart Analysis: All Platforms link on the left side of the page.
- While filing the bug in Bugzilla, make sure the summary has the following format: branch [@ Stack:Signature()][...] - bug_summary
- branch is the specific branch (e.g., FX102, TB10x, Trunk, M176, etc.). An 'x' denotes that any 1.0.x branch exhibits the crash.
- [@ Stack:Signature()] is each stack signature from Talkback, enclosed in [] with @ followed by a whitespace preceding the signature. You can also have multiple signatures in the summary. Note that you need not include the following in the signature:
- parenthesis in functions ()
- call line numbers from functions
- + signs
- 0x00000000 or other memory offsets
- bug_summary is, where applicable, the actual summary of the bug. If the bug was filed before topcrash analysis found it, it's fine to have this part of the summary at the beginning.
- Example: Crash in aviary builds using Linkification extension - FF10X [@ js_GC][@ js_LinkFunctionObject]
- The bug should also include the crash and topcrash keywords. If any of the comments have reproducible test steps, go ahead and use topcrash+.
- Go through either the Smart Analysis and/or Simple Reports pages to gather the bug content:
- Paste a link to a list of the Talkback incidents; simply copy the link for the corresponding Total number from the Topcrash Report page.
- Where applicable, enter a summary of users' comments; if there are reproducible steps, do add them here.
- Paste the first frame for the most commonly reported crashes for the given signature. This will make it easier to query Bugzilla for the bug.
- Choose the platform/OS based on the frequency of crashes listed under Lin (Linux), Win (Windows) and Mac —again, this in the Topcrash Reports page, currently on the right side of the table.
- Copy and paste the entire Smart Analysis content for the given signature into a separate text file, then attach it to the bug. This is useful if the stacks are quite long (or varying) or if there are a lot of user comments.
- Repeat steps 7-9 for other signatures which are similar or seem related to the bug.
