Platform/Uptime/NightlyCrashAnalysis: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Replaced content with "The contents of this page have moved to [NightlyCrashTriage].")
Line 1: Line 1:
This is a rough guide on how to analyze the crash reports for a particular day's Nightly build.
The contents of this page have moved to [NightlyCrashTriage].
 
== Crash Report Basics ==
 
The first thing you should do is watch David Baron's
[https://air.mozilla.org/a-talk-about-understanding-crash-reports/ talk about crash reports]. Watch it all the way through. It's full of useful information.
 
Also, see the [https://developer.mozilla.org/en-US/docs/Understanding_crash_reports documentation about reading individual crash reports] on MDN.
 
== Nightly Triage ==
 
The start point is [http://dbaron.org/mozilla/crashes-by-build this page]. You should look at the crashes for all four platforms (Windows, Mac, Linux, Android) for the the build you are analyzing.
 
We keep notes at [[Platform/Uptime#Notes]]. Please add your notes to the top of that page. You don't need to write a lot. The logs serves two main purposes:
* To show which builds have been analyzed (and any that might have been missed).
* To communicate anything that might be useful to the people coming up in the roster (e.g. if crash reports are currently broken on one platform, or something like that).
 
Nightly builds are created at 3am California time. It's generally best to wait at least 24 hours before analyzing a build, because that gives enough time for a decent number of crashes to come in.
 
Windows crash numbers are much higher than the other and you will likely spend most of your time looking at them. Typical numbers for the #1 crash signature on Windows will be anywhere from 10--50 crash reports, though there is plenty of variation from day to day.
 
The basic goal of triage is to get a sense of the current crash situation and take steps to improve it.
* Look closely at most/all of the top crash signatures (e.g. top 10--15 on Windows, top 5 or so on other platforms). (There are certain ones that are hard to act on, e.g. "ShutdownKill", "OOM | Small", Flash crashes.)
* When appropriate, file bugs for signatures that lack them (more details about this are below). If a signature only has an old bug, filing a new one might be appropriate.
* Check bugs for signatures that have them to see if they need additional information.
* If a bug has been closed but the crash is still happening, the bug might need to be reopened.
* If a bug is open but stalled, adding a NEEDINFO request might help, and adding information about crash frequency can be helpful. (Polite nagging can be effective!)
* Look for distinct crash signatures that might be related, e.g. multiple signatures relating to a11y, or to the JS JITs.
* The lower-ranked signatures are less important, though filing bugs for signatures that lack them can still be useful.
 
One caveat is that sometimes multiple crashes (possibly even 10s or 100s) with a particular signature all come from a single installation. Those ones are usually best ignored, because it's difficult to tell if it's a real problem or a problem with the user's machine or installation, and there are enough other crashes to deal with. Likewise, if a crash appears on one Nightly build but none before or after, it's probably not worth filing.
 
To determine how many installations a particular crash signature has affected:
* Look at the "Product*" box, which has an "installations" column. Note that this box gives results for the past 7 days, unlike the other boxes.
* Add the "install time" field to the search box in a signature report. If the install times are all identical it's almost certainly a single installation.
 
JS engine crashes can be difficult to deal with, especially those involving the JITs and the GC, because crashes with different causes can get lumped into the same signature.
 
== Filing Bug Reports ==
 
To file a bug report about a particular crash signature, view one crash report and use the relevant "Report this bug in" link. This pre-populates the bug report with the "crash" keyword and a link to the particular crash report.
 
When you file a bug report, things to mention include the following.
* The signature and the crash reason (or pattern of crash reasons)
* The Nightly build id, which looks like "20160506052823".
* How many crashes have occurred with this signature.
* Anything unusual (crash only on specific platforms, only in certain locales or on certain sites, extension or dll correlations of interest, whether it's a startup crash, etc.) The correlations tab in crash-stats is great for this.
* The rank (e.g. "this is the #1 top crash for Nightly 20160506052823").
* How many installations are involved.
* If the signature is new, and if so, which Nightly build it started in. One way to determine this is to modify the search parameters (by modifying the URL, or by modifying the search in the crash-stats UI) to broaden the search, e.g. by removing the particular Nightly build identifier and changing the date parameters.
* If possible, a regression window for that Nightly. You can get regression Windows by using the "Choose regression window" button on the [http://dbaron.org/mozilla/crashes-by-build start page].
* If possible, an indication of which bug's patches may have caused the crash.
 
Don't worry too much if you can't get all of these things. Getting the right person to look at a crash report is the most important thing. Therefore, you should always NEEDINFO someone in a bug report. If you look at the top few frames of one or more reports, try to see if there are any lines that changed recently. If so, that's a good clue as to what change caused the crash and you can NEEDINFO the author. Failing that, NEEDINFO someone appropriate based on your knowledge of the codebase, or by looking at who has modified nearby lines, or by consulting the [[Modules/All|module owners]] or [[Bugmasters/Project/Bug_Handling/Triage_Leads|triage leads]] lists.
 
Sometimes a single crash cause will result in multiple signatures. It's useful to link all the relevant signatures to the same bug. {{Bug|1229252}} is a good example, where the "crash signature" field grew steadily over a period of weeks.
 
A crash that occurs more than 10 times a day, across multiple installations, is probably impactful enough to be worth tracking. Set the appropriate tracking flag to '?' in the bug report, so that release drivers will be aware of it.
 
When you file a new bug for a crash signature, that bug won't show up in the "Bugs" columns of crash-stats for a few hours. Which is a shame, because it would be nice to double-check the linking immediately, but it's understandable that crash-stats doesn't ping Bugzilla all the time.
 
== What if a Nightly build is super crashy? ==
 
Occasionally a Nightly build is super crashy. When this happens, there are a few things to be done.
* First, tell an appropriate person (ping in #releng or maybe #moc) so that updates can be stopped as soon as possible.
* Then, get a tracking bug (Release Engineering :: General) filed, either by you or the person stopping the updates.
* Then, if the regressing patch can be identified, get it backed out from mozilla-central.
* Then, consider asking an appropriate person (via #sheriffs, #taskcluster, #releng, or the [https://mail.mozilla.org/listinfo/sheriffs mailing list]) to "respin" Nightly, i.e. rebuild and re-release it. This takes a couple of hours, so is only worthwhile if the next day's Nightly build is still some time off.
* Finally, once the issue has been resolved and new Nightly builds are ready to go out to users, contact the person who froze updates in step 1 so that updates can be unfrozen.
 
== Miscellaneous ==
 
At the end of each development cycle, it's worth paying attention to which crashes affect both Nightly and Beta, and therefore have fixes that need uplift to Beta.

Revision as of 01:33, 21 July 2017

The contents of this page have moved to [NightlyCrashTriage].

Retrieved from "https://wiki.mozilla.org/index.php?title=Platform/Uptime/NightlyCrashAnalysis&oldid=1176206"