CrashKill/2012-09-04

Welcome to the first Stability Newsletter! For some time now, the crashkill team has been thinking about publishing a regular newsletter to the broader Mozilla community. The goal is to engage users and developers in helping us improve the stability across all our products.

• The newsletter will cover a wide range of topics relating to crashes and crash analysis.
• We might focus on a particular problem, a specific product, discuss crash trends or simply provide a list of bugs we want help reproducing. The newsletter will be published twice during a full release cycle.
• The first one will go out on the Friday a week after we merge. This enables us to report on top problems once we have enough users on the first build for that channel.
• The second newsletter will go out later on during the cycle to highlight crashes or crash spikes on Beta. This might be a call to action for help reproducing tough issues.

Stability 101

Since this is our first newsletter, we thought it would be good to cover a few basics for those who may not be as familiar with crashkill and our activities.

• Meetings and Communications
  • The crashkill team meets weekly on Monday morning at 10:00am PT. Those in MV meet in Zombocom-3Z and remote folks dial into the Zombocom vidyo room. (NOTE:Setup crashkill vidyo room)
  • Organized crash triage is on an ad-hoc and as needed basis. Daily crash chatter happens in IRC. See IRC channel info below.
  • Members of the crashkill team attend and report on stability during the twice weekly Channel Meeting (Tuesday/Thursday @ 2:00pm PT) as well as the weekly Firefox Platform Meeting (Tuesday @ 11:00am PT).
• Queries and Tools
  • We use Socorro as our primary tool to run reports, search and track stability issues.
  • We often have the need to generate a number of manual reports to help analyze and diagnose a particular issue. Kairo has many in his own directory.
  • Manual reports are often temporary until we can integrate them into Socorro. Other times we need a manual report to track a short-term issue.
  • We used a number of keywords in bugzilla to track crashes - crash, topcrash, regression, reproducible and hang.
  • We also used a wide range of whiteboard tags to help us inventory crashes. These are often less consistent. Some like [startupcrash] are more recent and some like [crashkill] are old and not used that much any more. We will often use temporary tags to track crashes that correlate to a particular OS release, [Win8] for example.
• Reporting Crashes
  • If you crash, please select to send the report. Adding comments is also very helpful.
  • If you log a bug for a crash...
    • Include the format - crash in "top signature stack" - in the bug title. It will look something like - crash in JSContext::saveFrameChain.
    • Please include the crash keyword in the keyword field
    • We have a crash signature field which links bugs to signatures in Socorro. The format is a list of signatures like [@ signature], ie: [@ JSContext::saveFrameChain]. There might be multiple different signatures that associate back to the same bug.
• Wikis
  • The crashkill wiki is a pointer to meeting notes, queries, reports and other analysis.
• IRC
  • We hang out in #crashkill on irc
  • The Socorro folks hang out in #breakpad

Desktop

Flash

The increase in Flash related crashes with the release of Flash 11.3 continues to be a problem.

• We have been working closely Adobe to address these. Subsequent updates to 11.3 fixed a number of problems.
• We are tracking the crash rate closely - https://crash-analysis.mozilla.com/bsmedberg/flash-summary.html
• The adoption of 11.4 released last week is still relatively low so we haven't yet been able to measure the full impact.
• Hangs continue to be a major issue and we will be tackling this more aggressively.
• Our top Flash crashes are...(placeholder list of bugs)
• Out top Flash hangs are...(placeholder list of bugs)

Windows 8

• Talk about progress on Windows 8 stability issues
• How users can tag bugs
• What we need help testing...etc.

Tracking Aurora - Fx17

A week ago we merged the Fx17 train from m-c to Aurora. This is a list of topcrashes we are currently tracking for Fx17. We would use help isolating regression ranges or identifying reproducible steps.

• (List of top crashes we want help with)...

Tracking Beta - Fx16

Our first beta build is out with xxx users. Here are the top signatures we are tracking and could use help investigating and reproducing.

• (List of top crashes we want help with)...

Mobile

FirefoxOS

Some of you may have noticed a new Product listed in Socorro - B2G.

• As of today, we only support crashes generated from the Desktop builds and not device builds.
• There are no crashes logged yet but this shows we have some of the mechanics in place already.
• We are aiming to have device crash reporting up and running.
• ...getting more updates from team in Brazil