Release Management/WER Investigation

This page will track the initial investigation of Windows Error Reporting (WER). Our current plan is to hook our bug tracking and metrics into Winqual, the site/webAPI used to access Microsoft's crash info. Using this info will help diagnose hangs (which aren't currently tracked) on Windows XP and later, and may even uncover new crashes.

Areas of Interest

• Terms associated with using Winqual (data after 6mos, etc.)
• Creating a library/service to act as a
  • Sync layer between Winqual and our crash storage service
  • Method of providing symbols/app version/file mappings for new builds
  • Auto-filer for Bugzilla, possibly dealing with dupes
  • Provider for Socorro crash metrics
• Applicability for other platforms such as Mac OS X

Associated Bugs

• bug 429592 - discussion of what to do with process hangs
• bug 600275

Resources

• Short description on MSDN: WER: Getting Started
• Longer guide (more targeted at hw/drivers though): Developers Guide to WER
• Microsoft's WER client: Stackhash
• API documentation: StackHash source download > 3rdparty > WinQual API > Data Services.docx

Needed for Investigation

• VMWare/Windows 7 (bug needed)
• Winqual account

Questions that need Answering

• What needs to be done to creating the file mapping XML files from each build?
• What information is actually provided by Winqual (on the site, in the REST data, and in the CAB files)?
• How will we symbolicate the crash dumps?
  • The Developers Guide mentions that "Microsoft encourages people to submit their symbols when they submit drivers to be signed". Can we take advantage of that?
• How much overlap is there with current crash tracking at Mozilla?
• MS: Will we run into any limits if staying synchronized with the server?
• MS: Any limitations based upon the number of builds (delays, etc). Especially necessary if we decide to track nightlies.
• How does debugging using minidump files work?
  • Requires both the associated symbols as well as the associated images.
• Crash/hang "buckets" only collect CAB files for the first few instances of the crash - what does that mean for debugging?
  • In the Developers Guide, it mentions "We can collect additional files on request if you find that you need more information for debugging."

Future Investigations

• Consider providing a solution (link) when the user is presented with the Windows crash dialog
  • Can even link to an exe (if part of the "Designed for Windows" logo program), which may be a good idea for last ditch effort if even Firefox's safe mode fails (application files no longer pristine, need reinstall).