CrashKill/RapidReleaseCrashes

From MozillaWiki
Jump to: navigation, search
Feature Status ETA Owner
Top crashes and crashes per user by build id and channel] Putting together detailed spec. TBD Sheila Mooney

Summary

After completing a first release of Firefox using the new rapid release model, we have identified a number of modifications that have to be made in Socorro to help us more accurately track the crash rate and top crashes. Mostly this is due to the addition of a new channel and some changes to how we track versions during the beta and release cycle. After having gone through a first cycle with the release of Firefox 5, we have a better handle on what kind of scenarios we need to support.

  • We now use a single version number throughout the beta/release cycle.
    • We need to be able to track the crash rate and top crashes on a more granular basis by each beta We only display the top crashes for all builds across a single version.
    • Since we are no longer going to rebuild for a release, the last build id that is used spans both beta and release channels where the throttling changes. This makes the data confusing so need to differentiate the version on a specific channel.
  • We have an additional channel - Aurora
    • We have been treating this similar to Nightly where the crash rate and top crashes are a sum of what is happening across all the various builds. As we build that user base it might be worth comparing crashes and crash rate from one build to the next in the channel. It enables us to quickly validate that fixing a crash in a particular build, has the desired affect.
  • Nightly and Aurora data usually shows the crashes per user and top crash list for all crashes that occurred during that period across all builds. Really we want to see how many crashes there have been across builds dated in the past 1day, 3days etc.

Related Bugs & Dependencies

A couple of bugs are being used to track this feature.

Use Cases

  • Trunk
    • Show the top crash list for builds done in a specific time period.
      • The last 1 day, 3 days, 7 days, 14 days, 28 days.
    • Show the crashes per user for all crashes in builds dated within a specific date range.
      • Pick a date range and show crashes per user for all builds dated within that range.
  • Aurora
    • Show the top crash list for builds done in a specific time period.
      • The last 1 day, 3 days, 7 days, 14 days, 28 days.
    • Show the crashes per user for all crashes in builds dated within a specific date range.
      • Pick a date range and show crashes per user for all builds dated within that range.
  • Beta
    • Show top crash lists for a specific beta.
    • Show the crashes per user for a specific beta.
    • Compare the crashes per user for version X beta A and version X beta B
    • Compare the crashes per user for version X and version Y
    • Compare the crashes per use for version X beta A and version Y beta A
    • Show the top crash list for all crashes for a particular version.
    • Show the daily crashes per user for a specific version.
  • Release
    • Show the top crash list for all crashes of a particular version on the release channel.
    • Show the crash rate for a particular version on the release channel.
    • Compare the crash rate for version X on the release channel to version Y on the release channel.

Designs & Details

Version list

  • For Trunk and Aurora, we have distinct version numbers; 5.0a1, 5.0a2.
    • When our train hits the Beta channel, we have only a single version number ie:5.0 and unlike past releases, do not have a separate version for each Beta on the channel.
    • Since we want to be able to track the top crashes and crashes per user from one beta to the next, we need to group together the version and build id to create a beta version display name.
  • In the version list that appears throughout Socorro, we should have distinct entries for each version + build id combination ie: 5.0b1, 5.0b2, 5.0b3...etc.
    • This enables us to easily look at top crashers as well as compare crashes per user.
    • We also have the ability to have only the most recent beta display on the home page.
  • The version when it progresses to the release channel can be simply 5.0.

Home page (overview report)

We currently show up to 4 versions on the home page. These are configured using the admin screen. The graphs displayed and associated top crashes and links represent all crashes across all builds of that version.

  • Ideally the 4 current releases, represent the 4 trains; trunk, aurora, beta, release. One possibility would be to display the channel names and the version in brackets ie: Aurora (6.0a2).
  • The admin UI would have the list of versions described below. We could then choose any combinations of versions to display but it makes most sense to use whatever is currently on all the channels.
  • Clicking the list to top crashes and other links just drills down based on the current filter
    • For trunk and aurora we would show the crashes and crashes per user in builds dated in the last 7 days (or whatever the default is).

Crashes per user report

  • The crashes per user report would work the same as today. It's the more granular list of versions that combine version + build id and channel that will enable us to run all the various side-by-side comparisons.

Advanced filters

  • Using the advanced filters you can isolate a top crash list for any version and build id combination (this works today).
    • I don't think we would ever need to filter on channel here. The only situation where channel comes into play is on release but the current versions would always give you the top crashes by channel anyhow. I don't see why we would want the top crashes for betas once we have released.
  • Using the reports fields on the crashes per user page allows you to show crashes per user for any build id, version and channel combination.
  • Unless there is a need to specify multiple build ids, there are no changes to the filters.

Legend (remove if you like)

  Healthy: feature is progressing as expected.
  Blocked: feature is currently blocked.
  At Risk: feature is at risk of missing its targeted release.
ETA Estimated date for completion of the current feature task. Overall ETA for the feature is the product release date.