CrashKill/Crashr
Jump to navigation
Jump to search
Crash[r] -- Little snapshots of crash metrics on beta and final releases to help compare them.
General Info
This is temporary repository for overall crash trend analysis until we can get better reporting systems in place.
- release dates from https://wiki.mozilla.org/Releases/
- adu's from metrics
- crash data from soccoro custom slices
- some releases are throttled. throttle assumes 10% client throttling, then 15% server processing or ((adus * .10) * .15)
- "williness to submit" follows what ever the defaults are (this assumption needs testing) early beta testers might be more willing to submit but we don't account for that complexity yet. maybe we can back out willness to submit after many release studies.
Here's the funny stuff that is missing.
Other observations and learning from the data
- crashes per hundred users goes crazy on release days when there are two many moving parts in the metrics and crash reporting systems.
- 3.0.14 was more stable than 3.5.3
- we started to get some more consistent daily measurements on 3.6b1 once we got 200-250k users
- Here is a picture of how throttling works, and some estimates on user populations of large and small releases.
- maybe 500k beta users, even throttled, is better than 200k unthrottled users in predicting stability; maybe confirming the theory that a bigger starting pool of users participating in the testing helps the numbers be more predictive.
- 200-250k users on 3.6b1 may have helped the numbers be much more consistent if not predictive (time will tell on the predictive part).
- we need to get 3.6b2 up to the level to try and do comparisons soon.
- having many users on the same release for at least a week or two helps week out outliers in daily or event fluctuations and gives time to make more sense of the data.
- on days when we have seen 250k throttled or raw adus of an unthrottled release the number of #crashes, #signatures, and crashes per 100 users are all still pretty noisy.
release throt adu date #crash #sig per/100 adj'ed 3.5.3 277241 18482710 20090911-crashdata.csv 34912 8799 0.19 12.59 3.0.14 221928 14795188 20090911-crashdata.csv 15634 4669 0.11 7.04 3.0.14 227750 15183311 20090912-crashdata.csv 28298 7328 0.19 12.43 3.6b1 -> 241000 20091109-crashdata.csv 16934 2694 7.03 <- 3.6b2 -> 200882 20091115-crashdata.csv 16223 2572 8.08 <- 3.6b2 -> 241726 20091116-crashdata.csv 16209 2627 6.71 <-
- The numbers in the 3.5.5 section https://wiki.mozilla.org/CrashKill/Crashr#Release_3.5.5 sugggest an interesting idea.
((47million_adus * 10%_client_throttle) * 15%_server_throttle)=700k users_willing&enabled-to submit crashes
if we had 700k beta testers we wouldn't have the diversity of users as the major 40million user release has, but for "common" crashes that everyone sees about equally we ought to see about the same volume. If these theories hold true that would remove one area of risk, and make comparisons a lot easier.
- we still may need to adjust willingness to submit in the case of throttling or larger user base, or seek a broader testing pool and more users participating. any ideas on how we might back out a "williness to submit" are welcome.
Release Snapshots
Release 3.6
Release 3.6b1
unthrottled
throttl adus date #crashes $sigs crashs/100users adjusted
25 200910132349 0 0 0.00
4000 200910142354 8 8 0.20
18000 200910152349 196 101 1.09
24000 200910162349 353 177 1.47
28000 200910172349 339 195 1.21
40000 200910182350 453 232 1.13
46000 200910192349 486 257 1.06
49000 200910202351 476 229 0.97
49000 200910212349 396 218 0.81
45000 200910222349 436 245 0.97
45000 200910232349 416 207 0.92
53000 200910242349 485 248 0.92
54000 200910252352 468 252 0.87
54000 200910262349 474 249 0.88
55000 200910272349 410 230 0.75
54000 200910282349 453 249 0.84
55000 200910292349 527 266 0.96
54000 200910302349 544 297 1.01
65000 200910312349 1049 418 1.61 b1 released
89000 200911012350 1468 500 1.65
123000 200911022349 1939 601 1.58
158000 200911032353 2177 645 1.38
180000 200911042352 2321 729 1.29
198500 200911052354 5962 1364 3.00
213000 200911062349 14917 2537 7.00
198000 200911072349 15187 2685 7.67
209000 200911082349 16679 2814 7.98
241000 200911092349 16934 2694 7.03
248000 200911102349 16092 2653 6.49 b2 released
231874 200911112354 11563 2222 4.99
147085 200911122350 6994 1561 4.76
109913 200911132354 5876 1417 5.35
87168 200911142354 5204 1265 5.97
78135 200911152353 4563 1164 5.84
83892 200911162354 4206 1155 5.01
77983 200911172354 3925 1067 5.03
73494 200911182353 3700 1108 5.03
71089 200911192354 3417 1018 4.81
67131 200911202354 3160 912 4.71
61912 200911212354 3162 943 5.11
58483 200911222354 3162 988 5.41
67985 200911232354 3124 947 4.60
66186 200911242354 2949 952 4.46
63465 200911252354 2963 904 4.67
62903 200911262354 2780 877 4.42
59665 200911272354 2687 872 4.50
54798 200911282354 2952 953 5.39
52991 200911292354 2818 947 5.32
Release 3.6b2
unthrottled
throttle adu's date #crashes #sigs crash/100users 200911072349 0 0 200911082349 0 0 169 200911092349 29 15 17.16 916 200911102349 773 229 84.39 27326 200911112354 7240 1323 26.49 128715 200911122350 10676 1847 8.29 171291 200911132354 13535 2273 7.90 177005 200911142354 14792 2445 8.36 200882 200911152353 16223 2572 8.08 241726 200911162354 16209 2627 6.71 257010 200911172354 15311 2787 5.96 253166 200911182353 10641 2277 4.20 143473 200911192354 5066 1437 3.53 84244 200911202354 3082 934 3.66 58037 200911212354 2473 859 4.26 45560 200911222354 1979 674 4.34 44188 200911232354 1884 651 4.26 36710 200911242354 1422 529 3.87 31747 200911252354 1272 479 4.01 28199 200911262354 1314 506 4.66 25474 200911272354 1286 462 5.05 23976 200911282354 1171 469 4.88 23312 200911292354 1128 454 4.84
Release 3.6b3
throt adu's date #crashes #sigs crash/100users
? 200911142354 0 0
? 200911152353 1 1
? 200911162354 18 8
? 200911172354 309 101
18435 200911182353 5028 1329 27.27
142847 200911192354 10331 2402 7.23
207349 200911202354 11423 2647 5.51
217975 200911212354 12906 2828 5.92
243541 200911222354 13796 3074 5.66
294307 200911232354 14574 3112 4.95
321004 200911242354 14927 3332 4.65
319230 200911252354 15826 3449 4.96
313303 200911262354 12941 3043 4.13
227788 200911272354 6720 1858 2.95
111492 200911282354 3578 1168 3.21
80372 200911292354 2727 970 3.39
Release 3.6b4
not throttled
throt adu's date #crashes #sigs crash/100users 200911242354 0 0 159 200911252354 37 24 23.27 11003 200911262354 3217 1100 29.24 101832 200911272354 11267 2984 11.06 208895 200911282354 14040 3561 6.72 262879 200911292354 16181 3992 6.16 318380 200911302354 16560 4035 5.20
Release 3.5.x
Release 3.5.6pre
throt adu's date #crashes #sigs crash/100users adj'ed 0 200911022349 0 0 #DIV/0! #DIV/0! 0 1 200911032353 1 1 100.00 6666.67 2 108 200911042352 3 3 2.78 185.19 10 686 200911052354 8 5 1.17 77.75 16 1050 200911062349 21 16 2.00 133.33 51 3392 200911072349 59 37 1.74 115.96 78 5219 200911082349 40 26 0.77 51.10 103 6845 200911092349 22 14 0.32 21.43 132 8777 200911102349 34 24 0.39 25.83 147 9797 200911112354 21 17 0.21 14.29 157 10458 200911122350 26 20 0.25 16.57 169 11270 200911132354 51 33 0.45 30.17 165 11012 200911142354 29 19 0.26 17.56 139 9274 200911152353 21 17 0.23 15.10 180 12001 200911162354 47 33 0.39 26.11 186 12424 200911172354 32 26 0.26 17.17 192 12780 200911182353 31 27 0.24 16.17 192 12772 200911192354 27 21 0.21 14.09 186 12367 200911202354 23 18 0.19 12.40 155 10315 200911212354 34 18 0.33 21.97 160 10671 200911222354 36 19 0.34 22.49 199 13258 200911232354 38 32 0.29 19.11
Release 3.5.5
throttle assumes 10% client throttling, then 15% server processing or ((adus * .10) * .15)
throt adus date #crashes #sigs crash/100users throttle adj'ed 0 200911012350 0 0 0 200911022349 1 1 2 138 200911032353 0 0 0.00 0.00 4 295 200911042352 3 3 1.02 67.80 129 8588 200911052354 3548 1580 41.31 2754.23 33912 2260795 200911062349 41899 9698 1.85 123.55 321576 21438378 200911072349 81941 15360 0.38 25.48 455219 30347934 200911082349 95888 17386 0.32 21.06 589113 39274168 200911092349 100690 18062 0.26 17.09 657614 43840924 200911102349 104768 18557 0.24 15.93 684712 45647479 200911112354 110074 19201 0.24 16.08 714202 47613465 200911122350 100894 18127 0.21 14.13 704354 46956936 200911132354 107604 18872 0.23 15.28 619366 41291046 200911142354 104979 18519 0.25 16.95 654919 43661269 200911152353 111029 19122 0.25 16.95 773046 51536376 200911162354 115214 19656 0.22 14.90 786625 52441671 200911172354 124840 20048 0.24 15.87 793275 52885030 200911182353 119438 19943 0.23 15.06 793123 52874870 200911192354 115156 19858 0.22 14.52 767605 51173636 200911202354 112276 19719 0.22 14.63 672077 44805118 200911212354 109937 19335 0.25 16.36 699671 46644737 200911222354 117899 20390 0.25 16.85 826446 55096414 200911232354 120234 20916 0.22 14.55
Release 3.5.4
throttle assumes 10% client throttling, then 15% server processing or ((adus * .10) * .15)
throt adu's date #crashes #sigs crash/100users adj'ed 5975 398311 20091013-crashdata.csv 1457 793 0.37 24.39 6575 438328 20091014-crashdata.csv 1718 900 0.39 26.13 6815 454324 20091015-crashdata.csv 1597 780 0.35 23.43 6795 452985 20091016-crashdata.csv 1635 809 0.36 24.06 5932 395453 20091017-crashdata.csv 1653 843 0.42 27.87 6133 408839 20091018-crashdata.csv 1706 867 0.42 27.82 7360 490677 20091019-crashdata.csv 1819 896 0.37 24.71 7492 499473 20091020-crashdata.csv 1747 910 0.35 23.32 7533 502184 20091021-crashdata.csv 1695 905 0.34 22.5 7512 500774 20091022-crashdata.csv 1740 876 0.35 23.16 7275 484989 20091023-crashdata.csv 1795 912 0.37 24.67 6334 422238 20091024-crashdata.csv 1738 906 0.41 27.44 6515 434366 20091025-crashdata.csv 1852 908 0.43 28.42 7680 511986 20091026-crashdata.csv 1730 895 0.34 22.53 7730 515329 20091027-crashdata.csv 2825 1310 0.55 36.55 27357 1823786 20091028-crashdata.csv 34007 8190 1.86 124.31 -release 344243 22949509 20091029-crashdata.csv 79135 15053 0.34 22.99 488519 32567919 20091030-crashdata.csv 92431 16607 0.28 18.92
An example of extreme growth in a regression crash causing a respin of 3.5.4
https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20status1.9.1%3A.5-fixed
6 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091020-crashdata 5 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091021-crashdata 4 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091022-crashdata 9 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091023-crashdata 14 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091024-crashdata 10 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091025-crashdata 9 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091026-crashdata 33 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091027-crashdata 3.5.4 released -- 506 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091028-crashdata 1034 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091029-crashdata 712 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091030-crashdata 556 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091031-crashdata 597 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091101-crashdata 540 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091102-crashdata 474 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091103-crashdata 404 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091104-crashdata 371 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091105-crashdata 249 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091106-crashdata
On 20091024 just a few days before shipping that signature ranked #85 with 3 crashes that day.
85. 3 gfxWindowsFontGroup::WhichFontSupportsChar(nsTArray<nsRefPtr<FontEntry> > const&, unsigned int) 3.5.4
20091029 ranking
2. 1029 gfxWindowsFontGroup::WhichFontSupportsChar(nsTArray<nsRefPtr<FontEntry> > const&, unsigned int) 3.5.4
Release 3.5.3
throt adu's date #crashes #sigs crash/100users adj'ed 22 1466 20090901-crashdata.csv 7 5 0.48 31.83 719 47955 20090902-crashdata.csv 52 46 0.11 7.23 4959 330590 20090903-crashdata.csv 744 492 0.23 15 6180 411973 20090904-crashdata.csv 1319 734 0.32 21.34 5769 384603 20090905-crashdata.csv 1584 856 0.41 27.46 6151 410099 20090906-crashdata.csv 1444 802 0.35 23.47 7407 493812 20090907-crashdata.csv 1534 789 0.31 20.71 7965 530986 20090908-crashdata.csv 1625 892 0.31 20.4 8476 565087 20090909-crashdata.csv 1703 937 0.3 20.09 -- release 44559 2970600 20090910-crashdata.csv 5219 2081 0.18 11.71 277241 18482710 20090911-crashdata.csv 34912 8799 0.19 12.59 291479 19431957 20090912-crashdata.csv 61423 13297 0.32 21.07 382936 25529036 20090913-crashdata.csv 72537 14862 0.28 18.94 505111 33674054 20090914-crashdata.csv 87536 17086 0.26 17.33 553302 36886822 20090915-crashdata.csv 95342 18220 0.26 17.23 575617 38374498 20090916-crashdata.csv 97072 18528 0.25 16.86 588830 39255365 20090917-crashdata.csv 98769 18508 0.25 16.77 574876 38325034 20090918-crashdata.csv 98938 18718 0.26 17.21 490791 32719389 20090919-crashdata.csv 97003 18655 0.3 19.76 525336 35022427 20090920-crashdata.csv 94472 18103 0.27 17.98 621540 41436022 20090921-crashdata.csv 102442 19071 0.25 16.48 636066 42404407 20090922-crashdata.csv 106806 19752 0.25 16.79 645549 43036631 20090923-crashdata.csv 106061 19523 0.25 16.43 648983 43265543 20090924-crashdata.csv 106873 19798 0.25 16.47 628283 41885531 20090925-crashdata.csv 105275 19760 0.25 16.76 541861 36124056 20090926-crashdata.csv 109329 19166 0.3 20.18 570765 38051029 20090927-crashdata.csv 110597 19062 0.29 19.38 676617 45107781 20090928-crashdata.csv 108851 19565 0.24 16.09 684282 45618770 20090929-crashdata.csv 110626 20073 0.24 16.17 18.27
Release 3.0.x
Release 3.0.14
throt adu's date #crashes #sigs crash/100users adj'ed 4 299 20090901-crashdata.csv 1 1 0.33 22.30 133 8881 20090902-crashdata.csv 10 9 0.11 7.51 1113 74179 20090903-crashdata.csv 111 92 0.15 9.98 1435 95637 20090904-crashdata.csv 154 123 0.16 10.74 1329 88567 20090905-crashdata.csv 211 155 0.24 15.88 1441 96037 20090906-crashdata.csv 198 150 0.21 13.74 1853 123525 20090907-crashdata.csv 195 160 0.16 10.52 1967 131110 20090908-crashdata.csv 197 150 0.15 10.02 2062 137469 20090909-crashdata.csv 224 184 0.16 10.86 - release 24712 1647489 20090910-crashdata.csv 1405 741 0.09 5.69 221928 14795188 20090911-crashdata.csv 15634 4669 0.11 7.04 227750 15183311 20090912-crashdata.csv 28298 7328 0.19 12.43 311923 20794866 20090913-crashdata.csv 32374 8636 0.16 10.38 432973 28864876 20090914-crashdata.csv 41727 10312 0.14 9.64 480558 32037233 20090915-crashdata.csv 46357 10951 0.14 9.65 500022 33334831 20090916-crashdata.csv 47023 11157 0.14 9.40 510106 34007050 20090917-crashdata.csv 47409 11296 0.14 9.29 490892 32726129 20090918-crashdata.csv 47116 11218 0.14 9.60 406300 27086687 20090919-crashdata.csv 45673 11156 0.17 11.24 439670 29311359 20090920-crashdata.csv 43943 10913 0.15 9.99 535907 35727115 20090921-crashdata.csv 48341 11631 0.14 9.02 544515 36301033 20090922-crashdata.csv 49944 11692 0.14 9.17 549171 36611388 20090923-crashdata.csv 49985 11796 0.14 9.10 548170 36544655 20090924-crashdata.csv 49626 11699 0.14 9.05 524175 34945024 20090925-crashdata.csv 48913 11522 0.14 9.33 438887 29259151 20090926-crashdata.csv 58846 11362 0.2 13.41 465577 31038440 20090927-crashdata.csv 61053 11084 0.2 13.11 569360 37957328 20090928-crashdata.csv 53189 11947 0.14 9.34 570608 38040564 20090929-crashdata.csv 52586 12059 0.14 9.22 10.57