View source for CrashKill/Crashr

Crash[r]  --  Little snapshots of crash metrics on beta and final releases to help compare them.

this page has now been replaced with a charting tool built into http://crash-stats.mozilla.com/daily

here is what to play with

http://crash-stats.mozilla.com/daily?form_selection=by_version&p=Firefox&v[]=3.6&throttle[]=15&v[]=3.5.8&throttle[]=3.75&v[]=3.0.18&throttle[]=3.75&v[]=&throttle[]=100&os[]=Windows&os[]=Mac&os[]=Linux&date_start=2010-02-21&date_end=2010-03-07&submit=Generate 

for 3.0.x and 3.5.x
25% of users opt in to sending (10% have box checked and another 15% check it)  times 15% server throttle  = 3.75% total effective throttle rate

for 3.6
its no client throttling and 15% server throttle since 1/29
full history of throttling adjustments for 3.6 is at
https://wiki.mozilla.org/CrashKill/Crashr#3.6_RC1.2C_RC2.2C_Final




= General Info =

This is temporary repository for overall crash trend analysis until we can get better reporting systems in place.

* release dates from https://wiki.mozilla.org/Releases/
* adu's from metrics
* crash data from soccoro custom slices
* some releases are throttled.  throttle assumes 10% client throttling, then 15% server processing or ((adus * .10) * .15)  
* "williness to submit" follows what ever the defaults are (this assumption needs testing)  early beta testers might be more willing to submit but we don't account for that complexity yet.  maybe we can back out willness to submit after many release studies.

[http://wiki.mozilla.org/CrashKill/Crashr/Funny Here's] the funny stuff that is missing.

= Other observations and learning from the data =

* crashes per hundred users should be ignored in the first couple days after release.  while crash data is reported in real-time, ADU pings suffer from a delay of one or two days, which means that when there's a massive increase in usage in the day or two after a release, ADU data will be behind (crash data won't be) and the ratio will be way off.
* 3.0.14 was more stable than 3.5.3
* we started to get some more consistent daily measurements on 3.6b1 once we got 200-250k users
* Here is a picture of how throttling works, and some estimates on user populations of large and small releases. 
**  https://wiki.mozilla.org/File:IMG_2248.jpg 
* maybe 500k beta users, even throttled, is better than 200k unthrottled users in predicting stability; maybe confirming the theory that a bigger starting pool of users participating in the testing helps the numbers be more predictive.
* 200-250k users on 3.6b1 may have helped the numbers be much more consistent if not predictive (time will tell on the predictive part).
* we need to get 3.6b2 up to the level to try and do comparisons soon.
* having many users on the same release for at least a week or two helps week out outliers in daily or event fluctuations and gives time to make more sense of the data.

* on days when we have seen 250k throttled or raw adus of an unthrottled release the number of #crashes, #signatures, and crashes per 100 users are all still pretty noisy.  

<pre>

release throt    adu      date                  #crash #sig     per/100 adj'ed

3.5.3  277241	18482710 20090911-crashdata.csv	34912	8799	0.19	12.59
3.0.14 221928	14795188 20090911-crashdata.csv	15634	4669	0.11	7.04
3.0.14 227750	15183311 20090912-crashdata.csv	28298	7328	0.19	12.43
3.6b1  	 ->       241000 20091109-crashdata.csv	16934	2694	7.03     <-
3.6b2    ->       200882 20091115-crashdata.csv	16223	2572	8.08     <-
3.6b2    ->	  241726 20091116-crashdata.csv	16209	2627	6.71     <-


</pre>

* The numbers in the 3.5.5 section https://wiki.mozilla.org/CrashKill/Crashr#Release_3.5.5 sugggest an interesting idea.
 
 ((47million_adus * 10%_client_throttle) * 15%_server_throttle)=700k users_willing&enabled-to submit crashes

if we had 700k beta testers we wouldn't have the diversity of users as the major 40million user release has, but for "common" crashes that everyone sees about equally we ought to see about the same volume.  If these theories hold true that would remove one area of risk, and make comparisons a lot easier.

* we still may need to adjust willingness to submit in the case of throttling or larger user base, or seek a broader testing pool and more users participating. any ideas on how we might back out a "williness to submit" are welcome.

= Release Snapshots =

== Release 3.6 ==

=== Release 3.6b1 ===
unthrottled
<pre>
throttl adus	date	  #crashes   $sigs	crashs/100users	adjusted
	25	200910132349	0	0	0.00	
	4000	200910142354	8	8	0.20	
	18000	200910152349	196	101	1.09	
	24000	200910162349	353	177	1.47	
	28000	200910172349	339	195	1.21	
	40000	200910182350	453	232	1.13	
	46000	200910192349	486	257	1.06	
	49000	200910202351	476	229	0.97	
	49000	200910212349	396	218	0.81	
	45000	200910222349	436	245	0.97	
	45000	200910232349	416	207	0.92	
	53000	200910242349	485	248	0.92	
	54000	200910252352	468	252	0.87	
	54000	200910262349	474	249	0.88	
	55000	200910272349	410	230	0.75	
	54000	200910282349	453	249	0.84	
	55000	200910292349	527	266	0.96	
	54000	200910302349	544	297	1.01	
	65000	200910312349	1049	418	1.61  b1 released	
	89000	200911012350	1468	500	1.65	
	123000	200911022349	1939	601	1.58	
	158000	200911032353	2177	645	1.38	
	180000	200911042352	2321	729	1.29	
	198500	200911052354	5962	1364	3.00	
	213000	200911062349	14917	2537	7.00	
	198000	200911072349	15187	2685	7.67	
	209000	200911082349	16679	2814	7.98	
	241000	200911092349	16934	2694	7.03	
	248000	200911102349	16092	2653	6.49  b2 released
	231874	200911112354	11563	2222	4.99	
	147085	200911122350	6994	1561	4.76	
	109913	200911132354	5876	1417	5.35	
	87168	200911142354	5204	1265	5.97	
	78135	200911152353	4563	1164	5.84	
	83892	200911162354	4206	1155	5.01	
	77983	200911172354	3925	1067	5.03	
	73494	200911182353	3700	1108	5.03
	71089	200911192354	3417	1018	4.81  b3 released
	67131	200911202354	3160	 912	4.71
        61912	200911212354	3162	 943	5.11
	58483	200911222354	3162	 988	5.41
	67985	200911232354	3124	 947	4.60
	66186	200911242354	2949	 952	4.46
	63465	200911252354	2963	 904	4.67
	62903	200911262354	2780	 877	4.42  b4 released
	59665	200911272354	2687	 872	4.50
	54798	200911282354	2952	 953	5.39
	52991	200911292354	2818	 947	5.32
	61073	200911302354	2771	918	4.54
	60188	200912012354	2894	944	4.81
	59034	200912022354	3044	906	5.16
	58093	200912032354	2915	853	5.02

</pre>

=== Release 3.6b2 ===
unthrottled
<pre>
throttle adu's	date	     #crashes	#sigs crash/100users
		200911072349	    0	   0	
		200911082349	    0	   0	
	   169	200911092349	   29	  15	17.16
	   916	200911102349	  773	 229	84.39
	 27326	200911112354	 7240	1323	26.49
	128715	200911122350	10676	1847	8.29
	171291	200911132354	13535	2273	7.90
	177005	200911142354	14792	2445	8.36
	200882	200911152353	16223	2572	8.08
	241726	200911162354	16209	2627	6.71
	257010	200911172354	15311	2787	5.96
	253166	200911182353	10641	2277	4.20
	143473	200911192354	 5066	1437	3.53
	 84244	200911202354	 3082	 934	3.66
	 58037	200911212354	 2473	 859	4.26
	 45560	200911222354	 1979	 674	4.34
	 44188	200911232354	 1884	 651	4.26
	36710	200911242354	1422	529	3.87
	31747	200911252354	1272	479	4.01
	28199	200911262354	1314	506	4.66
	25474	200911272354	1286	462	5.05
	23976	200911282354	1171	469	4.88
	23312	200911292354	1128	454	4.84
	25705	200911302354	1189	474	4.63
	24856	200912012354	1176	424	4.73
	23848	200912022354	1177	443	4.94
	23111	200912032354	1247	403	5.40

</pre>

=== Release 3.6b3 ===
<pre>
throt	adu's	date	#crashes	#sigs	crash/100users
	?	200911142354	    0	   0	
	?	200911152353	    1	   1	
	?	200911162354	   18	   8	
	?	200911172354	  309	 101	
	 18435	200911182353	 5028	1329   27.27
	142847	200911192354	10331	2402	7.23
	207349	200911202354	11423	2647	5.51
        217975	200911212354	12906	2828	5.92
        243541	200911222354	13796	3074	5.66
        294307	200911232354	14574	3112	4.95
	321004	200911242354	14927	3332	4.65
	319230	200911252354	15826	3449	4.96
	313303	200911262354	12941	3043	4.13
	227788	200911272354	6720	1858	2.95
	111492	200911282354	3578	1168	3.21
	80372	200911292354	2727	970	3.39
	79695	200911302354	2699	974	3.39
	58951	200912012354	2138	746	3.63
	47254	200912022354	1988	650	4.21
	40100	200912032354	1574	560	3.93

</pre>

=== Release 3.6b4 ===

not throttled

<pre>
throt	adu's	date	     #crashes   #sigs	crash/100users
		200911242354	    0	   0	
	   159	200911252354	   37	  24	23.27
	 11003	200911262354	 3217	1100	29.24  -- release
	101832	200911272354	11267	2984	11.06
	208895	200911282354	14040	3561	 6.72
	262879	200911292354	16181	3992	 6.16
	318380	200911302354	16560	4035	 5.20
	354012	200912012354	17445	4230	 4.93
	377984	200912022354	18626	4437	 4.93
	394451	200912032354	18419	4482	 4.67
	399269	200912042353	18337	4398	4.59
	375329	200912052354	18789	4588	5.01
	390387	200912062354	19333	4668	4.95
	447912	200912072354	20135	4824	4.50
	460269	200912082354	20275	4980	4.41
	472872	200912092353	20361	5003	4.31
	479922	200912102353	20653	5033	4.30
	464588	200912112354	20990	5248	4.52
	446400	200912122354	20683	5165	4.63
	459436	200912132354	21457	5397	4.67
	529838	200912142354	22406	5425	4.23
	533851	200912152354	22727	5729	4.26
	539636	200912162354	21925	5448	4.06
	536638	200912172354	21206	5421	3.95
	482650	200912182354	13889	4006	2.88
	244476	200912192354	7096	2586	2.90
	161075	200912202354	4999	2040	3.10

</pre>

=== Relases 3.6b5 ===

<pre>
    	adu's	date	    #crashes	#sigs	crash/100users	
					
	1	200912042353	0	0	
	135	200912052354	3	3	2.22
	556	200912062354	28	20	5.04
	1199	200912072354	65	41	5.42
	1874	200912082354	108	61	5.76
	2361	200912092353	69	49	2.92
	2864	200912102353	104	66	3.63
	3223	200912112354	125	80	3.88
	3187	200912122354	125	94	3.92
	3607	200912132354	575	123	15.94
	4746	200912142354	136	84	2.87
	4994	200912152354	201	109	4.02
	5122	200912162354	167	112	3.26
	6113	200912172354	985	500	16.11  --release
	50978	200912182354	7848	2604	15.39
	259273	200912192354	14558	4145	5.61
        357068	200912202354	16666	4526	4.67
	n/a	200912212354	18467	5070	#DIV/0!
	486443	200912222354	17982	4989	3.70
	489788	200912232354	18820	4992	3.84
	470922	200912242353	17752	4912	3.77
	460922	200912252354	18097	4885	3.93
	471404	200912262354	19242	5092	4.08
	487620	200912272354	20527	5396	4.21
	548337	200912282353	21708	5515	3.96
	562748	200912292354	21374	5576	3.80
	568013	200912302354	21922	5520	3.86
	528415	200912312354	18786	5128	3.56
	515642	201001012354	19605	5177	3.80
	553745	201001022354	21457	5432	3.87
	568976	201001032354	21586	5497	3.79
	637244	201001042354	23540	5735	3.69
	650022	201001052354	22468	5620	3.46
	656689	201001062354	23275	5705	3.54
	663466	201001072354	23204	5722	3.50

</pre>

=== 3.6 RC1, RC2, Final ===

<pre>
    	adu's	date	    #crashes	#sigs	crash/100users
	4	201001042354	0	0	0.00
	8	201001052354	0	0	0.00
	678	201001062354	71	46	10.47
	3093	201001072354	211	113	6.82
	12593	201001082354	1758	749	13.96  -- RC1 released
	42634	201001092354	5455	1878	12.79
	156051	201001102354	10425	3156	6.68
	303986	201001112354	15796	4277	5.20   -- RC1 announced
	455997	201001122354	20082	5155	4.40
	558918	201001132354	23174	5600	4.15
	620807	201001142354	25135	5952	4.05
	587824	201001152354	25220	5930	4.29
	628645	201001162354	26040	6207	4.14 -- RC2 released
	687117	201001172354	26920	6196	3.92
	848320	201001182354	29956	6769	3.53 -- RC2 announced
	835593	201001192354	30765	6868	3.68
	866032	201001202354	31111	7027	3.59


trying some capacity experiments see  https://bugzilla.mozilla.org/show_bug.cgi?id=541873

throt	adu's	date	      #crashes	#sigs	crash/100users	adj'ed

169746	1131644	201001212354	11534	3841	1.02	6.79 release .15 throt.
819142	2340408	201001222354	38033	8617	1.63	4.64 50% throt 1/2day or 35%
1554227	3108455	201001232354	86637	14743	2.79	5.57 50% throt full day
1432037	4091536	201001242353	82166	14378	2.01	5.74 35% throt backed off to 25% 1/2 day
1348775	5395100	201001252354	59319	11927	1.10	4.4  25% throt all day
1545632	6182530	201001262353	70773	12315	1.14	4.58 25% throt all day
1734045	6936180	201001272354	79803	13514	1.15	4.6
1899947	7599789	201001282154	71366	13522	0.94	3.76 -socorro 1.4 downtime


1139956	7599708	201001292359	51680	11315	0.68	4.53	15% throt
1197119	7980796	201001302359	45635	10609	0.57	3.81	
1116785	7445235	201001312359	47569	11062	0.64	4.26	
2054602	8218407	201002012359	52568	11521	0.64	2.56	
1501008	10006721 201002022359	57501	12230	0.57	3.83	
1687279	11248526 201002032359	54787	11839	0.49	3.25	
1762359	11749061 201002042359	60644	12972	0.52	3.44	
1785989	11906591 201002051657	48740	11414	0.41	2.73
1630856	10872370 201002062359	61327	13090	0.56	3.76
1744223	11628156 201002072359	65058	13748	0.56	3.73
2096979	13979861 201002082359	71433	14431	0.51	3.41


</pre>

=== 3.6.4 ===
<pre>

	adus	day	   Tl-rpts   num-sigs	crash	hang Crashes/100users
                                                       	           Reports/100users
		20100401	462	287	462	0		
		20100402	681	351	681	0		
		20100403	690	428	690	0		
		20100404	767	417	767	0		
		20100405	726	373	726	0		
		20100406	741	383	741	0		
		20100407	843	415	843	0		
		20100408	739	392	739	0		
		20100409	707	385	707	0		
		20100410	737	411	737	0		
		20100411	580	324	580	0		
		20100412	705	383	705	0		
		20100413	768	366	768	0		
	305	20100414	790	395	790	0	259.02	259.02
	692	20100415	787	407	787	0	113.73	113.73
	9794	20100416	1196	540	1196	0	12.21	12.21
	85949	20100418	9248	2539	9248	0	10.76	10.76
	160152	20100419	11871	3104	11871	0	7.41	7.41
	233291	20100420	12392	3221	12392	0	5.31    5.3
        270527	20100421	16781	4322	16781	0	6.2	6.2
	302005	20100422	16043	3947	16043	0	5.31	5.31
	310027	20100423	18156	4371	18156	0	5.86	5.86
	281726	20100424	15923	3907	15923	0	5.65	5.65
	297872	20100425	15844	3921	15844	0	5.32	5.32
	361567	20100426	15283	3777	15283	0	4.23	4.23
	370104	20100427	15153	3734	15153	0	4.09	4.09
	367489	20100428	15662	3873	12658	 3004	3.44	4.26 <socorro1.6>
	367367	20100429	14967	3600	4156	10811	1.13	4.07
	360720	20100430	15855	3808	4165	11690	1.15	4.4
	311825	20100501	15628	3786	4083	11545	1.31	5.01
	329985	20100502	15611	3851	4398	11213	1.33	4.73
	382206	20100503	15392	3905	5232	10160	1.37	4.03
	400797	20100504	14868	3767	5560	 9308	1.39	3.71
	405217	20100505	15784	3707	6128	 9656	1.51	3.9
	412228	20100506	17807	3852	7957	 9850	1.93	4.32
	406629	20100507	19322	4005	9275	10047	2.28	4.75
	359375	20100508	19898	3260	9738	10160	2.71	5.54
	365403	20100509	19742	3162	9812	 9930	2.69	5.4
	436228	20100510	19971	3121	9942	10029	2.28	4.58
	446153	20100511	21107	3191	10554	10553	2.37	4.73
	449961	20100512	20166	2938	10816	 9350	2.4	4.48
	440549	20100513	21073	3167	10618	10455	2.41	4.78
	442143	20100514	20931	3056	10788	10143	2.44	4.73
	389488	20100515	20097	3104	10378	 9719	2.66	5.16
	397634	20100516	18824	3230	8568	10256	2.15	4.73
	473572	20100517	17933	3162	8052	 9881	1.7	3.79
	473517	20100518	17922	3198	7540	10382	1.59	3.78



</pre>

=== 3.6.4 redo ===

the socorro 1.6 update allows a bit more fine grained analysis to breakdown where the crashes are happening.  from the .csv reports we can now determine:

<pre>
* tl_rpts are all the reports we get

* crashes
   fx_crsh = process_type ~ \N && hangid ~ \N
    p_crsh = process_type ~ [plugin] && hangid ~ \N
* hang pairs
   fx_hang = process_type ~ \N && hangid !~ \N   
    p_hang = process_type ~ [plugin] && hangid !~ \N

* tl_events subtract out about half of the hang pair report (just uses fx_hang)
* fx is the pct. of tl_events that are firefox crashes (browser restart required)
* oopp is the pct. of tl_events that are out or process (page reload/plugin restart required)


date    tl_rpts fx_crsh p_crsh  [hang pair] oopp_event tl_events  fx    oopp  oopcrsh oophang
                                 fx plugin

20100512 20169	2907	5504	[6402 5356]	11906	13767	0.211	0.865	0.400	0.465
20100513 21075	2452	6818	[5873 5932]	12691	15202	0.161	0.835	0.448	0.386
20100514 20931	2561	6867	[5709 5794]	12576	15222	0.168	0.826	0.451	0.375
20100515 20098	2689	6444	[5437 5528]	11881	14661	0.183	0.810	0.440	0.371
20100516 18826	2533	4714	[5767 5812]	10481	13059	0.194	0.803	0.361	0.442
20100517 17934	2626	4138	[5562 5608]	9700	12372	0.212	0.784	0.334	0.450
20100518 17939	2650	3642	[5800 5847]	9442	12139	0.218	0.778	0.300	0.478
20100519 19540	2679	3333	[6738 6790]	10071	12802	0.209	0.787	0.260	0.526

</pre>

more data now at 

http://people.mozilla.com/~chofmann/crash-stats/  date  / crash-counts.txt

== Other Branch Comparisons ==

=== Release 3.0.16 ===

Assumes: 
* 1 in 10 keep default opt-in
* 1.5 in 10 override default opt-out and participate in sending
* 7.5 keep default opt out


<pre>

throt	adu's	date	#crashes	#sigs	crashes/100users	adj'ed
690934	18424903	200912202354	31573	8839	0.17	4.57
882320	23528535	200912212354	33796	9350	0.14	3.83
900637	24016981	200912222354	32934	9315	0.14	3.66
878647	23430588	200912232354	31838	8980	0.14	3.62
750945	20025201	200912242353	27882	8178	0.14	3.71
684988	18266357	200912252354	27312	7937	0.15	3.99
729035	19440943	200912262354	29740	8433	0.15	4.08
761571	20308573	200912272354	32072	8922	0.16	4.21
914821	24395238	200912282353	34889	9567	0.14	3.81
918464	24492361	200912292354	34346	9626	0.14	3.74
903389	24090382	200912302354	33410	9127	0.14	3.70
756432	20171530	200912312354	26586	7795	0.13	3.51
741078	19762079	201001012354	28990	8310	0.15	3.91
802441	21398426	201001022354	31661	8891	0.15	3.95
837543	22334477	201001032354	33591	9333	0.15	4.01

</pre>

=== Release 3.5.6 ===

Assumes: 
* 1 in 10 keep default opt-in
* 1.5 in 10 override default opt-out and participate in sending
* 7.5 keep default opt out


<pre>
throt	adu's	date	#crashes	#sigs	crash/100users	adj'ed
0	1	200911032353	1	1	100.00	2666.67
4	108	200911042352	3	3	2.78	74.07
26	686	200911052354	8	5	1.17	31.10
39	1050	200911062349	21	16	2.00	53.33
127	3392	200911072349	59	37	1.74	46.38
196	5219	200911082349	40	26	0.77	20.44
257	6845	200911092349	22	14	0.32	8.57
329	8777	200911102349	34	24	0.39	10.33
367	9797	200911112354	21	17	0.21	5.72
392	10458	200911122350	26	20	0.25	6.63
423	11270	200911132354	51	33	0.45	12.07
413	11012	200911142354	29	19	0.26	7.02
348	9274	200911152353	21	17	0.23	6.04
450	12001	200911162354	47	33	0.39	10.44
466	12424	200911172354	32	26	0.26	6.87
479	12780	200911182353	31	27	0.24	6.47
479	12772	200911192354	27	21	0.21	5.64
464	12367	200911202354	23	18	0.19	4.96
387	10315	200911212354	34	18	0.33	8.79
400	10671	200911222354	36	19	0.34	9.00
497	13258	200911232354	38	32	0.29	7.64
501	13371	200911242354	32	21	0.24	6.38
489	13033	200911252354	25	20	0.19	5.12
416	11094	200911262354	20	14	0.18	4.81
518	13800	200911272354	24	22	0.17	4.64
516	13765	200911282354	28	28	0.20	5.42
2003915	11094	200911292354	29	29	0.26	0.00
332	8866	200911302354	34	30	0.38	10.23
286	7626	200912012354	36	29	0.47	12.59
8	215	200912022354	2	2	0.93	24.81
23	621	200912032354	2	2	0.32	8.59
59	1580	200912042353	41	38	2.59	69.20
1435	38261	200912052354	424	311	1.11	29.55
7166	191092	200912062354	740	519	0.39	10.33
10051	268018	200912072354	845	568	0.32	8.41
12655	337479	200912082354	932	624	0.28	7.36
13648	363938	200912092353	967	665	0.27	7.09
10376	276706	200912102353	983	629	0.36	9.47
13583	362226	200912112354	953	612	0.26	7.02
12320	328527	200912122354	963	636	0.29	7.82
12630	336812	200912132354	1033	671	0.31	8.18
15283	407549	200912142354	1072	684	0.26	7.01
15382	410196	200912152354	3071	1603	0.75	19.96
70961	1892297	200912162354	38537	10288	2.04	54.31       --- release
1001744	26713178	200912172354	80443	16951	0.30	8.03 
1408636	37563622	200912182354	94168	18885	0.25	6.69
1396724	37245967	200912192354	97103	19127	0.26	6.95
1547848	41275945	200912202354	105852	19938	0.26	6.84
1881333	50168875	200912212354	113209	21145	0.23	6.02
1930619	51483178	200912222354	111865	21156	0.22	5.79
1902380	50730145	200912232354	108806	20795	0.21	5.72
1685839	44955701	200912242353	95784	18990	0.21	5.68
1578158	42084225	200912252354	93840	18579	0.22	5.95
1661335	44302279	200912262354	103336	19964	0.23	6.22
1738281	46354170	200912272354	110489	20633	0.24	6.36
2003915	53437740	200912282353	118482	22105	0.22	5.91
2027493	54066489	200912292354	117276	21892	0.22	5.78
2012969	53679184	200912302354	115481	21262	0.22	5.74
1743204	46485440	200912312354	93512	18576	0.20	5.36
1744223	46512624	201001012354	100126	19477	0.22	5.74
1856723	49512624	201001022354	109588	20749	0.22	5.90
1938716	51699096	201001032354	114935	21535	0.22	5.93


</pre>

== Older Releases ==

=== Release 3.5.5 ===

throttle assumes 10% client throttling, then 15% server processing or ((adus * .10) * .15) 

<pre>
throt	adus	date	     #crashes #sigs crash/100users throttle adj'ed
0		 200911012350	0	0		
0		 200911022349	1	1		
2	138	 200911032353	0	0	0.00	0.00
4	295	 200911042352	3	3	1.02	67.80
129	8588	 200911052354	3548	1580	41.31	2754.23  -releasee
33912	2260795	 200911062349	41899	9698	1.85	123.55
321576	21438378 200911072349	81941	15360	0.38	25.48
455219	30347934 200911082349	95888	17386	0.32	21.06
589113	39274168 200911092349	100690	18062	0.26	17.09
657614	43840924 200911102349	104768	18557	0.24	15.93
684712	45647479 200911112354	110074	19201	0.24	16.08
714202	47613465 200911122350	100894	18127	0.21	14.13
704354	46956936 200911132354	107604	18872	0.23	15.28
619366	41291046 200911142354	104979	18519	0.25	16.95
654919	43661269 200911152353	111029	19122	0.25	16.95
773046	51536376 200911162354	115214	19656	0.22	14.90
786625	52441671 200911172354	124840	20048	0.24	15.87
793275	52885030 200911182353	119438	19943	0.23	15.06
793123	52874870 200911192354	115156	19858	0.22	14.52
767605	51173636 200911202354	112276	19719	0.22	14.63
672077	44805118 200911212354	109937	19335	0.25	16.36
699671	46644737 200911222354	117899	20390	0.25	16.85
826446	55096414 200911232354	120234	20916	0.22	14.55
					
</pre>

=== Release 3.5.4 ===

throttle assumes 10% client throttling, then 15% server processing or ((adus * .10) * .15)

<pre>

throt	adu's	date	        #crashes   #sigs crash/100users adj'ed
5975	398311	 20091013-crashdata.csv	1457	793	0.37	24.39
6575	438328	 20091014-crashdata.csv	1718	900	0.39	26.13
6815	454324	 20091015-crashdata.csv	1597	780	0.35	23.43
6795	452985	 20091016-crashdata.csv	1635	809	0.36	24.06
5932	395453	 20091017-crashdata.csv	1653	843	0.42	27.87
6133	408839	 20091018-crashdata.csv	1706	867	0.42	27.82
7360	490677	 20091019-crashdata.csv	1819	896	0.37	24.71
7492	499473	 20091020-crashdata.csv	1747	910	0.35	23.32
7533	502184	 20091021-crashdata.csv	1695	905	0.34	22.5
7512	500774	 20091022-crashdata.csv	1740	876	0.35	23.16
7275	484989	 20091023-crashdata.csv	1795	912	0.37	24.67
6334	422238	 20091024-crashdata.csv	1738	906	0.41	27.44
6515	434366	 20091025-crashdata.csv	1852	908	0.43	28.42
7680	511986	 20091026-crashdata.csv	1730	895	0.34	22.53
7730	515329	 20091027-crashdata.csv	2825	1310	0.55	36.55
27357	1823786	 20091028-crashdata.csv	34007	8190	1.86	124.31  -release
344243	22949509 20091029-crashdata.csv	79135	15053	0.34	22.99
488519	32567919 20091030-crashdata.csv	92431	16607	0.28	18.92
534332	35622134	200911012350	105485	18390	0.3	19.74
655674	43711622	200911022349	108838	18666	0.25	16.6
692718	46181179	200911032353	110940	18843	0.24	16.02
702750	46850020	200911042352	107798	18628	0.23	15.34
706014	47067594	200911052354	104667	18213	0.22	14.83 -3.5.5repl
668776	44585086	200911062349	69825	13500	0.16	10.44
286885	19125665	200911072349	31838	7655	0.17	11.1
186887	12459142	200911082349	21151	5723	0.17	11.32
171256	11417053	200911092349	16872	4860	0.15	9.85
114661	7644079	        200911102349	12372	3824	0.16	10.79
86634	5775579	        200911112354	10099	3290	0.17	11.66
67889	4525902	        200911122350	7938	2692	0.18	11.69
53840	3589364	        200911132354	7426	2474	0.21	13.79
41540	2769331	        200911142354	6667	2321	0.24	16.05
37504	2500244	        200911152353	6019	2181	0.24	16.05
40472	2698155	        200911162354	5937	2135	0.22	14.67
35387	2359153	        200911172354	5445	1918	0.23	15.39
31936	2129046	        200911182353	5098	1810	0.24	15.96
28871	1924766	        200911192354	4830	1745	0.25	16.73
25825	1721658	        200911202354	4637	1688	0.27	17.96
25825	1721658	        200911212354	4364	1605	0.25	16.9
21167	1411151	        200911222354	3993	1526	0.28	18.86
20086	1339089	        200911232354	4113	1557	0.31	20.48
23053	1536836	        200911242354	3856	1534	0.25	16.73
21460	1430689	        200911252354	3734	1436	0.26	17.4
19610	1307319	        200911262354	3392	1276	0.26	17.3
17672	1178106	        200911272354	3586	1292	0.3	20.29
16183	1078880	        200911282354	3185	1158	0.3	19.68
13890	926007	        200911292354	3123	1179	0.34	22.48
13751	916761	        200911302354	3269	1194	0.36	23.77

</pre>

=== An example of extreme growth in a regression crash causing a respin of 3.5.4 ===

https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20status1.9.1%3A.5-fixed

<pre>  
6  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091020-crashdata
5  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091021-crashdata
4  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091022-crashdata
9  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091023-crashdata
14  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091024-crashdata
10  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091025-crashdata
9  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091026-crashdata
33  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091027-crashdata

3.5.4 released --

506  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091028-crashdata
1034  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091029-crashdata
712  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091030-crashdata
556  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091031-crashdata
597  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091101-crashdata
540  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091102-crashdata
474  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091103-crashdata
404  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091104-crashdata
371  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091105-crashdata
249  crashes for gfxWindowsFontGroup::WhichFontSupportsChar on  20091106-crashdata

</pre>

On 20091024 just a few days before shipping that signature ranked #85 with 3
crashes that day.

85. 3 gfxWindowsFontGroup::WhichFontSupportsChar(nsTArray<nsRefPtr<FontEntry> >
const&, unsigned int) 3.5.4

20091029 ranking

2. 1029 gfxWindowsFontGroup::WhichFontSupportsChar(nsTArray<nsRefPtr<FontEntry> > const&, unsigned int) 3.5.4

=== Release 3.5.3 ===

<pre>


throt	adu's	date	#crashes	#sigs	crash/100users	adj'ed
22	1466	20090901-crashdata.csv	7	5	0.48	31.83
719	47955	20090902-crashdata.csv	52	46	0.11	7.23
4959	330590	20090903-crashdata.csv	744	492	0.23	15
6180	411973	20090904-crashdata.csv	1319	734	0.32	21.34
5769	384603	20090905-crashdata.csv	1584	856	0.41	27.46
6151	410099	20090906-crashdata.csv	1444	802	0.35	23.47
7407	493812	20090907-crashdata.csv	1534	789	0.31	20.71
7965	530986	20090908-crashdata.csv	1625	892	0.31	20.4
8476	565087	20090909-crashdata.csv	1703	937	0.3	20.09 -- release
44559	2970600	20090910-crashdata.csv	5219	2081	0.18	11.71
277241	18482710 20090911-crashdata.csv	34912	8799	0.19	12.59
291479	19431957 20090912-crashdata.csv	61423	13297	0.32	21.07
382936	25529036 20090913-crashdata.csv	72537	14862	0.28	18.94
505111	33674054 20090914-crashdata.csv	87536	17086	0.26	17.33
553302	36886822 20090915-crashdata.csv	95342	18220	0.26	17.23
575617	38374498 20090916-crashdata.csv	97072	18528	0.25	16.86
588830	39255365 20090917-crashdata.csv	98769	18508	0.25	16.77
574876	38325034 20090918-crashdata.csv	98938	18718	0.26	17.21
490791	32719389 20090919-crashdata.csv	97003	18655	0.3	19.76
525336	35022427 20090920-crashdata.csv	94472	18103	0.27	17.98
621540	41436022 20090921-crashdata.csv	102442	19071	0.25	16.48
636066	42404407 20090922-crashdata.csv	106806	19752	0.25	16.79
645549	43036631 20090923-crashdata.csv	106061	19523	0.25	16.43
648983	43265543 20090924-crashdata.csv	106873	19798	0.25	16.47
628283	41885531 20090925-crashdata.csv	105275	19760	0.25	16.76
541861	36124056 20090926-crashdata.csv	109329	19166	0.3	20.18
570765	38051029 20090927-crashdata.csv	110597	19062	0.29	19.38
676617	45107781 20090928-crashdata.csv	108851	19565	0.24	16.09
684282	45618770 20090929-crashdata.csv	110626	20073	0.24	16.17
						                18.27
</pre>

== Release 3.0.x ==

=== Release 3.0.14 ===

<pre>

throt	adu's	date	#crashes	#sigs	crash/100users	adj'ed
4	299	 20090901-crashdata.csv	1	1	0.33	22.30
133	8881	 20090902-crashdata.csv	10	9	0.11	7.51
1113	74179	 20090903-crashdata.csv	111	92	0.15	9.98
1435	95637	 20090904-crashdata.csv	154	123	0.16	10.74
1329	88567	 20090905-crashdata.csv	211	155	0.24	15.88
1441	96037	 20090906-crashdata.csv	198	150	0.21	13.74
1853	123525	 20090907-crashdata.csv	195	160	0.16	10.52
1967	131110	 20090908-crashdata.csv	197	150	0.15	10.02
2062	137469	 20090909-crashdata.csv	224	184	0.16	10.86 - release
24712	1647489	 20090910-crashdata.csv	1405	741	0.09	5.69
221928	14795188 20090911-crashdata.csv	15634	4669	0.11	7.04
227750	15183311 20090912-crashdata.csv	28298	7328	0.19	12.43
311923	20794866 20090913-crashdata.csv	32374	8636	0.16	10.38
432973	28864876 20090914-crashdata.csv	41727	10312	0.14	9.64
480558	32037233 20090915-crashdata.csv	46357	10951	0.14	9.65
500022	33334831 20090916-crashdata.csv	47023	11157	0.14	9.40
510106	34007050 20090917-crashdata.csv	47409	11296	0.14	9.29
490892	32726129 20090918-crashdata.csv	47116	11218	0.14	9.60
406300	27086687 20090919-crashdata.csv	45673	11156	0.17	11.24
439670	29311359 20090920-crashdata.csv	43943	10913	0.15	9.99
535907	35727115 20090921-crashdata.csv	48341	11631	0.14	9.02
544515	36301033 20090922-crashdata.csv	49944	11692	0.14	9.17
549171	36611388 20090923-crashdata.csv	49985	11796	0.14	9.10
548170	36544655 20090924-crashdata.csv	49626	11699	0.14	9.05
524175	34945024 20090925-crashdata.csv	48913	11522	0.14	9.33
438887	29259151 20090926-crashdata.csv	58846	11362	0.2	13.41
465577	31038440 20090927-crashdata.csv	61053	11084	0.2	13.11
569360	37957328 20090928-crashdata.csv	53189	11947	0.14	9.34
570608	38040564 20090929-crashdata.csv	52586	12059	0.14	9.22
  						                10.57
</pre>