Revision as of 20:58, 11 May 2010

Current Manual Testing

Top 500 addons

Linux:

sorted average Talos Ts

no plans to allow addons to 'call home' - we will still be working in the talos testing environment where we are proxied to localhost, so there will be no live web interaction
no current plans to interact with the addon (no clicks, no visiting specific pages)
- this sort of perf test would have to be designed/built per-addon to get the most bang for the buck

integrate into buildbot and have run on production machines
- how frequently?
- which tests?
  - for now, we'll limit to clean ts starts with just the addon installed
- where would the list of addons be maintained?
- where do we download the addons from?
results reported as .csv files
- where should these be sent?
- do we want the data on the graph server?
- how do we compare the results?

@@ Line 1: / Line 1: @@
-=code=
+=Current Manual Testing=
+==code==
 http://github.com/harthur/dirtyharry
-=results=
+==results==
 Top 500 addons
@@ Line 10: / Line 11: @@
 [http://github.com/harthur/dirtyharry/blob/master/results/raw_results_linux0.csv Talos Ts]
+=Automated Testing=
+== Limitations ==
+* no plans to allow addons to 'call home' - we will still be working in the talos testing environment where we are proxied to localhost, so there will be no live web interaction
+* no current plans to interact with the addon (no clicks, no visiting specific pages)
+** this sort of perf test would have to be designed/built per-addon to get the most bang for the buck
+== Plans ==
+* integrate into buildbot and have run on production machines
+** how frequently?
+** which tests?
+*** for now, we'll limit to clean ts starts with just the addon installed
+** where would the list of addons be maintained?
+** where do we download the addons from?
+* results reported as .csv files
+** where should these be sent?
+** do we want the data on the graph server?
+** how do we compare the results?