Buildbot/Talos: Difference between revisions

Jump to navigation Jump to search

Buildbot/Talos (view source)

Revision as of 20:49, 6 December 2012

1,978 bytes removed ,  6 December 2012
→‎Adding a new test
Line 176: Line 176:


== Adding a new test ==
== Adding a new test ==
 
If you are developer of a component just talk to the a*team (jhammel or jmaher) and they will handle the majority of the [https://wiki.mozilla.org/Buildbot/Talos/Misc#Adding_a_new_test steps required].
Everybody wants moar tests, there is a lot that goes into adding a talos test:
 
* file a bug to add appropriate rows to [http://hg.mozilla.org/graphs/file/tip/sql/data.sql graphserver].
** file an additional IT bug to deploy the sql changes on the production and staging graph servers
** if this is adding new pages, ensure these sql changes include [http://hg.mozilla.org/graphs/file/5dd6f4fb9c27/sql/pages_table.sql page definitions]. NOTE: this one detail is usually forgotten since it is so rare
* file a bug to add tests to [http://hg.mozilla.org/build/talos/file/tip/talos/test.py talos].
* create a talos.zip file and file a releng bug to upload it to the build network
* create a patch for buildbot to add a definition of this new test and turn it on for the current branch
* create a m-c (not inbound) patch to modify testing/talos/talos.json, get it reviewed and landed
* create bugs for each time we uplift from mozilla-central->aurora->beta->release->esr to turn on your test
* if this is an update to an existing test that could change the numbers, this needs to be treated as a new test and run side by side for a week to get a new baseline for the numbers.
* file a bug to get tbpl updated with a new letter to track this test
 
While that is a laundry list of items to do, if you are developer of a component just talk to the a*team (jhammel or jmaher) and they will handle the majority of the steps above.  When adding a new test, we really need to understand what we are doing.  Here are some questions that you should know the answer to before adding a new test:
* What does this test measure?
* Does this test overlap with any existing test?
* What is the unit of measurement that we are recording?
* What would constitute a regression? 
* What is the expected range in the results over time?
* Are there variables or conditions which would affect this test?
** browser configuration (prefs, environment variables)?
** OS, resources, time of day, etc... ?
* Indepenedent of Observation?  Will this test produce the same number regardless of what was run before it?
* What considerations are there for how this test should be run and what tools are required?


== Bugs ==
== Bugs ==
Jmaher
Confirmed users
3,376

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/492507"

Navigation menu