Confirmed users
3,376
edits
Buildbot/Talos/Running: Difference between revisions
Jump to navigation
Jump to search
m
some basic cleanup
(→Testing Locally: - oh, more cleanup now that I looked at it in mor edetail) |
m (some basic cleanup) |
||
| Line 70: | Line 70: | ||
The best way to answer this question is to push to try server and compare the reported numbers from the logs (use Treeherder as a log parser) and compare that with the [[http://graphs.mozilla.org/ graph server]]. I recommend using Treeherder to open the link to the graphs. | The best way to answer this question is to push to try server and compare the reported numbers from the logs (use Treeherder as a log parser) and compare that with the [[http://graphs.mozilla.org/ graph server]]. I recommend using Treeherder to open the link to the graphs. | ||
If you are planning on landing on mozilla-central, look at tests from mozilla-central. Be aware of PGO vs Non PGO and | If you are planning on landing on mozilla-central, look at tests from mozilla-central. Be aware of PGO vs Non PGO and e10s vs Non e10s. Treeherder makes this a moderately pain free process (i.e. about 30 minutes). | ||
== Using try server == | == Using try server == | ||
| Line 78: | Line 78: | ||
=== Testing changes to talos properly === | === Testing changes to talos properly === | ||
* Create a [https://developer.mozilla.org/en-US/docs/Creating_Mercurial_User_Repositories?redirectlocale=en-US&redirectslug=Publishing_Mercurial_Clones user repo] with talos, commit and push your changes | * Create a [https://developer.mozilla.org/en-US/docs/Creating_Mercurial_User_Repositories?redirectlocale=en-US&redirectslug=Publishing_Mercurial_Clones user repo] with talos, commit and push your changes | ||
* Check out a copy of Mozilla central | * Check out a copy of Mozilla central | ||
* Modify the file "testing/talos/talos.json" to point to your user repo and put the changeset as 'default'. | * Modify the file "testing/talos/talos.json" to point to your user repo and put the changeset as 'default'. | ||
| Line 100: | Line 99: | ||
* pip install virtualenv # http://www.virtualenv.org/en/latest/virtualenv.html#installation | * pip install virtualenv # http://www.virtualenv.org/en/latest/virtualenv.html#installation | ||
For the majority of the tests, we include test files and tools out of the box. | For the majority of the tests, we include test files and tools out of the box. You will need the [https://developer.mozilla.org/en-US/docs/Simple_Firefox_build mozilla-central code base]: | ||
* '''clone talos:''' | * '''clone talos:''' | ||
hg clone http://hg.mozilla.org/ | hg clone http://hg.mozilla.org/mozilla-central | ||
* ''' | * '''setup talos''' | ||
cd talos | cd mozilla-central/testing/talos | ||
virtualenv . | |||
. bin/activate (or on windows Scripts\activate.bat) | |||
python setup.py develop | |||
* '''run tests:''' | * '''run tests:''' | ||
talos | talos --develop --executablePath pathtofirefox --activeTests tart | ||
# Note: this will open and close the browser real fast for the warmup, then run the test | # Note: this will open and close the browser real fast for the warmup, then run the test | ||
** --develop indicates to run in develop mode and | ** --develop indicates to run in develop mode and store the results in local.json + local.out | ||
** --executablePath tells Talos where the firefox installation we want to run is located | ** --executablePath tells Talos where the firefox installation we want to run is located | ||
*** we have pathtofirefox as an example, you can use '~/mozilla/objdir/dist/bin/firefox' or whatever the full path is to your firefox executable that will be running the tests. | *** we have pathtofirefox as an example, you can use '~/mozilla/objdir/dist/bin/firefox' or whatever the full path is to your firefox executable that will be running the tests. | ||
| Line 145: | Line 131: | ||
== How Talos is Run in Production == | == How Talos is Run in Production == | ||
* | * buildbot constructs commands to launch mozharness, ex: | ||
/tools/buildbot/bin/python scripts/scripts/talos_script.py --suite chromez --add-option --webServer,localhost --branch-name Mozilla-Inbound-Non-PGO --system-bits 64 --cfg talos/linux_config.py --download-symbols ondemand --use-talos-json --blob-upload-branch Mozilla-Inbound-Non-PGO | |||
* | * mozharness [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/talos.py talos.py] downloads common tools, the talos source, pagesets and run the install commands. | ||
* | * talos [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/run_tests.py run_tests.py] is invoked with commandline options which then runs the test as you would locally | ||
* Talos will run the tests and measured results which are uploaded to [http://graphs.mozilla.org/ graphserver] (after being suitably averaged per-page for Pageloader tests) | * Talos will run the tests and measured results which are uploaded to [http://graphs.mozilla.org/ graphserver] (after being suitably averaged per-page for Pageloader tests) | ||
** (Talos also uploads raw results to [https:// | ** (Talos also uploads raw results to [https://treeherder.mozilla.org/perf.html#/graphs PerfHerder]) | ||
* the graphserver performs averaging across the pageset (for Pageloader tests) or across cycles (for Startup tests) and returns a number via HTTP to Talos which is then printed to the log | * the graphserver performs averaging across the pageset (for Pageloader tests) or across cycles (for Startup tests) and returns a number via HTTP to Talos which is then printed to the log | ||
| Line 163: | Line 150: | ||
=== Running Tp the Automation Way === | === Running Tp the Automation Way === | ||
In our automation, we run beneath many more restrictions than normal users. One of those restrictions is that our automation machines are walled off from the real-world networks. Because of this, and because we want to test pure page-loading and rendering time of Firefox, we serve the pages from localhost using | In our automation, we run beneath many more restrictions than normal users. One of those restrictions is that our automation machines are walled off from the real-world networks. Because of this, and because we want to test pure page-loading and rendering time of Firefox, we serve the pages from localhost using a built in python webserver thus eliminating all network latency and uncertainty. You've probably noticed this if you looked at the talos/page_load_test/tp4.manifest. | ||
To do this, we construct full downloads of sites in our manifest and they are placed on the automation slave at run time. Because we cannot at this time distribute our full page load test set, I'll walk through how these are set up and show you how to make your own. Note that our next version of the page load set will be distributable, so soon this won't be an issue. | To do this, we construct full downloads of sites in our manifest and they are placed on the automation slave at run time. Because we cannot at this time distribute our full page load test set, I'll walk through how these are set up and show you how to make your own. Note that our next version of the page load set will be distributable, so soon this won't be an issue. | ||
| Line 169: | Line 156: | ||
In the meantime, here's the instructions: | In the meantime, here's the instructions: | ||
<ol> | <ol> | ||
<li>Use | <li>Use the following wget command to fetch a page and everything it links to in order to have a complete page for offline use: <pre>wget -p -k -H -E -erobots=off --no-check-certificate -U "Mozilla/5.0 (firefox)" --restrict-file-names=windows --restrict-file-names=nocontrol $URL -o outputlog.txt</pre> | ||
</li> | </li> | ||
</ol> | </ol> | ||