|
|
| (42 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| == Overview == | | == Overview == |
|
| |
|
| Peptest is an automated testing framework designed to test whether or not the browser's UI thread remains responsive while performing a variety of actions. Tests are simple Javascript files which can optionally import Mozmill's driver to manipulate the user interface in an automated fashion. | | Peptest is an automated testing framework designed to test whether or not the browser's UI thread remains responsive while performing a variety of actions. Tests are simple JavaScript files which can optionally import Mozmill's driver to manipulate the user interface in an automated fashion. |
|
| |
|
| The code is currently located at https://github.com/mozilla/peptest. If you would like to help you can send pull requests, or ping ahal on irc for ways to get started.
| | See the [https://developer.mozilla.org/en/Mozilla_automated_testing/Peptest MDN page] for more information, including how to use peptest and write tests for it. |
|
| |
|
| == Using Peptest ==
| | Peptest statistics for try, mozilla-inbound, and mozilla-central are available at http://mrcote.info/peptest/ |
|
| |
|
| === Running Tests === | | == Further Work == |
|
| |
|
| Currently tests are run from the command line with python. Peptest currently depends on some external Mozilla python packages, namely: mozrunner, mozprocess, mozprofile, mozinfo, mozlog and manifestdestiny. These packages all live in the mozbase repository.
| | === Automatic reports === |
|
| |
|
| #Clone the mozbase repository at [https://github.com/mozilla/mozbase github.com/mozilla/mozbase].
| | {{bug|737886}} Set up reporting when significant deviations occur in buildbot's peptest results. Since we can't develop a simple pass/fail result, developers will have to rely on these reports to determine when responsiveness has gotten worse (or better). |
| #Create a new [http://pypi.python.org/pypi/virtualenv virtualenv] and install the mozinfo, mozlog, mozprofile, mozprocess, mozrunner and manifestdestiny packages into it (run 'python setup.py install' for each or run 'python setup_development.py' to install all the packages).
| |
| #Clone the Peptest repo at [https://github.com/mozilla/peptest github.com/mozilla/peptest]<br>
| |
| #Run the command (use --help for a full list of commands)<br><pre>python runpeptests.py --binary <path_to_binary> --test-path <path_to_test_manifest></pre>
| |
|
| |
|
| === Test Manifest === | | === Telemetry histograms === |
|
| |
|
| All parameters are optional except for the --test-path parameter. This parameter should contain the path to a manifest that lists all the tests that will be run. Manifests should be of the following format:
| | {{bug|740462}} Capturing telemetry information would be useful in diagnosing causes of responsiveness. Results would be available after running a test, and by clicking through on the stats UI. |
| <pre># test paths are relative to the manifest
| |
| [test1.js]
| |
| [test2.js]
| |
| [foo/bar/test3.js]
| |
| # this test is disabled and won't be run
| |
| [test4.js]
| |
| disabled = "Doesn't work, see bug 123456"
| |
| </pre>
| |
|
| |
|
| Alternatively, the --test-path parameter can point to a single test which might be useful for debugging purposes.
| | === Integration with SPS === |
|
| |
|
| === Test Format ===
| | {{bug|706847}} For each function wrapped in a performAction() call, we should enable the profiler, so we can tell where each action spent its time. We can use [https://developer.mozilla.org/en/Performance/Profiling_with_the_Built-in_Profiler SPS] to this end. |
| | |
| Tests are simply Javascript files that will be executed in chrome space. This means they have access to any of the API's that an extension would normally have access to. In addition to this, they also have access to [https://developer.mozilla.org/en/Mozmill Mozmill]'s driver for convenient automation of Firefox/Thunderbird's UI.
| |
| | |
| It is helpful to think of tests as a series of actions. That is, the UI thread will only be checked for responsiveness while an action is currently happening. This ensures that we are only testing the actions that we care about, and that the test isn't overrun with noise generated during setup or teardown. To accomplish this, tests call a function called 'performAction(name, func)' which takes in two parameters, a name and a function pointer whose contents consist of a single action.
| |
| | |
| The following example test will make sure that the browser remains responsive while opening a page and while searching in Google.
| |
| <pre>// import mozmill and initialize a controller
| |
| Components.utils.import('resource://mozmill/driver/mozmill.js');
| |
| let controller = getBrowserController();
| |
| | |
| // Create our first action which will open Google
| |
| performAction('open_google', function() {
| |
| controller.open('http://google.com');
| |
| controller.waitForPageLoad();
| |
| });
| |
| | |
| // stuff not inside a performAction() call won't be tested for responsiveness
| |
| let textbox = findElement.ID(controller.tabs.activeTab, 'lst-ib');
| |
| let button = findElement.Name(controller.tabs.activeTab, 'btnK');
| |
| | |
| // Create our second action which will perform a search in the google searchbox
| |
| performAction('search_google', function() {
| |
| textbox.sendKeys('foobar');
| |
| button.click();
| |
| controller.waitForPageLoad();
| |
| });
| |
| </pre>
| |
| For documentation on using Mozmill's driver see: [https://developer.mozilla.org/en/Mozmill#Reference_Desk The Mozmill Reference Desk]
| |
| | |
| There are a few [https://github.com/ahal/peptest/tree/master/tests/firefox example Peptests] in the repo.
| |
| | |
| For more complicated examples, see QA Automation's [http://hg.mozilla.org/qa/mozmill-tests/ mozmill-tests] repo (NB these aren't peptests, though the syntax minus the setupModule and testFoo methods is the same).
| |
| | |
| === Log Format ===
| |
| | |
| Peptest uses the new mozlog module to standardize on the logging format.
| |
| <pre>
| |
| PEP TEST-START | test_sanity.js
| |
| PEP TEST-UNEXPECTED-FAIL | test_sanity.js | loadPage | unresponsive time: 116 ms
| |
| PEP TEST-UNEXPECTED-FAIL | test_sanity.js | loadPage | unresponsive time: 61 ms
| |
| PEP TEST-UNEXPECTED-FAIL | test_sanity.js | loadPage | unresponsive time: 275 ms
| |
| PEP TEST-END | test_sanity.js | finished in: 8022 ms
| |
| PEP TEST-START | test_sanity2.js
| |
| PEP TEST-PASS | test_sanity2.js
| |
| PEP TEST-END | test_sanity2.js | finished in: 1536 ms
| |
| </pre>
| |
| | |
| When a responsiveness measurement runs over 50ms a TEST-UNEXPECTED-FAIL message is output. The second column is the name of the test file, the third column is the name of the particular action that was being performed and the last column is the time in milliseconds that the event took to process.
| |
| | |
| == Further Work ==
| |
|
| |
|
| === Mozmill e10s === | | === Mozmill e10s === |
|
| |
|
| ''' Note: this will likely be put on hold indefinitely until the status of e10s becomes more clear ''' | | '''Note: this will likely be put on hold indefinitely until the status of e10s becomes more clear ''' |
|
| |
|
| Mozmill doesn't use MessageManager when it interacts with content. This means that it will not work correctly with Electrolysis builds (which is mostly the entire point of Peptests). Mozmill will need to be refactored to use MessageManager and also to stop using the gBrowser object which doesn't exist in mobile Firefox. | | Mozmill doesn't use MessageManager when it interacts with content. This means that it will not work correctly with Electrolysis builds (which is mostly the entire point of Peptests). Mozmill will need to be refactored to use MessageManager and also to stop using the gBrowser object which doesn't exist in mobile Firefox. |
|
| |
|
| === Detailed results === | | == Related Bugs == |
| | |
| Dietrich has written some tools to determine exactly which Javascript calls were unresponsive. This is a very useful tool which should be incorporated into the Peptest harness. More details can be found [http://etherpad.mozilla.com:9000/responsiveness-profiling here]
| |
| | |
| I (ahal) would really like someone to own [https://bugzilla.mozilla.org/show_bug.cgi?id=580055 bug 580055], un-bitrot the patch and get it checked in (with ifdefs or something). If you want, I'll even un-bitrot the patch for you :)
| |
| | |
| === Buildbot Integration ===
| |
|
| |
|
| Work has been done on writing a mozharness peptest script in [https://bugzilla.mozilla.org/show_bug.cgi?id=692091 bug 692091]. There is a mozharness script checked in and work is being done to get it running on tryserver.
| | New bugs should be [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Peptest filed] under Testing / Peptest. Here are the [https://bugzilla.mozilla.org/buglist.cgi?list_id=2648204;resolution=---;query_format=advanced;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;component=Peptest;product=Testing open bugs]. |
Overview
Peptest is an automated testing framework designed to test whether or not the browser's UI thread remains responsive while performing a variety of actions. Tests are simple JavaScript files which can optionally import Mozmill's driver to manipulate the user interface in an automated fashion.
See the MDN page for more information, including how to use peptest and write tests for it.
Peptest statistics for try, mozilla-inbound, and mozilla-central are available at http://mrcote.info/peptest/
Further Work
Automatic reports
bug 737886 Set up reporting when significant deviations occur in buildbot's peptest results. Since we can't develop a simple pass/fail result, developers will have to rely on these reports to determine when responsiveness has gotten worse (or better).
Telemetry histograms
bug 740462 Capturing telemetry information would be useful in diagnosing causes of responsiveness. Results would be available after running a test, and by clicking through on the stats UI.
Integration with SPS
bug 706847 For each function wrapped in a performAction() call, we should enable the profiler, so we can tell where each action spent its time. We can use SPS to this end.
Mozmill e10s
Note: this will likely be put on hold indefinitely until the status of e10s becomes more clear
Mozmill doesn't use MessageManager when it interacts with content. This means that it will not work correctly with Electrolysis builds (which is mostly the entire point of Peptests). Mozmill will need to be refactored to use MessageManager and also to stop using the gBrowser object which doesn't exist in mobile Firefox.
Related Bugs
New bugs should be filed under Testing / Peptest. Here are the open bugs.