|
|
| (22 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| == Overview == | | == Overview == |
|
| |
|
| Peptest is an automated testing framework designed to test whether or not the browser's UI thread remains responsive while performing a variety of actions. Tests are simple Javascript files which can optionally import Mozmill's driver to manipulate the user interface in an automated fashion. | | Peptest is an automated testing framework designed to test whether or not the browser's UI thread remains responsive while performing a variety of actions. Tests are simple JavaScript files which can optionally import Mozmill's driver to manipulate the user interface in an automated fashion. |
|
| |
|
| The code is currently located at https://github.com/mozilla/peptest. If you would like to help you can send pull requests, or ping ahal on irc for ways to get started.
| | See the [https://developer.mozilla.org/en/Mozilla_automated_testing/Peptest MDN page] for more information, including how to use peptest and write tests for it. |
|
| |
|
| == Using Peptest ==
| | Peptest statistics for try, mozilla-inbound, and mozilla-central are available at http://mrcote.info/peptest/ |
|
| |
|
| === Running Tests === | | == Further Work == |
| | |
| Currently tests are run from the command line with python. Peptest currently depends on some external Mozilla python packages, namely: mozrunner, mozprocess, mozprofile, mozinfo, mozlog and manifestdestiny. These packages all live in the mozbase repository.
| |
| | |
| #Clone the mozbase repository at [https://github.com/mozilla/mozbase github.com/mozilla/mozbase].
| |
| #Create a new [http://pypi.python.org/pypi/virtualenv virtualenv] and install the mozinfo, mozlog, mozprofile, mozprocess, mozrunner and manifestdestiny packages into it (run 'python setup.py install' for each or run 'python setup_development.py' to install all the packages).
| |
| #Clone the Peptest repo at [https://github.com/mozilla/peptest github.com/mozilla/peptest]<br>
| |
| #Run the command (use --help for a full list of commands)<br><pre>python runpeptests.py --binary <path_to_binary> --test-path <path_to_test_manifest></pre>
| |
| | |
| === Test Manifest ===
| |
| | |
| All parameters are optional except for the --test-path parameter. This parameter should contain the path to a manifest that lists all the tests that will be run. Manifests should be of the following format:
| |
| <pre># test paths are relative to the manifest
| |
| [test1.js]
| |
| [test2.js]
| |
| [foo/bar/test3.js]
| |
| # this test is disabled and won't be run
| |
| [test4.js]
| |
| disabled = "Doesn't work, see bug 123456"
| |
| </pre>
| |
| | |
| Alternatively, the --test-path parameter can point to a single test which might be useful for debugging purposes.
| |
| | |
| Peptest uses ManifestDestiny. See [https://github.com/mozilla/mozbase/tree/master/manifestdestiny additional documentation].
| |
| | |
| ==== Fail Threshold ====
| |
| | |
| One more thing that can be specified from the manifest is the failure threshold. When each test finishes, the Peptest harness will aggregate all of the unresponsive times into one number that can be used for regression detection (see [https://wiki.mozilla.org/Auto-tools/Projects/peptest#Metric metric]). If you specify a failThreshold for a test, it will only be reported as a failure if it's metric is higher than the threshold.
| |
| <pre>
| |
| # no failThreshold, this test will fail if there is any unresponsiveness whatsoever
| |
| [test1.js]
| |
| # these tests will only fail if their metrics are greater than their failThresholds
| |
| [test2.js]
| |
| failThreshold = 42
| |
| [test3.js]
| |
| failThreshold = 500
| |
| </pre>
| |
| | |
| === Test Format ===
| |
| | |
| Tests are simply Javascript files that will be executed in chrome space. This means they have access to any of the API's that an extension would normally have access to. In addition to this, they also have access to [https://developer.mozilla.org/en/Mozmill Mozmill]'s driver for convenient automation of Firefox/Thunderbird's UI.
| |
| | |
| It is helpful to think of tests as a series of actions. That is, the UI thread will only be checked for responsiveness while an action is currently happening. This ensures that we are only testing the actions that we care about, and that the test isn't overrun with noise generated during setup or teardown. To accomplish this, tests call a function called 'performAction(name, func)' which takes in two parameters, a name and a function pointer whose contents consist of a single action.
| |
| | |
| The following example test will make sure that the browser remains responsive while opening a page and while searching in Google.
| |
| <pre>// import mozmill and initialize a controller
| |
| Components.utils.import('resource://mozmill/driver/mozmill.js');
| |
| let controller = getBrowserController();
| |
| | |
| // Create our first action which will open Google
| |
| performAction('open_google', function() {
| |
| controller.open('http://google.com');
| |
| controller.waitForPageLoad();
| |
| });
| |
| | |
| // stuff not inside a performAction() call won't be tested for responsiveness
| |
| let textbox = findElement.ID(controller.tabs.activeTab, 'lst-ib');
| |
| let button = findElement.Name(controller.tabs.activeTab, 'btnK');
| |
| | |
| // Create our second action which will perform a search in the google searchbox
| |
| performAction('search_google', function() {
| |
| textbox.sendKeys('foobar');
| |
| button.click();
| |
| controller.waitForPageLoad();
| |
| });
| |
| </pre>
| |
| For documentation on using Mozmill's driver see: [https://developer.mozilla.org/en/Mozmill#Reference_Desk The Mozmill Reference Desk]
| |
| | |
| Note that using mozmill may be convenient but is not required. Here is another example that tests resizing the window.
| |
| <pre>
| |
| // getWindow is a convenience function exposed to tests
| |
| // It uses nsIWindowMediator to get the current window
| |
| let window = getWindow();
| |
| let width = window.outerWidth;
| |
| let height = window.outerHeight;
| |
| | |
| performAction('resize_by', function() {
| |
| window.resizeBy(100, 100);
| |
| });
| |
| | |
| performAction('resize_to', function() {
| |
| window.resizeTo(800, 600);
| |
| });
| |
| | |
| // Tests should clean up after themselves
| |
| window.resizeTo(width, height);
| |
| </pre>
| |
| | |
| There are a few [https://github.com/ahal/peptest/tree/master/tests/firefox/examples example Peptests] in the repo.
| |
| | |
| === Log Format ===
| |
| | |
| Peptest uses the [https://github.com/mozilla/mozbase/tree/master/mozlog mozlog] module to standardize on the logging format.
| |
| <pre>
| |
| | |
| PEP TEST-START | test1.js
| |
| PEP TEST-PASS | test1.js | fail threshold: 0.0 | metric: 0.0
| |
| PEP TEST-END | test1.js | finished in: 1536 ms
| |
| PEP TEST-START | test2.js
| |
| PEP WARNING | test2.js | loadPage | unresponsive time: 116 ms
| |
| PEP WARNING | test2.js | loadPage | unresponsive time: 61 ms
| |
| PEP WARNING | test2.js | loadPage | unresponsive time: 275 ms
| |
| PEP TEST-UNEXPECTED-FAIL | test2.js | fail threshold: 90.0 | metric: 92.802
| |
| PEP TEST-END | test2.js | finished in: 8022 ms
| |
| PEP TEST-START | test3.js
| |
| PEP WARNING | test3.js | scroll_menu | unresponsive time: 121 ms
| |
| PEP TEST-PASS | test3.js | fail threshold: 15.0 | metric: 14.641
| |
| PEP TEST-END | test3.js | finished in: 2137 ms
| |
| </pre>
| |
| | |
| When a responsiveness measurement runs over 50ms a TEST-UNEXPECTED-FAIL message is output. The second column is the name of the test file, the third column is the name of the particular action that was being performed and the last column is the time in milliseconds that the event took to process.
| |
|
| |
|
| When the test is finished, the total run time and a metric are output. The metric is explained in the next section.
| | === Automatic reports === |
|
| |
|
| ==== Metric ====
| | {{bug|737886}} Set up reporting when significant deviations occur in buildbot's peptest results. Since we can't develop a simple pass/fail result, developers will have to rely on these reports to determine when responsiveness has gotten worse (or better). |
|
| |
|
| The metric is a number that is calculated based on all accumulated unresponsive times. The function used to calculate the metric is:
| | === Telemetry histograms === |
| <pre>
| |
| # values is a list of unresponsive values that occurred during the test
| |
| def get_metric(values):
| |
| return sum([x*x / 1000.0 for x in values])
| |
| </pre>
| |
|
| |
|
| The metric is useful for determining whether a test has regressed or not. For example if a test used to have a metric of 56.43 then all of a sudden has a metric of 193.42 we know a regression has happened.
| | {{bug|740462}} Capturing telemetry information would be useful in diagnosing causes of responsiveness. Results would be available after running a test, and by clicking through on the stats UI. |
|
| |
|
| '''Important:''' You cannot compare the metric from one test to that of another. Tests can be of an arbitrary length and therefore it is impossible to conclude that a test with a metric of 10 is more responsive than a test with a metric of 1000. You must only compare metric values to previous metric values of the same test.
| | === Integration with SPS === |
|
| |
|
| == Further Work ==
| | {{bug|706847}} For each function wrapped in a performAction() call, we should enable the profiler, so we can tell where each action spent its time. We can use [https://developer.mozilla.org/en/Performance/Profiling_with_the_Built-in_Profiler SPS] to this end. |
|
| |
|
| === Mozmill e10s === | | === Mozmill e10s === |
|
| |
|
| ''' Note: this will likely be put on hold indefinitely until the status of e10s becomes more clear ''' | | '''Note: this will likely be put on hold indefinitely until the status of e10s becomes more clear ''' |
|
| |
|
| Mozmill doesn't use MessageManager when it interacts with content. This means that it will not work correctly with Electrolysis builds (which is mostly the entire point of Peptests). Mozmill will need to be refactored to use MessageManager and also to stop using the gBrowser object which doesn't exist in mobile Firefox. | | Mozmill doesn't use MessageManager when it interacts with content. This means that it will not work correctly with Electrolysis builds (which is mostly the entire point of Peptests). Mozmill will need to be refactored to use MessageManager and also to stop using the gBrowser object which doesn't exist in mobile Firefox. |
|
| |
| === Detailed results ===
| |
|
| |
| Dietrich has written some tools to determine exactly which Javascript calls were unresponsive. This is a very useful tool which should be incorporated into the Peptest harness. More details can be found [http://etherpad.mozilla.com:9000/responsiveness-profiling here]
| |
|
| |
| I (ahal) would really like someone to own [https://bugzilla.mozilla.org/show_bug.cgi?id=580055 bug 580055], un-bitrot the patch and get it checked in (with ifdefs or something). If you want, I'll even un-bitrot the patch for you :)
| |
|
| |
| === Buildbot Integration ===
| |
|
| |
| Work has been done on writing a mozharness peptest script in [https://bugzilla.mozilla.org/show_bug.cgi?id=692091 bug 692091]. There is a mozharness script checked in and work is being done to get it running on tryserver.
| |
|
| |
|
| == Related Bugs == | | == Related Bugs == |
|
| |
|
| *{{bug|580055}} - responsiveness profiling (blocked on {{bug|702740}})
| | New bugs should be [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Peptest filed] under Testing / Peptest. Here are the [https://bugzilla.mozilla.org/buglist.cgi?list_id=2648204;resolution=---;query_format=advanced;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;component=Peptest;product=Testing open bugs]. |
| *{{bug|674606}} - initial harness creation
| |
| *<strike>{{bug|692091}}</strike> - mozharness script
| |
| *{{bug|700415}} - peptest on tryserver
| |
| *<strike>{{bug|700656}}</strike> - peptest in m-c/packaged tests
| |
| *{{bug|703266}} - mirror mozbase to m-c
| |
| *{{bug|703269}} - mirror peptest harness to m-c
| |
| | |
| == API Reference == | |
| | |
| There is a Peptest API that gets injected into the scope of every test. Each test can use the following methods:
| |
| | |
| === General ===
| |
| ==== performAction(name, function) ====
| |
| Performs an action during which responsiveness measurements are recorded
| |
| * name - the name of the action to run
| |
| * function - a function pointer of the action to run
| |
| | |
| ==== getWindow(type) ====
| |
| Returns a reference to the current window
| |
| * type - string representing the window type, defaults to "navigator:browser" (see [https://developer.mozilla.org/en/XUL/Attribute/windowtype windowType])
| |
| | |
| ==== sleep(milliseconds) ==== | |
| Unfortunately UI Automation is very dependent on timing and sometimes it is necessary to sleep, though you should avoid this method if you can
| |
| * milliseconds - the number of milliseconds to sleep for
| |
| | |
| === Logging ===
| |
| For logging messages to terminal
| |
| * msg - the message to log
| |
| | |
| Logging methods
| |
| ==== log.debug(msg) ====
| |
| ==== log.info(msg) ====
| |
| ==== log.warning(msg) ====
| |
| ==== log.error(msg) ====
| |