Canmove, confirm, emeritus
3,627
edits
Changes
→MozMill and Leak & bloat tests
# <strike>{{Bug|458352}} support for different profiles when running tests</strike>
# <strike>{{Bug|500142}} Integrate mozmill into the build system - tests can be run with "make mozmill" and "make mozmill-one"</strike>
# (in progress) {{Bug|504304}} Test VM/buildbot slave for initial results and automation. Aim is to a) check the stability, b) use as a platform for implementing and testing the necessary mozmill test support infrastructure, c) get initial results alongside builds on day-to-day basis.[http://tinderbox.mozilla.org/ThunderbirdTest/ ThunderbirdTest tinderbox]
# Integrate httpd server into mozmill run scripts - mozmill actually already has a httpd server integrated. See example on {{bug|492279}}
# {{Bug|500753}} Integrate fake servers into mozmill scripts.
* A pass count of zero should return failure.
* It would be neat if we could return the test results via a JSON blob somewhere rather than the current tradition of text-mangling. I am mainly thinking in terms of having a web UI that is better able to tell us where things went wrong. But that's just pipe-dreams and tinderpushlog with its magic already gets us more than 80% of the way towards usability...
=== Intermittent MozMill Failures ===
I'm using this here just to get an initial idea of the sort of failures we're getting. I'll update it once or twice a day with the status from the period before.
At some stage this will move into bugzilla, but let's get an idea of what is happening first.
Note: Only tracking 1.9.1 initially due to regressions on trunk.
==== 1.9.1 ====
From: 2009/07/28 09:41:01
To: 2009/07/28 19:12:02
Total Runs: 6
Total Green Runs: 4
===== Failure Mode 1 =====
Seen: once
Result: 137/8
Short output:
TEST-UNEXPECTED-FAIL | test_middle_click_with_many_things_selected_foreground
TEST-UNEXPECTED-FAIL | test_middle_click_on_existing_single_selection_background
TEST-UNEXPECTED-FAIL | test_middle_click_on_existing_single_selection_foreground
TEST-UNEXPECTED-FAIL | test_middle_click_on_existing_multi_selection_foreground
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | test_open_multiple_search_results_in_new_tabs
TEST-UNEXPECTED-FAIL | test_verify_saved_search
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
make: *** [mozmill] Error 1
Unknown Error: command finished with exit code: 2
Log: http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTest/1248799261.1248800444.11281.gz
===== Failure Mode 2 =====
Seen: once
Result: 142/2
Short output:
TEST-UNEXPECTED-FAIL | test_summarization_goes_away
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
make: *** [mozmill] Error 1
Unknown Error: command finished with exit code: 2
Log: http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTest/1248832277.1248833470.13976.gz
