canmove, Confirmed users, Bureaucrats and Sysops emeriti
3,628
edits
User:Standard8/Thunderbird Automated Tests Planning: Difference between revisions
Jump to navigation
Jump to search
User:Standard8/Thunderbird Automated Tests Planning (view source)
Revision as of 11:37, 29 July 2009
, 29 July 2009→MozMill and Leak & bloat tests
(→MozMill and Leak & bloat tests: Update http status and bug number for fakeserver) |
|||
| Line 27: | Line 27: | ||
# <strike>{{Bug|458352}} support for different profiles when running tests</strike> | # <strike>{{Bug|458352}} support for different profiles when running tests</strike> | ||
# <strike>{{Bug|500142}} Integrate mozmill into the build system - tests can be run with "make mozmill" and "make mozmill-one"</strike> | # <strike>{{Bug|500142}} Integrate mozmill into the build system - tests can be run with "make mozmill" and "make mozmill-one"</strike> | ||
# (in progress) {{Bug|504304}} Test VM/buildbot slave for initial results and automation. Aim is to a) check the stability, b) use as a platform for implementing and testing the necessary mozmill test support infrastructure, c) get initial results alongside builds on day-to-day basis. | # (in progress) {{Bug|504304}} Test VM/buildbot slave for initial results and automation. Aim is to a) check the stability, b) use as a platform for implementing and testing the necessary mozmill test support infrastructure, c) get initial results alongside builds on day-to-day basis. [http://tinderbox.mozilla.org/ThunderbirdTest/ ThunderbirdTest tinderbox] | ||
# Integrate httpd server into mozmill run scripts - mozmill actually already has a httpd server integrated. See example on {{bug|492279}} | # Integrate httpd server into mozmill run scripts - mozmill actually already has a httpd server integrated. See example on {{bug|492279}} | ||
# {{Bug|500753}} Integrate fake servers into mozmill scripts. | # {{Bug|500753}} Integrate fake servers into mozmill scripts. | ||
| Line 46: | Line 46: | ||
* A pass count of zero should return failure. | * A pass count of zero should return failure. | ||
* It would be neat if we could return the test results via a JSON blob somewhere rather than the current tradition of text-mangling. I am mainly thinking in terms of having a web UI that is better able to tell us where things went wrong. But that's just pipe-dreams and tinderpushlog with its magic already gets us more than 80% of the way towards usability... | * It would be neat if we could return the test results via a JSON blob somewhere rather than the current tradition of text-mangling. I am mainly thinking in terms of having a web UI that is better able to tell us where things went wrong. But that's just pipe-dreams and tinderpushlog with its magic already gets us more than 80% of the way towards usability... | ||
=== Intermittent MozMill Failures === | |||
I'm using this here just to get an initial idea of the sort of failures we're getting. I'll update it once or twice a day with the status from the period before. | |||
At some stage this will move into bugzilla, but let's get an idea of what is happening first. | |||
Note: Only tracking 1.9.1 initially due to regressions on trunk. | |||
==== 1.9.1 ==== | |||
From: 2009/07/28 09:41:01 | |||
To: 2009/07/28 19:12:02 | |||
Total Runs: 6 | |||
Total Green Runs: 4 | |||
===== Failure Mode 1 ===== | |||
Seen: once | |||
Result: 137/8 | |||
Short output: | |||
TEST-UNEXPECTED-FAIL | test_middle_click_with_many_things_selected_foreground | |||
TEST-UNEXPECTED-FAIL | test_middle_click_on_existing_single_selection_background | |||
TEST-UNEXPECTED-FAIL | test_middle_click_on_existing_single_selection_foreground | |||
TEST-UNEXPECTED-FAIL | test_middle_click_on_existing_multi_selection_foreground | |||
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run | |||
TEST-UNEXPECTED-FAIL | test_open_multiple_search_results_in_new_tabs | |||
TEST-UNEXPECTED-FAIL | test_verify_saved_search | |||
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run | |||
make: *** [mozmill] Error 1 | |||
Unknown Error: command finished with exit code: 2 | |||
Log: http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTest/1248799261.1248800444.11281.gz | |||
===== Failure Mode 2 ===== | |||
Seen: once | |||
Result: 142/2 | |||
Short output: | |||
TEST-UNEXPECTED-FAIL | test_summarization_goes_away | |||
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run | |||
make: *** [mozmill] Error 1 | |||
Unknown Error: command finished with exit code: 2 | |||
Log: http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTest/1248832277.1248833470.13976.gz | |||