Thunderbird:Testing/Automated Update Testing

From MozillaWiki
Jump to: navigation, search

The automated update testing suite is based on MozMill and will automatically test updates for multiple revisions on specified update channels.

Setting up

(this will need to be done for each platform).

  • Install MozMill as per this page
    • You may also need to easy_install Mercurial (standard8 has problems with this on windows).
  • Create an empty directory, e.g. update-tests
mkdir update-tests
cd update-tests
  • Clone the required repository
hg clone http://hg.mozilla.org/qa/mozmill-automation/
  • Get the script to download the builds from bug 580237, and create the download directory for the builds
wget -O download.py --no-check-certificate \
  https://bugzilla.mozilla.org/attachment.cgi?id=459821
chmod 755 download.py
mkdir builds

You are now ready to run the tests.

Running Tests

  • Make sure you are in the update-tests directory
  • Download one or more builds to test update from, these do not have to be the same version (e.g. 3.0.x and 3.1.x builds to test together would be fine)
    • Firefox typically test en-US plus two to four other locales for each point release.
./download.py -v 3.1 -p mac -l en-US

Parameters: -v <version> -l <locale> -p <platform> Note: Platform is one off win, mac, linux

  • Run the update tests:
./mozmill-automation/testrun_update.py --channel=release \
  --application=thunderbird builds

Parameters: --channel is the main one to change here.

  • Once the update tests are complete, you'll get a pass or fail summary for each test, including:
    • What was updated to/from
    • Locale
    • Build IDs

Useful Notes

  • The builds downloaded into builds/ are left unchanged, so you can test against them multiple times, e.g. for different update channels (releasetest, release etc).
  • The tests use their own profile
  • The automation code downloads a second repository that contains the actual javascript files for running the mozmill tests, that code can be found here.