Auto-tools/Projects/Mozharness

From MozillaWiki
Jump to: navigation, search

In the A-team we have recently been focusing on making mozharness easier to use by developers and give more flexibility in its use. Mozharness was originally started with Release Engineering and the A-team has been able to use it to contribute on the side of test jobs.

Some of the changes agreed to be tackled over the next couple of quarters are defined in Mozharness changes. We will migrate that page into here.

Introduction

This projects' goal

Every work day at Mozilla we run almost a 100,000 jobs a day and most are through mozharness. You can see some of them in here.

Mozharness is mainly developed by Release Engineering (Mozharness). This lines up well with the A-team's mission. This specific page project is making Mozharness easier for developers.

If you want to understand more about mozharness read this page.

Browsing the code

You can use dxr to browse Mozharness's code. You can also use mxr to search repositories that are involved with mozharness.

Filing bugs

If you find any issues running mozharness locally, please let us know by filing a bug. Please attach logs/log_raw.log to the bug it will help us see what you're facing.

Getting help

Join our IRC channel or ask questions in Google group (alternatively you can join the mirroring list).

If you're familiar with mozharness get your name in here so people can reach out to your on IRC:

  • armenzg - Helping out remove rough edges for developers
  • jlund - One of the main contributors
  • catlee - Long time contribuor
  • ahal - Long time contribuor
  • gbrown - - Long time contribuor - Familiar with Android scripts

Setup

Using mozharness is very easy. Here's what it takes to run it:

hg clone http://hg.mozilla.org/build/mozharness
cd mozharness

Check the "examples" section for seeing how to run mozharness locally.

NOTE: For some test jobs you will need LDAP credentials. This might limit the ability of community members to try those type of jobs.

Prerequisites

You need to have:

  • Mercurial
  • virtualenv

For Windows users you can get all this by installing MozillaBuild.

Examples

The following are some examples of test jobs that you can run locally: - Run a Firefox desktop reftest

  • On Linux:
python scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --reftest reftest \
--installer-url ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-35.0a1.en-US.linux-x86_64.tar.bz2 \
--test-url ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-35.0a1.en-US.linux-x86_64.tests.zip \
--cfg developer_config.py
  • On Windows:
python scripts/desktop_unittest.py --cfg unittests/win_unittest.py --reftest-suite reftest \
--installer-url ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-35.0a1.en-US.win32.zip \
--test-url ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/7/firefox-35.0a1.en-US.win32.tests.zip \
--cfg developer_config.py

- Run a Firefox for Android reftest:

python scripts/android_emulator_unittest.py --cfg android/androidarm.py --test-suite reftest-1 \
--installer-url ftp://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android/fennec-35.0a1.en-US.android-arm.apk \
--test-url ftp://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android//fennec-35.0a1.en-US.android-arm.tests.zip \
--cfg developer_config.py

- Run a Firefox OS emulator reftest job (NOTE: You will need LDAP credentials):

python scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite reftest --this-chunk 1 --total-chunks 20 \
--installer-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20141001060621/emulator.tar.gz \
--test-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20141001060621/b2g-35.0a1.en-US.android-arm.tests.zip \
--cfg developer_config.py

If you want to learn a bit more about different ways we can run jobs running in treeherder.mozilla.org locally, follow this article: "How to run Mozharness as a developer".

Testing your mozharness changes

You can try your changes locally and then ask for review in the bug.

Unit tests
You can run ./unit.sh to ensure that you pass all unit tests.

Testing live Mozharness patches can be tested on the TryServer if you have commit access and a user repo. You can read this and ask for information on how to get the required privileges to do so. Meanwhile, your mentor can help you test the changes for you.

Submitting your patches for review

You can do so by:

  • Create a patch by running "hg diff > my_patch.diff"
  • Click on "Add an attachment" and attach the file
  • Add a description on how this works and known issues
  • Change the feedback dropbox to "?" and enter the email address of the person will check your code
    • That person will receive an email notification and should be getting back to you in the next day or so

Milestones

Move and extend information used from in-tree mozharness configs

Status: [ON TRACK]

  • [DONE] - Move more mozharness configs to the tree (see here)
  • Extend which information is read from in-tree configs:
    • run_file_names [DONE] bug 1070041
    • all_$category_suites [DONE] bug 1070041
    • specific_tests_zip_dirs [ON TRACK]
    • extra_args [ON TRACK]

Value gained:

  • It will allow us to further experiment with try
  • It give us more control and flexibility
  • It does not require moving mozharness to the tree
  • It helps develop new platforms and new test job support
  • It reduces review burden for releng
  • It reduces the need to change mozharness code
    • Since we can change it in the tree
  • It unblocks other proposals below
  • It allow us to use these configs by mach

Separate production information from job definitions

Separate configuration information into three:

  • [mozharness] generic production relevant config (e.g. generic_config.py in here)
    • This config file can be used by many other jobs, however, specific values should go into their own (e.g. default_actions)
  • [mozharness] job specific values (e.g. mulet_config.py in here)
    • This config should have info with regards to which in-tree config to use and which actions to take
  • [in-tree] suite configuration information (e.g. [1])

Get rid of self.tree_config and only use self.config

jlund has more information about this.

Ability to specify a different output parser

STATUS: [DONE] bug 1068153

Value gained:

  • Switch to structured logging parsing

Lock mozharness' repository and version used from the tree

STATUS: [DONE] bug 791924

Value gained:

  • We can test on Try mozharness changes
    • We can not really test properly on Ash (stepping on each others' toes)
  • We remove the need for Ash, Cedar and Cypress

Pin mozharness on all trees

STATUS: {{ok|bug 1110286

Value gained:

  • We can easily backout
  • Reduced risk of breaking the tree across branches
  • Remove the need to merge to the production branch

Move harness command line options into configuration files

All command line options we pass to each harness should live within configuration files. Both mach and mozharness more or less have little test harness configuration in them. This makes it easy to unpack a test package and easily point to a configuration file. This depends on fully moving mozharness configs into the tree (see 3.1)

Value gained:

  • Both mach and mozharness have the same entry points
  • Make it easier to run outside of releng CI
  • There is less diverging paths making maintenance cost lower

Unified return code

Set the results status out of mozharness so that one can be sure that the overall result you get from a mach command is the same as the overall result you would get from a run on infra, given the same output.

Value gained:

  • Both mach and mozharness can output the same result

Move core mozharness functionality to modules

We now have all of the setup being done on the mozharness world (virtualenv, downloads, extracting). We don't have good libraries on the in-tree side to do any of this work. The proposal would be to create mozbase packages that give us the ability to create in-tree scripts that have the power and reliability that mozharness gives.

The proposed functionality to modularize are:

  • mozdownload
  • mozextract
  • mozvirtualenv
  • mozvcs

NOTE: At first, these packages do not need to live inside of mozilla-central since we're not planning on including them inside test.zip files.
NOTE2: This project is likely not to happen before any of the others ones happen first. We want to get a pulse of where we stand at that point.

Value gained:

  • Capacity to write scripts that are well suited to run reliably in our CI and meeting expectations
    • We can write them in a more python-fashioned way
    • Easier to write bootstrapping code
      • Mozharness currently does this by using subclassing and Mixin
  • Make mozharness scripts much simpler
    • Download specified in-tree script and run it
    • Less setup in the mozharness side
    • Easier to write tests for mozharness scripts
    • Other teams can write more reliable automation in their own setups
  • Mozharness core functionality becomes part of the tree
    • We get try support
  • Mozharness core functionality is modularized
    • It increases the amount of people that can contribute to the core mozharness functionality

Cons:

  • We would have to re-write mozharness scripts to use these new mozbase packages
    • We need to make sure we don't have two sets of libraries or we will fall apart

Experimenting - Using python packages freely within mozharness

This needs to be explored (minimum viable product), scoped and bugs filed.

Here's an example of a python package being used within mozharness: http://hg.mozilla.org/build/mozharness/file/default/mozharness/mozilla/testing/mozpool.py#l42

Scope ideas:

  • Find mozharness functionality and move into their own python packages (e.g. bug 1109799)
  • Modify mozharness to include that new python package
  • Test that it works and explain to others how you made it happen
  • Spot which other mozharness functionalisties could be moved into these new python packages
  • Bring the conversation to the mozharness table and we can review it and file bugs

Where can I help?

In order to make mozharness easier for developers we have to make it a delightful and well integrated experience. If you want to help, you can find the list of bugs needed to accomplish listed by difficulty.

Good first easier-mozharness bugs

Link to the list of bugs.
These are good starting bugs:

ID Summary Priority Status

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Good first bugs

There are also bugs not specific to making mozharness easier, however, they also help release engineering's load. This is a super set of the previous section. Link to the list of bugs.
These are good starting bugs:

ID Summary Priority Status
1151955 push_apk.py should use the future get_apk.py from bug 1151954 -- NEW

1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Good next bugs

Link to the list of bugs.
If the good first bugs are depleted or want to try something a bit more complicated (NOTE: these bugs might be a bit more challenging):

ID Summary Priority Status
965027 mozharness download_file should verify SSL certs -- NEW
991178 Add support for more Python exceptions to Mozharness' errors.py -- NEW
1007280 Allow Mozharness to have r/w options (live_config) with caching capabilities to help individual steps easily -- NEW
1007292 for Mozharness, hide Buildbot integration -- NEW
1036468 reduce overhead in job setup in buildbot as well as test scripts + mozharness -- NEW
1087664 Move download logic in ScriptMixin to mozharness/lib/python/download.py -- NEW
1113174 Make spidermonkey mozharness builds to be developer friendly -- NEW

7 Total; 7 Open (100%); 0 Resolved (0%); 0 Verified (0%);


All easier-mozharness bugs

Link to the list of bugs.
These bugs are likely to only require a bit of mozharness hacking and a lot of other code repositories.

ID Summary Priority Status
978865 Separate out mozharness scripts from build properties -- NEW
995118 Devise a standard way for mozharness to pass down a "your timeout should be less than mine" to test harnesses -- NEW
1007280 Allow Mozharness to have r/w options (live_config) with caching capabilities to help individual steps easily -- NEW
1007292 for Mozharness, hide Buildbot integration -- NEW
1067354 Improve documentation for non-test type of jobs (e.g. rooting analysis) and find ways to run it locally -- NEW
1068757 Mozharness unit tests fail with newer versions of Mercurial -- NEW
1073583 Establish an approach to ensure minimum requirements are met -- NEW
1078638 Talos for mozharness cannot be run against a local file or an ftp url -- NEW
1087664 Move download logic in ScriptMixin to mozharness/lib/python/download.py -- NEW
1094378 For mozharness test jobs we should output a line saying "Run these steps if you want to run this job locally" -- NEW
1111835 Separate mozharness' virtualenv from the scripts' virtualenv -- NEW
1113174 Make spidermonkey mozharness builds to be developer friendly -- NEW

12 Total; 12 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Completed bugs

Bugs fixed in September

ID Summary Priority Status
1019962 Make some mozharness jobs easier to run outside of automation + allow for http authentication when running outside of the releng network -- RESOLVED
1064002 Add {mochitest-gl,robocop,jsreftest,crashtest} suite_definitions to in-tree configurations -- RESOLVED
1066700 Unzip on developer's Mac do not like test.zip files produced in releng infra -- RESOLVED
1070041 Unify in-tree configs, move run_filename into the tree and move more options into the tree -- RESOLVED
1075896 Do not request for http credentials for developer mode unless we actually need it -- RESOLVED
1075992 If download-and-extract is not run then we don't call _read_tree_config and can't run-tests -- RESOLVED
1076172 Make mozharness http authentication for developer mode to be easier to use repeteadly -- RESOLVED
1078619 Allow to run talos jobs as a developer -- RESOLVED
1078644 No curated guide on how Mozharness works and how to write for it -- RESOLVED
1079579 Move the spidermonkey automation build scripts into the tree -- RESOLVED
1086672 Running android_emulator_unittest.py fails locally due to missing files -- RESOLVED
1093749 Allow b2g build to download a local mozilla-central repository -- RESOLVED
1113081 Remove references to metro mode from talos.py script -- RESOLVED
1127449 For developer mode, do not store the credentials in plain text if the developer does not want to -- RESOLVED

14 Total; 0 Open (0%); 14 Resolved (100%); 0 Verified (0%);