MozillaWiki

Auto-tools/Projects/Mozharness: Difference between revisions

Page Discussion

Auto-tools/Projects/Mozharness (view source)

Revision as of 13:44, 13 February 2018

8,675 bytes added ,  13 February 2018
m
remove my name
m (remove my name)
 
(10 intermediate revisions by 2 users not shown)
Line 5: Line 5:
We will migrate that page into here.
We will migrate that page into here.


= Understanding mozharness =
= Introduction =
If you want to understand more about mozharness read [https://developer.mozilla.org/en-US/docs/Mozharness_FAQ this FAQ].
== This projects' goal ==
Every work day at Mozilla we run almost a 100,000 jobs a day and most are through mozharness. You can see some of them in [https://treeherder.mozilla.org/#/jobs?repo=mozilla-central here].


= Filing bugs =
Mozharness is mainly developed by Release Engineering ([[ReleaseEngineering/Mozharness|Mozharness]]). This lines up well with the [[Auto-tools#Our_Mission|A-team's mission]].
This specific page project is '''making Mozharness easier for developers'''.
 
If you want to understand more about mozharness read [http://moz-releng-docs.readthedocs.org/en/latest/software.html#mozharness this page].
 
== Browsing the code ==
You can use [http://dxr.mozilla.org/build:mozharness/source dxr] to browse Mozharness's code.
You can also use [http://mxr.mozilla.org/build mxr] to search repositories that are involved with mozharness.
 
== Filing bugs ==
If you find any issues running mozharness locally, please let us know by [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Mozharness filing a bug].
If you find any issues running mozharness locally, please let us know by [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Mozharness filing a bug].
If you could attach logs/log_raw.log to the bug it will help us see what you're facing.
Please attach logs/log_raw.log to the bug it will help us see what you're facing.
 
== Getting help ==
Join our [https://chat.mibbit.com/?url=irc%3A%2F%2Firc.mozilla.org%2F%23ateam IRC channel] or ask questions in [https://groups.google.com/forum/#!forum/mozilla.tools Google group] (alternatively you can join [https://lists.mozilla.org/listinfo/tools the mirroring list]).
 
If you're familiar with mozharness get your name in here so people can reach out to your on IRC:
* [https://mozillians.org/en-US/u/jlund jlund] - One of the main contributors
* [https://mozillians.org/en-US/u/catlee catlee] - Long time contribuor
* [https://mozillians.org/en-US/u/ahal ahal] - Long time contribuor
* [https://mozillians.org/en-US/u/geoffbrown gbrown] - - Long time contribuor - Familiar with Android scripts


= Setup =
= Setup =
Line 17: Line 36:
  hg clone http://hg.mozilla.org/build/mozharness
  hg clone http://hg.mozilla.org/build/mozharness
  cd mozharness
  cd mozharness
Check the "examples" section for seeing how to run mozharness locally.


NOTE: For some test jobs you will need LDAP credentials. This might limit the ability of community members to try those type of jobs.
NOTE: For some test jobs you will need LDAP credentials. This might limit the ability of community members to try those type of jobs.
Line 53: Line 74:
  --cfg developer_config.py
  --cfg developer_config.py


If you want to learn a bit more about different ways we can run jobs running in tbpl.mozilla.org locally, follow this article: "[[ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer|How to run Mozharness as a developer]]
If you want to learn a bit more about different ways we can run jobs running in treeherder.mozilla.org locally, follow this article: "[[ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer|How to run Mozharness as a developer]]".


= Testing your mozharness changes =
= Testing your mozharness changes =
You can try your changes locally and then ask for review.
You can try your changes locally and then ask for review in the bug.<br />
In the future, we would like to allow for testing of changes on the try server: {{bug|791924}}
 
'''Unit tests'''<br />
You can run ./unit.sh to ensure that you pass all unit tests.
 
'''Testing live'''
Mozharness patches can be tested on the TryServer if you have commit access and a user repo.
You can [http://armenzg.blogspot.ca/2014/11/pinning-mozharness-from-in-tree-aka.html read this] and ask for information on how to get the required privileges to do so.
Meanwhile, your mentor can help you test the changes for you.
 
= Submitting your patches for review =
You can do so by:
* Create a patch by running "hg diff > my_patch.diff"
* Click on "Add an attachment" and attach the file
* Add a description on how this works and known issues
* Change the feedback dropbox to "?" and enter the email address of the person will check your code
** That person will receive an email notification and should be getting back to you in the next day or so
 
= Milestones =
== Move and extend information used from in-tree mozharness configs ==
Status: {{ok|}}
* {{bug|1067535}} and {{bug|1070041}}
 
* {{done|}} - Move more mozharness configs to the tree (see [http://hg.mozilla.org/mozilla-central/file/default/testing/config/mozharness here])
* Extend which information is read from in-tree configs:
** run_file_names {{done|{{bug|1070041}} }}
** all_$category_suites {{done|{{bug|1070041}} }}
** specific_tests_zip_dirs {{ok|}}
** extra_args {{ok|}}
 
Value gained:
* It will allow us to further experiment with try
* It give us more control and flexibility
* It does not require moving mozharness to the tree
* It helps develop new platforms and new test job support
* It reduces review burden for releng
* It reduces the need to change mozharness code
** Since we can change it in the tree
* It unblocks other proposals below
* It allow us to use these configs by mach
 
== Separate production information from job definitions ==
Separate configuration information into three:
* [mozharness] generic production relevant config (e.g. generic_config.py in [https://bug1043699.bugzilla.mozilla.org/attachment.cgi?id=8494540 here])
** This config file can be used by many other jobs, however, specific values should go into their own (e.g. default_actions)
* [mozharness] job specific values (e.g. mulet_config.py in [https://bug1043699.bugzilla.mozilla.org/attachment.cgi?id=8494540 here])
** This config should have info with regards to which in-tree config to use and which actions to take
* [in-tree] suite configuration information (e.g. [https://bug1043699.bugzilla.mozilla.org/attachment.cgi?id=8494000])
 
== Get rid of self.tree_config and only use self.config ==
jlund has more information about this.
 
== Ability to specify a different output parser ==
'''STATUS''': {{done|{{bug|1068153}}}}
 
Value gained:
* Switch to structured logging parsing
 
== Lock mozharness' repository and version used from the tree ==
'''STATUS''': {{done|{{bug|791924}}}}
 
Value gained:
* We can test on Try mozharness changes
** We can not really test properly on Ash (stepping on each others' toes)
* We remove the need for Ash, Cedar and Cypress
 
== Pin mozharness on all trees ==
'''STATUS''': {{ok|{{bug|1110286}}
 
Value gained:
* We can easily backout
* Reduced risk of breaking the tree across branches
* Remove the need to merge to the production branch
 
== Move harness command line options into configuration files ==
All command line options we pass to each harness should live within configuration files.
Both mach and mozharness more or less have little test harness configuration in them.
This makes it easy to unpack a test package and easily point to a configuration file.
This depends on fully moving mozharness configs into the tree (see 3.1)
 
Value gained:
* Both mach and mozharness have the same entry points
* Make it easier to run outside of releng CI
* There is less diverging paths making maintenance cost lower
 
== Unified return code ==
Set the results status out of mozharness so that one can be sure that the overall result you get from a mach command is the same as the overall result you would get from a run on infra, given the same output.
 
Value gained:
* Both mach and mozharness can output the same result
 
== Move core mozharness functionality to modules  ==
We now have all of the setup being done on the mozharness world (virtualenv, downloads, extracting).
We don't have good libraries on the in-tree side to do any of this work.
The proposal would be to create mozbase packages that give us the ability to create in-tree scripts that have the power and reliability that mozharness gives.
 
The proposed functionality to modularize are:
* mozdownload
* mozextract
* mozvirtualenv
* mozvcs
 
NOTE: At first, these packages do not need to live inside of mozilla-central since we're not planning on including them inside test.zip files.<br />
NOTE2: This project is likely not to happen before any of the others ones happen first. We want to get a pulse of where we stand at that point.
 
Value gained:
* Capacity to write scripts that are well suited to run reliably in our CI and meeting expectations
** We can write them in a more python-fashioned way
** Easier to write bootstrapping code
*** Mozharness currently does this by using subclassing and Mixin
* Make mozharness scripts much simpler
** Download specified in-tree script and run it
** Less setup in the mozharness side
** Easier to write tests for mozharness scripts
** Other teams can write more reliable automation in their own setups
* Mozharness core functionality becomes part of the tree
** We get try support
* Mozharness core functionality is modularized
** It increases the amount of people that can contribute to the core mozharness functionality
 
Cons:
* We would have to re-write mozharness scripts to use these new mozbase packages
** We need to make sure we don't have two sets of libraries or we will fall apart
 
= Experimenting - Using python packages freely within mozharness =
This needs to be explored (minimum viable product), scoped and bugs filed.
 
Here's an example of a python package being used within mozharness:
http://hg.mozilla.org/build/mozharness/file/default/mozharness/mozilla/testing/mozpool.py#l42
 
Scope ideas:
* Find mozharness functionality and move into their own python packages (e.g. {{bug|1109799}})
* Modify mozharness to include that new python package
* Test that it works and explain to others how you made it happen
* Spot which other mozharness functionalisties could be moved into these new python packages
* Bring the conversation to the mozharness table and we can review it and file bugs


= Where can I help? =
= Where can I help? =
Line 63: Line 218:
If you want to help, you can find the list of bugs needed to accomplish listed by difficulty.
If you want to help, you can find the list of bugs needed to accomplish listed by difficulty.


== Good first bugs ==
== Good first easier-mozharness bugs ==
[https://bugzilla.mozilla.org/buglist.cgi?list_id=11281800&resolution=---&emailtype1=exact&status_whiteboard_type=allwordssubstr&query_format=advanced&emailassigned_to1=1&status_whiteboard=%5Bgood%20first%20bug%5D%5Beasier-mozharness%5D&email1=nobody%40mozilla.org&component=Mozharness&product=Release%20Engineering Link to the list of bugs].<br />
[https://bugzilla.mozilla.org/buglist.cgi?list_id=11281800&resolution=---&emailtype1=exact&status_whiteboard_type=allwordssubstr&query_format=advanced&emailassigned_to1=1&status_whiteboard=%5Bgood%20first%20bug%5D%5Beasier-mozharness%5D&email1=nobody%40mozilla.org&component=Mozharness&product=Release%20Engineering Link to the list of bugs].<br />
These are good starting bugs:
These are good starting bugs:
Line 72: Line 227:
         "component": "Mozharness",
         "component": "Mozharness",
         "whiteboard": "[good first bug][easier-mozharness]"
         "whiteboard": "[good first bug][easier-mozharness]"
    }
</bugzilla>
== Good first bugs ==
There are also bugs not specific to making mozharness easier, however, they also help release engineering's load.
This is a super set of the previous section.
[https://bugzilla.mozilla.org/buglist.cgi?list_id=11281800&resolution=---&emailtype1=exact&status_whiteboard_type=allwordssubstr&query_format=advanced&emailassigned_to1=1&status_whiteboard=%5Bgood%20first%20bug%5D&email1=nobody%40mozilla.org&component=Mozharness&product=Release%20Engineering Link to the list of bugs].<br />
These are good starting bugs:
<bugzilla>
    {
        "quicksearch": "status:new,assigned,reopened,unconfirmed",
        "product": "Release Engineering",
        "component": "Mozharness",
        "whiteboard": "[good first bug]"
     }
     }
</bugzilla>
</bugzilla>


== Good next bugs ==
== Good next bugs ==
[https://bugzilla.mozilla.org/buglist.cgi?list_id=11281800&resolution=---&emailtype1=exact&status_whiteboard_type=allwordssubstr&query_format=advanced&emailassigned_to1=1&status_whiteboard=%5Bgood%20next%20bug%5D%5Beasier-mozharness%5D&email1=nobody%40mozilla.org&component=Mozharness&product=Release%20Engineering Link to the list of bugs].<br />
[https://bugzilla.mozilla.org/buglist.cgi?list_id=11281800&resolution=---&emailtype1=exact&status_whiteboard_type=allwordssubstr&query_format=advanced&emailassigned_to1=1&status_whiteboard=%5Bgood%20next%20bug%5D&email1=nobody%40mozilla.org&component=Mozharness&product=Release%20Engineering Link to the list of bugs].<br />
If the good first bugs are depleted or want to try something a bit more complicated (NOTE: these bugs might be a bit more challenging):
If the good first bugs are depleted or want to try something a bit more complicated (NOTE: these bugs might be a bit more challenging):
<bugzilla>
<bugzilla>
Line 83: Line 252:
         "product": "Release Engineering",
         "product": "Release Engineering",
         "component": "Mozharness",
         "component": "Mozharness",
         "whiteboard": "[good next bug][easier-mozharness]"
         "whiteboard": "[good next bug]"
     }
     }
</bugzilla>
</bugzilla>
Armenzg
Confirmed users
3,990

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1021640...1188937"