Auto-tools/Projects/Mozharness: Difference between revisions

← Older edit

Auto-tools/Projects/Mozharness (view source)

Revision as of 13:44, 13 February 2018

5,613 bytes added , 13 February 2018

m

remove my name

Armenzg

Confirmed users

3,990

edits

@@ Line 26: / Line 26: @@
 If you're familiar with mozharness get your name in here so people can reach out to your on IRC:
-* [https://mozillians.org/en-US/u/armenzg armenzg] - Helping out remove rough edges for developers
 * [https://mozillians.org/en-US/u/jlund jlund] - One of the main contributors
 * [https://mozillians.org/en-US/u/catlee catlee] - Long time contribuor
@@ Line 75: / Line 74: @@
   --cfg developer_config.py
-If you want to learn a bit more about different ways we can run jobs running in tbpl.mozilla.org locally, follow this article: "[[ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer|How to run Mozharness as a developer]]".
+If you want to learn a bit more about different ways we can run jobs running in treeherder.mozilla.org locally, follow this article: "[[ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer|How to run Mozharness as a developer]]".
 = Testing your mozharness changes =
 You can try your changes locally and then ask for review in the bug.<br />
+'''Unit tests'''<br />
+You can run ./unit.sh to ensure that you pass all unit tests.
+'''Testing live'''
+Mozharness patches can be tested on the TryServer if you have commit access and a user repo.
+You can [http://armenzg.blogspot.ca/2014/11/pinning-mozharness-from-in-tree-aka.html read this] and ask for information on how to get the required privileges to do so.
+Meanwhile, your mentor can help you test the changes for you.
+= Submitting your patches for review =
 You can do so by:
 * Create a patch by running "hg diff > my_patch.diff"
@@ Line 87: / Line 95: @@
 ** That person will receive an email notification and should be getting back to you in the next day or so
-'''Unit tests'''<br />
+= Milestones =
-You can run ./unit.sh to ensure that you pass all unit tests.
+== Move and extend information used from in-tree mozharness configs ==
+Status: {{ok|}}
+* {{bug|1067535}} and {{bug|1070041}}
+* {{done|}} - Move more mozharness configs to the tree (see [http://hg.mozilla.org/mozilla-central/file/default/testing/config/mozharness here])
+* Extend which information is read from in-tree configs:
+** run_file_names {{done|{{bug|1070041}} }}
+** all_$category_suites {{done|{{bug|1070041}} }}
+** specific_tests_zip_dirs {{ok|}}
+** extra_args {{ok|}}
+Value gained:
+* It will allow us to further experiment with try
+* It give us more control and flexibility
+* It does not require moving mozharness to the tree
+* It helps develop new platforms and new test job support
+* It reduces review burden for releng
+* It reduces the need to change mozharness code
+** Since we can change it in the tree
+* It unblocks other proposals below
+* It allow us to use these configs by mach
+== Separate production information from job definitions ==
+Separate configuration information into three:
+* [mozharness] generic production relevant config (e.g. generic_config.py in [https://bug1043699.bugzilla.mozilla.org/attachment.cgi?id=8494540 here])
+** This config file can be used by many other jobs, however, specific values should go into their own (e.g. default_actions)
+* [mozharness] job specific values (e.g. mulet_config.py in [https://bug1043699.bugzilla.mozilla.org/attachment.cgi?id=8494540 here])
+** This config should have info with regards to which in-tree config to use and which actions to take
+* [in-tree] suite configuration information (e.g. [https://bug1043699.bugzilla.mozilla.org/attachment.cgi?id=8494000])
+== Get rid of self.tree_config and only use self.config ==
+jlund has more information about this.
+== Ability to specify a different output parser ==
+'''STATUS''': {{done|{{bug|1068153}}}}
+Value gained:
+* Switch to structured logging parsing
+== Lock mozharness' repository and version used from the tree ==
+'''STATUS''': {{done|{{bug|791924}}}}
+Value gained:
+* We can test on Try mozharness changes
+** We can not really test properly on Ash (stepping on each others' toes)
+* We remove the need for Ash, Cedar and Cypress
+== Pin mozharness on all trees ==
+'''STATUS''': {{ok|{{bug|1110286}}
+Value gained:
+* We can easily backout
+* Reduced risk of breaking the tree across branches
+* Remove the need to merge to the production branch
+== Move harness command line options into configuration files ==
+All command line options we pass to each harness should live within configuration files.
+Both mach and mozharness more or less have little test harness configuration in them.
+This makes it easy to unpack a test package and easily point to a configuration file.
+This depends on fully moving mozharness configs into the tree (see 3.1)
+Value gained:
+* Both mach and mozharness have the same entry points
+* Make it easier to run outside of releng CI
+* There is less diverging paths making maintenance cost lower
+== Unified return code ==
+Set the results status out of mozharness so that one can be sure that the overall result you get from a mach command is the same as the overall result you would get from a run on infra, given the same output.
+Value gained:
+* Both mach and mozharness can output the same result
+== Move core mozharness functionality to modules  ==
+We now have all of the setup being done on the mozharness world (virtualenv, downloads, extracting).
+We don't have good libraries on the in-tree side to do any of this work.
+The proposal would be to create mozbase packages that give us the ability to create in-tree scripts that have the power and reliability that mozharness gives.
+The proposed functionality to modularize are:
+* mozdownload
+* mozextract
+* mozvirtualenv
+* mozvcs
+NOTE: At first, these packages do not need to live inside of mozilla-central since we're not planning on including them inside test.zip files.<br />
+NOTE2: This project is likely not to happen before any of the others ones happen first. We want to get a pulse of where we stand at that point.
+Value gained:
+* Capacity to write scripts that are well suited to run reliably in our CI and meeting expectations
+** We can write them in a more python-fashioned way
+** Easier to write bootstrapping code
+*** Mozharness currently does this by using subclassing and Mixin
+* Make mozharness scripts much simpler
+** Download specified in-tree script and run it
+** Less setup in the mozharness side
+** Easier to write tests for mozharness scripts
+** Other teams can write more reliable automation in their own setups
+* Mozharness core functionality becomes part of the tree
+** We get try support
+* Mozharness core functionality is modularized
+** It increases the amount of people that can contribute to the core mozharness functionality
+Cons:
+* We would have to re-write mozharness scripts to use these new mozbase packages
+** We need to make sure we don't have two sets of libraries or we will fall apart
+= Experimenting - Using python packages freely within mozharness =
+This needs to be explored (minimum viable product), scoped and bugs filed.
+Here's an example of a python package being used within mozharness:
+http://hg.mozilla.org/build/mozharness/file/default/mozharness/mozilla/testing/mozpool.py#l42
-'''Testing live'''
+Scope ideas:
-Mozharness patches can be tested on the TryServer if you have commit access and a user repo.
+* Find mozharness functionality and move into their own python packages (e.g. {{bug|1109799}})
-You can [http://armenzg.blogspot.ca/2014/11/pinning-mozharness-from-in-tree-aka.html read this] and ask for information on how to get the required privileges to do so.
+* Modify mozharness to include that new python package
-Meanwhile, your mentor can help you test the changes for you.
+* Test that it works and explain to others how you made it happen
+* Spot which other mozharness functionalisties could be moved into these new python packages
+* Bring the conversation to the mozharness table and we can review it and file bugs
 = Where can I help? =