MozillaWiki

ReleaseEngineering/TryServer: Difference between revisions

Page Discussion

ReleaseEngineering/TryServer (view source)

Revision as of 22:21, 5 March 2020

9,492 bytes added ,  5 March 2020
m
→‎Re-running tasks with custom parameters from Treeherder
 
(123 intermediate revisions by 55 users not shown)
Line 1: Line 1:
= Try Server =
= Try Server =
The try server is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.
The [https://treeherder.mozilla.org/#/jobs?repo=try try server] is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.


To use try server, you need a [http://www.mozilla.org/hacking/committer/ Mozilla hg account] ([http://www.mozilla.org/hacking/commit-access-policy/ level 1] is sufficient).
== Getting access to the Try Server  ==
To use the try server, you'll need [http://www.mozilla.org/hacking/commit-access-policy/ Level 1 Commit Access]. You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: [http://www.mozilla.org/hacking/committer/ Becoming a Mozilla Contributor]


'''Breaking News''' - an experimental feature now allows you to push to try via Bugzilla - see [https://wiki.mozilla.org/Build:Autoland Autoland!]
After you have level 1 commit access, you'll need to do a bit of setup before you'll be able to push. See [https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/auth.html this guide] for instructions.
 
== Configuration ==
Before using try, there is some recommended configuration you should set up. This can be accomplished by running:
 
$ ./mach vcs-setup
 
Be sure to at least enable the [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html firefoxtree extension] and the push-to-try extension. The push-to-try extension is required if you wish to use the |mach try| command (see below). Firefoxtree will give you a handy 'try' alias you can use for pushing, and prevent you from accidentally pushing multiple heads. If for some reason you prefer not to use firefoxtree, you can set the same alias up manually in your hgrc:
 
[paths]
try = ssh://hg.mozilla.org/try
 
Note: Never pull the whole try repo. You'll end up with hundreds of heads containing everyone's half baked and broken pushes.
 
For more information, see [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html mercurial for mozillians].


== How to push to try ==
== How to push to try ==
=== Running a subset of builds/test/talos available to Try===
There are a few ways to schedule jobs on try.
You must use [[Build:TryChooser]] to choose which builds, tests, and talos you would like run on your push to try. Make sure you place the try chooser text in your ''topmost commit''. The [http://people.mozilla.org/~lsblakk/trychooser/ TryChooser] web page can help you build a commit message for custom requests, so can the [https://github.com/pbiggar/trychooser mercurial extension]
 
=== Attach new jobs from the review ===
 
For every patch submitted for review in Phabricator, a new Try run is automatically created. This run is created for static analysis, linting and other tasks.
As described below, attaching new jobs to the run is easy and doesn't require more actions from the developer.
Just click on the ''Treeherder Jobs''.
 
[[File:Screenshot 2019-10-07 ⚙ D48316 Bug 1585593 - Enable Fission for Raptor.png]]
 
For more details, see:
https://firefox-source-docs.mozilla.org/tools/try/index.html#attaching-new-jobs-from-a-review
 
=== Scheduling jobs with Mach Try ===
 
The recommended way to [https://firefox-source-docs.mozilla.org/tools/try/index.html interact with try] is via the command line tool |mach try|. Since try can be used for a wide range of use cases, by developers with varying levels of experience, there is no one size fits all solution when it comes to selecting which tasks to run. Therefore |mach try| offers a range of subcommands called "selectors" for a variety of purposes. To see a list of available selectors, run:
 
  $ mach try --help
 
For deeper information on a given selector, run:
 
  $ mach try <selector> --help
 
If you don't specify a selector/subcommand to try, it will default to using the `syntax` selector. You can change which selector gets used by default by creating ~/.mozbuild/machrc and adding:
 
  [try]
  default=<selector>
 
See [https://firefox-source-docs.mozilla.org/tools/try/selectors/index.html this documentation] for more information on all the available selectors, and [https://firefox-source-docs.mozilla.org/tools/try/presets.html this document] for a list of available presets (derived from [https://searchfox.org/mozilla-central/source/tools/tryselect/try_presets.yml try_presets.yml]).
 
 
=== Scheduling jobs with Treeherder ===
 
You can push to try with:
 
$ ./mach try empty
 
Once the push succeeds, you can open it in Treeherder and then sign in to schedule jobs manually.
 
Select the drop-down arrow at the top right of your push, and choose "Add new Jobs". It might take several seconds for the jobs to load ({{bug|1288028}}) :
 
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042414/TH_dropdown.png
 
If it helps, you can use the filter box in the site header to restrict the list of runnable jobs being displayed.  Click the job symbols you wish to schedule.
 
If you select a test job, the required build will automatically be scheduled:
 
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042426/TH_with_jobs.png
 
Finally, click "Trigger New Jobs" near the top right of your push.
 
NOTE: An action task gets scheduled which will schedule all the tasks you chose.
 
=== Scheduling jobs with Treeherder (Search) ===
 
Similar to the above method, there is now an additional way to add jobs to an existing push via Treeherder. Open the push in Treeherder and then choose "Add New Jobs (Search)" from the push's menu:
 
[[File:Add New Jobs Search.png]]
 
Treeherder will then fetch the list of runnable jobs in the background, then pop up a dialog window listing those jobs:
 
[[File:Addnewjobssearchinterface.png]]


<pre>
At the top of the dialog, there is a search bar. Type part of your desired job name into the search bar and press 'Enter' to filter the list of jobs. (This uses a fuzzy search library, so some minimal typos are allowed). From the filtered list of jobs, select as many of the results as you want (ctrl/shift modifiers work on this list for selecting multiple jobs):
% hg qref --message "try: -b o -e -p all -u all -t none"
% hg push -f try
</pre>


=== Pushing to try ===
[[File:Filteredaddnewjobssearch.png]]


To submit your changes to the try server (assuming they're modifications to mozilla-central or a similar branch, e.g. tracemonkey), you have a few options:
Then click the green "Add Selected" button to add the selected jobs to the list of jobs to be run (or click the green "Add All" button to add the entire list of jobs currently in view.
* <code>hg push -f ssh://hg.mozilla.org/try/</code>, or
* <code>hg push -f ssh://&lt;username@host&gt;@hg.mozilla.org/try/</code> <strike>?</strike>, or
* <code>hg push -f ssh://hg.mozilla.org/try/ -e 'ssh -l &lt;username@host&gt;</code>


==== Creating an alias ====
Repeat the filter and selection process until you have all of your desired jobs added to the list. You can remove jobs from the list to be run using the red buttons lower on the dialog if you selected them in error:
To save yourself some typing, you can add an alias to your hgrc:
<pre>[paths]
try = ssh://hg.mozilla.org/try
</pre>
and then push with
<pre>hg push -f try</pre>


==== ~/.ssh/config ====
[[File:Addnewjobssearchselectedjobs.png]]
ssh host settings to make life easier


<pre>
When you are happy with your selection, click the blue "Trigger (X) Selected Jobs" button at the bottom of the dialog. Treeherder will send out a request for those jobs to be run, and an action task will be spawned to do all of the behind the scenes magic to make that happen.
Host hg.mozilla.org
  User <commit_user_email_address>
  # IdentityFile ~/.ssh/alternate_public_identity_key_file
</pre>


=== hg phases causing "abort: popping would remove an immutable revision" ===
=== Pushing Directly ===
<p>[http://mercurial.selenic.com/wiki/Phases hg phases] were introduced in Mercurial 2.1.  Their purpose is to keep you from accidentally modifying changesets you've shared with others.  In particular, this keeps you from qpop'ing patches you've pushed to try!


You can force hg to let you <tt>qpop</tt> your patch queue by running
It's also possible to push to try directly from hg (for now). This might be required if you're using [https://www.mercurial-scm.org/wiki/MqExtension mercurial queues]. In this case, the only way to select tasks is by manually adding try syntax to your commit message:
<pre>hg phase -f --secret qparent:tip</pre>


You can probably turn off phases in your hgrc, but I haven't figured out how.  If you figure it out, please update this page!
  $ hg commit -m "try: -b o -p linux64 -u mochitest"
  $ hg push -f try


=== Viewing the results ===
Using |mach try| is recommended as that will automatically create and cleanup a temporary commit no your behalf, whereas with this method you are responsible for cleaning up any extra commits you make or commit messages you modify.


== Viewing the results ==
You can see the results of your tryserver build in a number of ways:
You can see the results of your tryserver build in a number of ways:


* You'll get an email on a successful push with a link to tbpl for your revision as well as emails on any non-successful build/test/talos results (this setting can be adjusted using [[Build:TryChooser]] args for email notification)
* You'll get an email on a successful push with a link to treeherder for your revision as well as optional emails on any non-successful build/test/talos results
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax (see: [[Build:TryChooser]] for examples)
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax
* Look for your changeset on the [http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTry Try Tinderbox] or [http://tbpl.mozilla.org/?tree=Try Try TBPL] ([http://ehsanakhgari.org/tinderboxpushlog/index.html?tree=MozillaTry alternate TBPL] [http://bbpl.dbaron.org/?tree=MozillaTry alternate TBPL]). You can add '''&pusher=YOUR.EMAIL''' to only see your pushes.
* The link to treeherder will be printed on the command line.
* Compare Talos perf numbers using Pike's [http://github.com/Pike/talos-node talos-node] or mconnor's [http://perf.snarkfest.net/compare-talos/ compare-talos].
* Look for your changeset on [https://treeherder.mozilla.org/#/jobs?repo=try Treeherder]. You can add '''&author=YOUR.EMAIL''' to only see your pushes.
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].


If you're using [https://developer.mozilla.org/en/Mercurial_Queues Mercurial queues], the <code>push -f</code> command pushes any patches that are currently applied, and the Try server will build the result. (This is an awesome feature, not a bug!)
== Using a custom mozconfig  ==
The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.
 
If you want to apply the same mozconfig changes to multiple platforms, you can edit <tt>build/mozconfig.common.override</tt> instead.  This file is included at the end of each of the in-tree mozconfig files.
 
Android mozconfigs are in mobile/android/config/mozconfigs.
 
If you modify a mozconfig file named <tt>nightly</tt>, please make sure you also change any mozconfig file named <tt>beta</tt> and/or <tt>release</tt> in the same directory, otherwise the test_compare_mozconfigs test will fail.
 
Note:
* TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.
** Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.
 
== Re-running tasks with custom parameters from Treeherder ==
 
It is possible to run a task again, with certain types of changes to its run time environment and parameters. This feature is commonly used to enable debug logging, or run a test repeatedly to reproduce test failures.
 
In treeherder, select the job you want to re-run (Note: it currently only works for [https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/actions/retrigger_mochitest.py#30-31 some jobs]). The details panel should open. Look for the 3 dots icon ("Other job details") and select "Custom actions...".
 
[[File:Custom-action.png|thumb|Selecting "Custom Action..." in treeherder.]]
 
The <tt>retrigger-custom</tt> action is the one selected in the drop-down menu. Modify the task parameters as desired and select "Trigger".  


You don’t need to clone or pull from the <code>try</code> repo, and you probably don’t want to. You’d get every half-baked changeset anybody ever tested.
[[File:Custom Taskcluster Job Actions.png|thumb|Treeherder modal to request an action associated to a task]]


See [http://blog.mozilla.com/jorendorff/2008/08/18/push-to-try/ Jorendorff's blog] for more details.
This will schedule the task with your specified parameters. The new task will be labeled as <tt>custom-<original_task_name></tt>.


== Using a custom mozconfig  ==
Some of the parameters that you can change are (see screenshot above):
 
* Environment variables (e.g. <tt>MOZ_LOG</tt>)
* Modify Python's logging level
* Path of the test to execute
* Gecko preferences (think of <tt>about:config</tt>)
* Run a test repeatedly
* Run until the test fails
 
Common problems:
* By default the entire task is re-run, running all the tests in the original task. If repeat/runUntilFailure is specified, it may take a long time to repeat all the tests, and the log file may be huge. Consider using the path: parameter to limit the run to the test(s) of interest.
 
== Getting debug symbols ==
 
NOTE: You might be able to accomplish this by re-running a build task from Treeherder by adding the env variable to the new task.
 
By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add <tt>MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1</tt> to the in-tree mozconfigs. You can do this for all platforms by importing [http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols this patch] into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.
 
== Adding new jobs ==
It's possible to create new jobs (or modify existing ones) directly *in* your try push, provided you use taskcluster. Just edit the relevant configuration in testing/taskcluster. For more information on creating jobs, see the [http://docs.taskcluster.net/ taskcluster docs].
 
== Desktop l10n jobs ==
You can use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.


The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.
The jobs can be customized by modifying files prior to pushing:
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW). Leaving a full list of locales is likely to hit a timeout on Mac and Windows
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario


=== Old crufty way ===
The resulting builds are uploaded to the same sub-directory as en-US builds, eg try-linux/ for 32bit linux.


For revisions prior to around Oct 4, 2011, you'll be using the default buildbot-configs based mozconfigs, and will need to add some extra files to your push.
== Desktop l10n jobs (on Taskcluster) ==


If you want to use setting other than those in the default mozconfigs, you can push an '''extra file''' to the $topsrcdir:
To get Desktop l10n jobs on taskcluster on try, simply pass `-b o -p linux64-l10n,linux-l10n` with your try push. You'll also get these jobs automatically with `-p linux64,linux` if you are touching one of the files known to be heavily involved in l10n jobs. (Taskcluster support for other platforms is not yet available)


*'''mozconfig-extra''' with settings to be applied to all mozconfigs
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.
*'''mozconfig-extra-$platform''' to apply changes only to that platform's mozconfig, where $platform is one of linux, linux64, win32, macosx, macosx64, linux-android, maemo5-gtk, maemo5-qt


The options you enable/disable in your custom mozconfig are '''appended''' to the existing config.  
The jobs can be customized by modifying files prior to pushing:
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW).
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario


The default mozconfigs used for tryserver builds are available in Hg: [http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla2/ http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla2/]$platform/try ([http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla2/linux/try linux example])
The resulting builds are uploaded as a task artifact, and are not yet signed.


== Using older GCC ==
== Android Single-Locale l10n jobs (on Taskcluster) ==
Linux and Linux64 build bots are using GCC 4.5 for Try builds. But some branches will fail to build or run tests with this version of GCC because of the libstdc++ dependency it adds. Such branches are those where [https://bugzilla.mozilla.org/show_bug.cgi?id=643690 bug 643690] hasn't landed. If you want to push such a branch to Try, you need to add the following to '''mozconfig-extra-linux''' and/or '''mozconfig-extra-linux64''' in the top directory:


CC=/tools/gcc-4.3.3/installed/bin/gcc
(NOTE: Only supported on Gecko 51a1 and above)
CXX=/tools/gcc-4.3.3/installed/bin/g++


== How to get non-PGO coverage ==
To get Android l10n jobs on taskcluster on try, simply pass `-p android-api-15-l10n` with your try push. You'll also get these jobs automatically with `-p android-api-15` if you are touching one of the files known to be heavily involved in l10n jobs.
On branches where [https://bugzilla.mozilla.org/show_bug.cgi?id=643704 bug 643704] has landed, adding the following in a '''mozconfig-extra''' or '''mozconfig-extra-$platform''' file will suffice:
mk_add_options MOZ_PGO=


On other branches, fool the system!
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.


Include this in your patch.
The jobs can be customized by modifying files prior to pushing:
* reducing the number of locales by limiting <tt>mobile/android/locales/all-locales</tt> (eg top-locales like de fr ru zh-TW).
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try_android-api-15.py#3 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario


diff --git a/client.mk b/client.mk
The resulting builds are uploaded as a task artifact, and are not yet signed.
--- a/client.mk
+++ b/client.mk
@@ -210,5 +210,5 @@ else
  endif
-profiledbuild::
+Xprofiledbuild::
        $(MAKE) -f $(TOPSRCDIR)/client.mk build MOZ_PROFILE_GENERATE=1
        $(MAKE) -C $(PGO_OBJDIR) package
@@ -352,5 +352,5 @@ endif
  # Build it
-build::  $(OBJDIR)/Makefile $(OBJDIR)/config.status
+profiledbuild::  $(OBJDIR)/Makefile $(OBJDIR)/config.status
        $(MOZ_MAKE)


== Server Status ==
== Server Status ==
 
* Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending
* Try server load can be seen at http://build.mozilla.org/builds/pending/try.html
* In-progress builds by revision are are https://secure.pub.build.mozilla.org/buildapi/running
* Pending builds by revision are at http://build.mozilla.org/builds/pending.html
* In-progress builds by revision are are http://build.mozilla.org/builds/running.html


== Other Notes ==
== Other Notes ==
* Finished builds will live on http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/{your_email_changeset} for 4 days
* Finished builds will live in  http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/<your_ldap_email>-<revision> for 14 days before deletion
** They are moved to http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/old/{your_email_changeset} for an additional '''14 days''' before removal
* Treeherder data is purged after 4 months.
* Try repo is reset '''occasionally''', and you may not be able to use TBPL to see your test results after it has been reset.
* If you have any problems please [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Release%20Engineering&status_whiteboard=tryserver file a bug]
* If you have any problems please [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Release%20Engineering&status_whiteboard=tryserver file a bug]
* Windows builds have symbols uploaded to http://build.mozilla.org/tryserver-symbols. Windbg and the Visual Studio debugger may use them to help debug crashing try server builds. Instructions for setting this up can be found here: http://developer.mozilla.org/en/docs/Using_the_Mozilla_symbol_server. Make sure you use the aforementioned URL instead of http://symbols.mozilla.org/firefox.


* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}
* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}


== Other Mozilla Try Servers ==
== Other Mozilla Try Servers ==
* [https://wiki.mozilla.org/Thunderbird/Infrastructure/TryServer Thunderbird Try Server] for the comm-central repository
* [[ReleaseEngineering/ThunderbirdTryServer|Thunderbird Try Server]] for the comm-central repository


== Problem Diagnosis ==
== Problem Diagnosis ==
=== Can not access try server ===
Test your account & configuration
Test your account & configuration
* <code>ssh hg.mozilla.org</code>, response: "No Interactive shells allowed here!"
* <code>ssh hg.mozilla.org</code>, response: "No Interactive shells allowed here!"
* <code>ssh hg.mozilla.org clone invalid_sandbox</code>, response: menu display and interactive prompting.
* <code>ssh hg.mozilla.org clone invalid_sandbox</code>, response: menu display and interactive prompting.


<div id="long_try_push"></div>
=== Pushes to try take a very long time ===
Note: if a fellow developer cancelled their try push, they have saddled you with the cost of rebuilding the cache. (See [http://dtor.com/halfire/2014/07/02/2014_06_try_server_update.html#caching caching] details.)
If you're experiencing excessive wait times (> 45min) pushing to try, please file a bug asking IT to reset the try repository using [https://bugzilla.mozilla.org/enter_bug.cgi?comment=please%20schedule%20a%20reset%20of%20the%20try%20repository%20ASAP%20with%20sheriffs%20%26%20releng.&component=WebOps%3A%20Source%20Control&op_sys=All&product=Infrastructure%20%26%20Operations&rep_platform=All&short_desc=Push%20to%20try%20taking%20XXX%20minutes this template] (please include specifics of your experience). They will coordinate with sheriffs and release engineering as needed.
=== Waiting for Lock ===
If you get a message similar to:
    remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:23974'
    remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:30549
It means several developers are trying to push to try at the same time. In the case above, nothing appears to be wrong, as the PID changes between the messages.
=== Waiting for Lock multiple times with the same pid ===
Similar to the above case, but with the same pid when you retry over and over again.
Please retry your push. If you see messages indicating the same process has been pushing for more than 15 minutes, treat as [[#long_try_push|above]].
== CiDuty issues ==
=== How do I trigger additional talos/test runs for a given try build? ===
If your syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button. For test suites you didn't request originally you can use the "Add New Jobs" interface in Treeherder.
=== How do I cancel existing jobs? ===
For individual jobs, select the relevant one on Treeherder and use the cancel button. To cancel all jobs, use the menu arrow shown on the header row for each push, and then the "Cancel all" option.


== See Also ==
== See Also ==
* http://developer.mozilla.org/en/Creating_Mercurial_User_Repositories
* https://developer.mozilla.org/en/Creating_Mercurial_User_Repositories
* http://people.mozilla.org/~lsblakk/trychooser
* https://wiki.mozilla.org/Sheriffing/How:To:Recommended_Try_Practices
* https://wiki.mozilla.org/Build:TryChooser
* https://wiki.mozilla.org/ReleaseEngineering:Autoland
* [http://build.mozilla.org/buildapi/self-serve Manage Submissions] [[https://build.mozilla.org/buildapi/self-serve/try try]]
* [https://catlee.github.io/highscores/highscores.html Scoreboard]
Gbrown
Confirmed users
1,759

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/395101...1224683"