ReleaseEngineering/TryServer: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎HOWTO: Building a non-unified build: The --disable-unified-compilation flag is no longer supported.)
(Update page for modern workflows and best practices; Remove some obsolete and/or irrelevant documentation)
Line 3: Line 3:


== Getting access to the Try Server  ==
== Getting access to the Try Server  ==
To use the try server, you'll need [http://www.mozilla.org/hacking/commit-access-policy/ Level 1 Commit Access].  
To use the try server, you'll need [http://www.mozilla.org/hacking/commit-access-policy/ Level 1 Commit Access]. You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: [http://www.mozilla.org/hacking/committer/ Becoming a Mozilla Contributor]


You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: [http://www.mozilla.org/hacking/committer/ Becoming a Mozilla Contributor.]
After you have level 1 commit access, you'll need to do a bit of setup before you'll be able to push. See [https://developer.mozilla.org/en-US/docs/Mercurial/Using_Mercurial#How_do_I_check_stuff_in.3F this guide] for instructions. Assuming you only have level 1 access, you won't be able to push to mozilla-central, but you can replace 'hg.mozilla.org/mozilla-central' with 'hg.mozilla.org/try'.


== How to push to try ==
== Configuration ==
=== Running a subset of builds/test/talos available to Try===
Before using try, there is some recommended configuration you should set up. This can be accomplished by running:
You must use [[Build:TryChooser]] to choose which builds, tests, and talos you would like run on your push to try.  Make sure you place the try chooser text in your ''topmost commit''.  The [http://trychooser.pub.build.mozilla.org/ TryChooser] web page can help you build a commit message for custom requests, so can the [https://bitbucket.org/sfink/trychooser mercurial extension]


<pre>
$ ./mach mercurial-setup
% hg qref --message "try: <your-computed-syntax-here>"
% hg push -f try
</pre>


Where the computed syntax will be something like <code>-b d -p linux64 -u all -t none</code>. Please note that running all tests on all platforms is very expensive (at least 300 compute-hours), and probably overkill.
Be sure to at least enable the [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html firefoxtree extension] and the push-to-try extension. The push-to-try extension is required if you wish to use the |mach try| command (see below). Firefoxtree will give you a handy 'try' alias you can use for pushing, and prevent you from accidentally pushing multiple heads. If for some reason you prefer not to use firefoxtree, you can set the same alias up manually in your hgrc:


=== Extending jobs on a push ===
[paths]
We will soon be releasing a version of Treeherder which allows you to add any jobs through the UI.
try = ssh://hg.mozilla.org/try


To be announced.
Note: Never pull from try. You'll end up with hundreds of heads containing everyone's half baked and broken pushes.


=== Pushing to try ===
For more information, see [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html mercurial for mozillians].


To submit your changes to the try server (assuming they're modifications to mozilla-central or a similar branch, e.g. tracemonkey), you have a few options:
== How to push to try ==
* <code><nowiki>hg push -f ssh://hg.mozilla.org/try/</nowiki></code> or
There are two ways to schedule jobs on try. You can either push like normal, and select which jobs you want using [https://wiki.mozilla.org/EngineeringProductivity/Projects/Treeherder treeherder], or you can specify a try syntax in your commit message to schedule jobs automatically.  
* <code><nowiki>hg push -f ssh://<username@host>@hg.mozilla.org/try/</nowiki></code> <strike>?</strike> or
* <code><nowiki>hg push -f ssh://hg.mozilla.org/try/ -e 'ssh -l <username@host></nowiki></code>


==== Creating an alias ====
=== Scheduling jobs with Treeherder ===
To save yourself some typing, you can add an alias to your hgrc:
Note: This method does not currently support taskcluster jobs. See {{bug|1246167}}.
<pre>[paths]
try = ssh://hg.mozilla.org/try
</pre>
and then push with
<pre>hg push -f try</pre>


==== ~/.ssh/config ====
With this method, just push to try like you would to inbound (don't create any try syntax):
ssh host settings to make life easier


<pre>
$ hg push -f -r . try
Host hg.mozilla.org
  User <commit_user_email_address>
  IdentityFile ~/.ssh/private_key_file
</pre>


=== Viewing the results ===
You should see a warning about no jobs being scheduled automatically, followed by a link to your push in treeherder. Open the link, select the drop-down arrow at the top right, and choose "Add new Jobs":


You can see the results of your tryserver build in a number of ways:
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042414/TH_dropdown.png


* You'll get an email on a successful push with a link to Treeherder for your revision as well as optional emails on any non-successful build/test/talos results (this setting can be adjusted using [[Build:TryChooser]] args for email notification)
Click the job symbols you wish to schedule. If you select a test job, the required build will automatically be scheduled:
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax (see: [[Build:TryChooser]] for examples)
* Look for your changeset on [https://treeherder.mozilla.org/#/jobs?repo=try Treeherder]. You can add '''&author=YOUR.EMAIL''' to only see your pushes.
* [Obsolete] Compare Talos perf numbers using Pike's [http://github.com/Pike/talos-node talos-node] or mconnor's [http://perf.snarkfest.net/compare-talos/ compare-talos].
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].


If you're using [https://developer.mozilla.org/en/Mercurial_Queues Mercurial queues], the <code>push -f</code> command pushes any patches that are currently applied, and the Try server will build the result. (This is an awesome feature, not a bug!)
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042426/TH_with_jobs.png


You don’t need to clone or pull from the <code>try</code> repo, and you probably don’t want to. You’d get every half-baked changeset anybody ever tested.
Finally, click "Trigger New Jobs" near the top right of your push.


See [http://blog.mozilla.com/jorendorff/2008/08/18/push-to-try/ Jorendorff's blog] for more details.
=== Scheduling jobs with Try Syntax ===
If you know exactly what you want to run, you can use [[Build:TryChooser]] to select which jobs run directly in your commit message. Make sure the commit message containing try syntax is placed in your ''topmost commit''.  The [http://trychooser.pub.build.mozilla.org/ TryChooser] web page can help you build a commit message for custom requests. It will generate both a syntax string as well as a mach command you can simply paste into your terminal.


== Using a custom mozconfig  ==
==== Using mach ====
The recommended way to push with a try syntax, is to use |mach try|. It works with both mercurial and git, and you can use the [http://trychooser.pub.build.mozilla.org/ TryChooser] web page to generate the command. For example:


The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.
$ mach try -b o -p linux -u mochitest-1 -t none


If you want to apply the same mozconfig changes to multiple platforms, you can edit <tt>build/mozconfig.common.override</tt> instead.  This file is included at the end of each of the in-tree mozconfig files.
The |mach try| command also has some advanced features. It can actually change how the job gets run within automation. For example, to run only mochitests under 'dom/indexedDB':


Android mozconfigs are in mobile/android/config/mozconfigs.
$ mach try -b o -p linux -u mochitests --and dom/indexedDB


Note:
For more information see:
* TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.
** Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.


== Using a custom Gaia ==
$ mach try --help


For Gaia-related tests on B2G desktop builds, you can use an arbitrary branch on any github Gaia fork in your try job. To do this, you need to modify $src/b2g/config/gaia.json to point to your repo.  Normally, gaia.json looks something like this:
Note: This method doesn't work well with mq.


{
==== Using the Trychooser Extension ====
    "revision": "6fbf376b98130fca7198d7d2e5e588cb37dd07c4",
You can also use the [https://bitbucket.org/sfink/trychooser mercurial trychooser extension] from sfink. It has some neat features, like a curses ui that lets you graphically choose a try syntax. Though {{bug|1197868}} tracks implementing this feature in |mach try|. Eventually trychooser will be deprecated in favor of |mach try|.
    "repo_path": "/integration/gaia-central"
}


You should edit this file to include some additional fields:
==== Using mq ====
The aforementioned tools may not play all that nicely with [https://www.mercurial-scm.org/wiki/MqExtension mercurial queues]. If you use mq, you can still push to try manually. First build your try syntax with the [http://trychooser.pub.build.mozilla.org/ TryChooser] web page. Then run:


  {
  $ hg qref --message "try: <your-computed-syntax-here>"
    "revision": "6fbf376b98130fca7198d7d2e5e588cb37dd07c4",
  $ hg push -f try
    "repo_path": "/integration/gaia-central",
    "git": {
        "remote": "",
        "branch": "",
        "git_revision": ""
    }
  }


You must specify remote, and can specify either branch or revision.
== Viewing the results ==
{
You can see the results of your tryserver build in a number of ways:
    "revision": "6fbf376b98130fca7198d7d2e5e588cb37dd07c4",
    "repo_path": "/integration/gaia-central",
    "git": {
        "remote": "https://github.com/jonallengriffin/gaia.git",
        "branch": "gaia_unit_crash",
        "git_revision": "gaia_unit_crash"
    }
}


When the 'git' dict in gaia.json is populated in this way, the hg attributes for revision and repo_path will be ignored, and the Gaia will be cloned from the given remote and used to run the tests.
* You'll get an email on a successful push with a link to treeherder for your revision as well as optional emails on any non-successful build/test/talos results (this setting can be adjusted using [[Build:TryChooser]] args for email notification)
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax (see: [[Build:TryChooser]] for examples)
* The link to treeherder will be printed on the command line.
* Look for your changeset on [https://treeherder.mozilla.org/#/jobs?repo=try Treeherder]. You can add '''&author=YOUR.EMAIL''' to only see your pushes.
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].


== Using a custom gonk-misc ==


You can also specify a custom gonk-misc repository for tests. To do this, you need to edit the file $src/b2g/config/<target>/sources.xml and then add your remote, for example:
== Using a custom mozconfig  ==
The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.


<remote fetch="https://github.com/walac" name="walac"/>
If you want to apply the same mozconfig changes to multiple platforms, you can edit <tt>build/mozconfig.common.override</tt> instead.  This file is included at the end of each of the in-tree mozconfig files.


After that, you have to modify the gonk-misc project line to point to your repository and the desired revision:
Android mozconfigs are in mobile/android/config/mozconfigs.


<project name="gonk-misc" path="gonk-misc" remote="walac" revision="b8fd88760316cbcef261830f4031992fa6c379c6"/>
Note:
 
* TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.
You then need to commit these changes and push to the try server.
** Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.


== Getting debug symbols ==
== Getting debug symbols ==
By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add <tt>MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1</tt> to the in-tree mozconfigs. You can do this for all platforms by importing [http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols this patch] into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.
By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add <tt>MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1</tt> to the in-tree mozconfigs. You can do this for all platforms by importing [http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols this patch] into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.


=== Debugging with debug symbols ===
== Adding new jobs ==
 
It's possible to create new jobs (or modify existing ones) directly *in* your try push, provided you use taskcluster. Just edit the relevant configuration in testing/taskcluster. For more information on creating jobs, see the [http://docs.taskcluster.net/ taskcluster docs].
* Windows
** Unzip the crashreporter-symbols-full.zip package somewhere, let's call it c:\try_symbols
** Set your symbol path to <tt>cache*c:\symcache;SRV*http://msdl.microsoft.com/download/symbols;SRV*c:\try_symbols</tt> where <tt>c:\symcache</tt> is a writable directory where the debugger can store symbols.
*** If using WinDBG, you can type <tt>.sympath ...</tt> with the above path in the command window.
*** If using Visual Studio, you can set the <tt>_NT_SYMBOL_PATH</tt> environment variable to the above. (Note that it looks like adding the path to the unzipped symbols in the Visual Studio settings doesn't work, you really need to add it to the <tt>_NT_SYMBOL_PATH</tt> environment variable. Also, setting a <tt>_NT_SYMCACHE_PATH</tt> environment variable instead of adding <tt>cache*...;</tt> to <tt>_NT_SYMBOL_PATH</tt> doesn't let Visual Studio find the symbols.)
*** If the above approaches don't work for you, an alternative manual approach is described [https://bugzilla.mozilla.org/show_bug.cgi?id=936784#c69 here].
* Linux
** TODO
* Mac
** TODO


== Server Status ==
== Server Status ==
* Try server load can be seen at https://secure.pub.build.mozilla.org/builddata/reports/pending/try.html
* Try server load can be seen at https://secure.pub.build.mozilla.org/builddata/reports/pending/try.html
* Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending
* Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending
Line 150: Line 106:


* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}
* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}
=== HOWTO: Preserve a commit message between try pushes ===
Simply create a new patch in your queue for the try attempt commit message.
Future pushes can simply push/pop the try patch onto the applied queue when needed.
<pre>
% hg qref --message "bug xxx - a spell/patch of improvements"
% hg qnew patch.try
% hg qref --message "try: -b o -e -p all -u all -t none"
% hg push -f try
% hg qpop patch.try
</pre>
If you know that your patches commit message is correct you can simply use:
<pre>
% hg qref
% hg qnew patch.try
% hg qref --message "try: -b o -e -p all -u all -t none"
% hg push -f try
% hg qpop patch.try
</pre>


== Other Mozilla Try Servers ==
== Other Mozilla Try Servers ==
* [[ReleaseEngineering/ThunderbirdTryServer|Thunderbird Try Server]] for the comm-central repository
* [[ReleaseEngineering/ThunderbirdTryServer|Thunderbird Try Server]] for the comm-central repository
* You can push b2g18 to the regular Firefox tryserver, but you will get a lot of oranges.  There is currently no work-around.  It's possible to eke out some useful information from such a try push -- ask a sheriff, who should be familiar with the expected failures, or otherwise push your qtip rev to try and compare the results to the patched version.  Good luck.


== Problem Diagnosis ==
== Problem Diagnosis ==
Line 203: Line 135:
== Buildduty issues ==  
== Buildduty issues ==  
=== How do I trigger additional talos/test runs for a given try build? ===
=== How do I trigger additional talos/test runs for a given try build? ===
If your trychooser syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button.
If your trychooser syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button. For test suites you didn't request originally you can use the "Add New Jobs" interface in Treeherder.
 
For test suites you didn't request originally you can ask Release Engineering to [[ReleaseEngineering/How_To/Trigger_Talos_Jobs |do a sendchange']], look for someone who is <person>|buildduty in #releng on IRC. Alternatively you can push again with a different try syntax.


=== How do I cancel existing jobs? ===
=== How do I cancel existing jobs? ===

Revision as of 17:13, 3 March 2016

Try Server

The try server is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.

Getting access to the Try Server

To use the try server, you'll need Level 1 Commit Access. You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: Becoming a Mozilla Contributor

After you have level 1 commit access, you'll need to do a bit of setup before you'll be able to push. See this guide for instructions. Assuming you only have level 1 access, you won't be able to push to mozilla-central, but you can replace 'hg.mozilla.org/mozilla-central' with 'hg.mozilla.org/try'.

Configuration

Before using try, there is some recommended configuration you should set up. This can be accomplished by running: 

$ ./mach mercurial-setup

Be sure to at least enable the firefoxtree extension and the push-to-try extension. The push-to-try extension is required if you wish to use the |mach try| command (see below). Firefoxtree will give you a handy 'try' alias you can use for pushing, and prevent you from accidentally pushing multiple heads. If for some reason you prefer not to use firefoxtree, you can set the same alias up manually in your hgrc: 

[paths]
try = ssh://hg.mozilla.org/try

Note: Never pull from try. You'll end up with hundreds of heads containing everyone's half baked and broken pushes.

For more information, see mercurial for mozillians.

How to push to try

There are two ways to schedule jobs on try. You can either push like normal, and select which jobs you want using treeherder, or you can specify a try syntax in your commit message to schedule jobs automatically.

Scheduling jobs with Treeherder

Note: This method does not currently support taskcluster jobs. See bug 1246167.

With this method, just push to try like you would to inbound (don't create any try syntax): 

$ hg push -f -r . try

You should see a warning about no jobs being scheduled automatically, followed by a link to your push in treeherder. Open the link, select the drop-down arrow at the top right, and choose "Add new Jobs":

TH_dropdown.png

Click the job symbols you wish to schedule. If you select a test job, the required build will automatically be scheduled:

TH_with_jobs.png

Finally, click "Trigger New Jobs" near the top right of your push.

Scheduling jobs with Try Syntax

If you know exactly what you want to run, you can use Build:TryChooser to select which jobs run directly in your commit message. Make sure the commit message containing try syntax is placed in your topmost commit. The TryChooser web page can help you build a commit message for custom requests. It will generate both a syntax string as well as a mach command you can simply paste into your terminal.

Using mach

The recommended way to push with a try syntax, is to use |mach try|. It works with both mercurial and git, and you can use the TryChooser web page to generate the command. For example: 

$ mach try -b o -p linux -u mochitest-1 -t none

The |mach try| command also has some advanced features. It can actually change how the job gets run within automation. For example, to run only mochitests under 'dom/indexedDB': 

$ mach try -b o -p linux -u mochitests --and dom/indexedDB

For more information see: 

$ mach try --help

Note: This method doesn't work well with mq.

Using the Trychooser Extension

You can also use the mercurial trychooser extension from sfink. It has some neat features, like a curses ui that lets you graphically choose a try syntax. Though bug 1197868 tracks implementing this feature in |mach try|. Eventually trychooser will be deprecated in favor of |mach try|.

Using mq

The aforementioned tools may not play all that nicely with mercurial queues. If you use mq, you can still push to try manually. First build your try syntax with the TryChooser web page. Then run: 

$ hg qref --message "try: <your-computed-syntax-here>"
$ hg push -f try

Viewing the results

You can see the results of your tryserver build in a number of ways:

  • You'll get an email on a successful push with a link to treeherder for your revision as well as optional emails on any non-successful build/test/talos results (this setting can be adjusted using Build:TryChooser args for email notification)
  • You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax (see: Build:TryChooser for examples)
  • The link to treeherder will be printed on the command line.
  • Look for your changeset on Treeherder. You can add &author=YOUR.EMAIL to only see your pushes.
  • Download your completed builds from firefox/tryserver-builds on ftp.m.o.


Using a custom mozconfig

The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.

If you want to apply the same mozconfig changes to multiple platforms, you can edit build/mozconfig.common.override instead. This file is included at the end of each of the in-tree mozconfig files.

Android mozconfigs are in mobile/android/config/mozconfigs.

Note:

  • TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.
    • Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.

Getting debug symbols

By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1 to the in-tree mozconfigs. You can do this for all platforms by importing this patch into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.

Adding new jobs

It's possible to create new jobs (or modify existing ones) directly *in* your try push, provided you use taskcluster. Just edit the relevant configuration in testing/taskcluster. For more information on creating jobs, see the taskcluster docs.

Server Status

Other Notes

Other Mozilla Try Servers

Problem Diagnosis

Can not access try server

Test your account & configuration

  • ssh hg.mozilla.org, response: "No Interactive shells allowed here!"
  • ssh hg.mozilla.org clone invalid_sandbox, response: menu display and interactive prompting.

Pushes to try take a very long time

Note: if a fellow developer cancelled their try push, they have saddled you with the cost of rebuilding the cache. (See caching details.)

If you're experiencing excessive wait times (> 45min) pushing to try, please file a bug asking IT to reset the try repository using this template (please include specifics of your experience). They will coordinate with sheriffs and release engineering as needed.

Waiting for Lock

If you get a message similar to: 

   remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:23974'
   remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:30549

It means several developers are trying to push to try at the same time. In the case above, nothing appears to be wrong, as the PID changes between the messages.

Waiting for Lock multiple times with the same pid

Similar to the above case, but with the same pid when you retry over and over again.

Please retry your push. If you see messages indicating the same process has been pushing for more than 15 minutes, treat as above.

Buildduty issues

How do I trigger additional talos/test runs for a given try build?

If your trychooser syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button. For test suites you didn't request originally you can use the "Add New Jobs" interface in Treeherder.

How do I cancel existing jobs?

For individual jobs, select the relevant one on Treeherder and use the cancel button. To cancel all jobs, use the menu arrow shown on the header row for each push, and then the "Cancel all" option.

TryChooser

See the TryChooser wiki page.

See Also

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/TryServer&oldid=1119861"