MozillaWiki

ReleaseEngineering/How To/VCSSync: Difference between revisions

Page Discussion

ReleaseEngineering/How To/VCSSync (view source)

Revision as of 02:12, 30 June 2018

12,632 bytes added ,  30 June 2018
new mapper url
(new mapper url)
 
(38 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{Release Engineering How To|Deal with VCS Sync}}
{{Release Engineering How To|VCS Sync}}
{{Release Engineering Deployment|VCS Sync}}
This is documentation on the mozharness-based vcs sync processes.
This is documentation on the mozharness-based vcs sync processes.


<b>See [http://people.mozilla.org/~hwine/tmp/vcs2vcs/ Hal's docs] for the legacy process.</b>
<div id="legacy"></div>
<b>See [http://people.mozilla.org/~hwine/tmp/vcs2vcs/ legacy docs] for the legacy (aka "old") process.</b>


= Quick notes =
= Quick notes =
"VCS-sync" is a system which has evolved from it's original purpose to being the tool used whenever "this repository needs to be publicly visible 'over here'", perhaps with modifications, and a few exceptions on "publicly visible". The current (2016 Q1) business needs it addresses are:
# Support for gecko developers who prefer to work with git:
#* All gecko primary development and release branches are merged into a single git repository (with CVS history) in "gecko-dev.git".
#* All gecko twig and misc branches are merged into a single git repository (with CVS history) in "gecko-projects.git".
# Support for Mozilla development tools and processes which (for operational needs (e.g. travis-ci), or developer preference, should appear in both Mercurial and Git repositories.
#* These are "mirrors" from one system to the other.
#* One repo is always designated the RoR (Repository of Record), and is the only one where commits will "stick".
#* Refer to the system configurations for each mirror to determine the RoR. (It can, and has, changed over time for some repositories.)
# Support for Partner facing FxOS development processes.
#* The URL's for these repositories can not change, as they are referenced in manifest files which comprise the delivery to Partners.
#* These repositories on git.mozilla.org satisfy various contractual needs with FxOS partners.
#* These repositories must not contain either object deletions, or non-fast-forward commits.
#* These repositories only contain content that was "delivered" to Partners, with branches named according to FxOS conventions.
#* These repositories include any "Mozilla Contributions" to FxOS, which includes code beyond gecko.git and gaia.git.
# Support for internal FxOS development processes.
#* Many Android repositories have been mirrored to git.mozilla.org for the purpose of having a known-good-repository for the build systems to clone from.
#* Some tests still require translation from Git &rarr; Mercurial format. Each such gaia.git branch is mapped to a distinct Mercurial repository.
== What's running where ==
Anything not listed below is probably running in the new system. The system pushing the production content is listed first. If the other system is listed, we're in parallel operations mode and changes should be made to both systems.
=== Repositories needed to Build Mozilla Products ===
{| class="fullwidth-table sortable"
| style="background:#cccccc" | '''Repository'''
| style="background:#cccccc" | '''Old/New'''
| style="background:#cccccc" | '''Web views at'''
| style="background:#cccccc" | '''Notes'''
|-
| gecko-dev.git
| new
| [https://github.com/mozilla/gecko-dev.git github], [http://git.mozilla.org/?p=integration/gecko-dev.git;a=summary Moz]
|
|-
| gecko-projects.git
| new
| [https://github.com/mozilla/gecko-projects.git github], [http://git.mozilla.org/?p=integration/gecko-projects.git;a=summary Moz]
|
|-
| releases/gecko.git
| old, new
| [http://git.mozilla.org/?p=releases/gecko.git;a=summary Moz]
| Partner facing FxOS <nowiki>[1]</nowiki>
|-
| releases/gaia.git
| old, new
| [http://git.mozilla.org/?p=releases/gaia.git;a=summary Moz]
| Partner facing FxOS <nowiki>[1]</nowiki>
|-
| releases/l10n/<locale>/{gecko,gaia}.git
| old, new
| [http://git.mozilla.org/ Moz] (browse from there)
| Partner facing FxOS <nowiki>[1]</nowiki>
|-
| integration/gaia*
| old
| [http://hg.mozilla.org/integration/ Moz] (browse from there)
| Hg versions of gaia branches to support older tooling
|-
| b2g mirrors
| old
| [http://git.mozilla.org/ Moz] browse <tt>external</tt> and <tt>b2g</tt> from there (r/o)
| Local copies to ensure availability during builds.
|}
Notes:
# Partner facing repositories are configured to:
#* not accept non fast forward commits
#* not accept deletes
#* only contain branches related to shipping versions of FxOS
=== Repositories Supporting Mozilla Web Products ===
{| class="fullwidth-table sortable"
| style="background:#cccccc" | '''Repository'''
| style="background:#cccccc" | '''Old/New'''
| style="background:#cccccc" | '''Web views at'''
| style="background:#cccccc" | '''Notes'''
|-
| bugzilla-{bugzilla,qa}.git
| old
| [http://git.mozilla.org/ Moz] (r/w), [https://github.com/bugzilla/ github] (r/o)
| Only the bugzilla.git & qa.git repos are mirrored to github
|-
| webtools-bmo-{bugzilla,qa}.git
| old
| [http://git.mozilla.org/ Moz] (r/w), [https://github.com/mozilla/ webtools-bmo-* on github] (r/o)
|
|}
=== Repositories Supporting Infrastructure, Tooling, and Miscellaneous ===
See also [[ReleaseEngineering/Repositories|full list of RelEng repositories]].
{| class="fullwidth-table sortable"
| style="background:#cccccc" | '''Repository'''
| style="background:#cccccc" | '''Old/New'''
| style="background:#cccccc" | '''Web views at'''
| style="background:#cccccc" | '''Notes'''
|-
| build-releng-api.git
| old
| [http://github.com/mozilla/build-releng-api.git github] (r/o), [http://hg.mozilla.org/build/releng-api Moz] (r/o)
|-
| automation/fuzz-tools.git
| old
| [http://git.mozilla.org/?p=automation/fuzz-tools.git Moz] (r/w), [https://github.com/mozilla/automation-fuzz-tools.git github] (r/o)
|
|-
| qa/mozmill-tests
| old
| [http://hg.mozilla.org/qa/mozmill-tests Moz] (r/w), [https://github.com/mozilla/qa-mozmill-tests github] (r/o)
|
|}
== Machines ==
== Machines ==
2013.10.14: We're currently using the following machines:
2013.10.14: We're currently using the following machines:
Line 12: Line 130:
| style="background:#cccccc" | '''Process'''
| style="background:#cccccc" | '''Process'''
| style="background:#cccccc" | '''Location'''
| style="background:#cccccc" | '''Location'''
| style="background:#cccccc" | '''User'''
|-
|-
| vcssync1.srv.releng.usw2.mozilla.com
| vcssync1.srv.releng.usw2.mozilla.com
| beagle (gecko-dev)
| beagle (gecko-dev)
| /opt/vcs2vcs
| /opt/vcs2vcs
| vcs2vcs
|-
|-
| vcssync2.srv.releng.usw2.mozilla.com
| vcssync2.srv.releng.usw2.mozilla.com
| project-branches (gecko-projects)
| project-branches (gecko-projects)
| /opt/vcs2vcs
| /opt/vcs2vcs
| vcs2vcs
|-
|-
| github-sync2.dmz.scl3.mozilla.com
| github-sync2.dmz.scl3.mozilla.com
| staging (initial conversion, gecko-git, l10n)
| build repos (autoland, buildapi, buildbot-configs, buildbotcustom, cloud-tools, mozharness, opsi-package-sources, partner-repacks, preproduction, puppet, puppet-manifests, rpm-sources, talos, tools)
| /home/asasaki/{initial2,gecko,l10n}
| /opt/vcs2vcs
| vcs2vcs
|-
|-
| github-sync4.dmz.scl3.mozilla.com
| github-sync4.dmz.scl3.mozilla.com
| staging (project branches)
| gecko l10n and gaia l10n
| /home/asasaki/projects
| /opt/vcs2vcs
| vcs2vcs
|}
|}


Plus a [http://people.mozilla.org/~hwine/tmp/vcs2vcs/trouble_shooting.html#accessing-the-machines few other github-sync machines] for Hal's legacy processes.
Plus a [http://people.mozilla.org/~hwine/tmp/vcs2vcs/trouble_shooting.html#accessing-the-machines few other github-sync machines] for legacy processes.


== Branch ==
== Branch ==
2013.10.14: We're currently based off Aki's [https://github.com/escapewindow/mozharness/tree/vcs_multi_repo vcs_multi_repo] branch off github.  Once all of {{bug|847727}} lands, we'll go to the [http://hg.mozilla.org/build/mozharness mozharness production branch].
2013.10.14: We're based off the [http://hg.mozilla.org/build/mozharness mozharness production branch].


= Specific instructions =
= Specific instructions =
Line 39: Line 162:
== Emails ==
== Emails ==


== Maintenance ==
=== Successful noop emails ===
 
<pre>
From: vcs2vcs@vcssync2.srv.releng.usw2.mozilla.com
Subject: [vcs2vcs] Successful conversion for project-branches (12s) <EOM>
</pre>
 
These are pretty much there to tell you that the job's running, everything's fine, and the noop time was (in this case) 12 seconds.  The <b><EOM></b> tells you that the body of the email is empty (end of message).
 
The noop time varies depending on hg.m.o load, as well as log upload times.
 
See [https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo#How_to_adjust_email_notifications here] for enabling/disabling these... they can be pretty spammy.
 
=== Successful emails with messages ===
 
<pre>
From: vcs2vcs@vcssync1.srv.releng.usw2.mozilla.com
Subject: [vcs2vcs] Successful conversion for beagle (72s)
 
 
Summary is non-zero:
 
info - Successfully pushed mozilla-aurora.
error - Error getting changes for mozilla-inbound; skipping!
 
 
02:16:08    ERROR -  abort: HTTP Error 500: Internal Server Error
02:16:08    ERROR -  Automation Error: hg not responding
02:16:08    ERROR - Return code: 65280
02:16:08    ERROR - Error getting changes for mozilla-inbound; skipping!
</pre>
 
Whenever the summary of a job is non-zero, or the error log is non-zero, we'll get a message body in the success email.  This is usually a successful push notification, but can include the contents of the error log or any other information we put in the summary.  (At some future point we may add l10n repo creation on git.m.o in here).
 
In this case, the ISE from hg.m.o is an intermittent issue.  In general, the script should auto-fix by clobbering+recloning, and the conversion will happen later, though it'll be delayed by this action.
 
=== Failure emails ===
 
<pre>
From: vcs2vcs@vcssync-dev.build.mozilla.com
Subject: [vcs2vcs] Failed conversion for beagle
 
Unable to push these repos:
mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
This was a test push that failed; not proceeding any further with mozilla-beta!
 
Summary is non-zero:
 
error - Unable to push mozilla-beta. failed.
 
 
14:32:30    ERROR -  remote: error: denying non-fast-forward refs/heads/GECKO90_2011121217_RELBRANCH (you should pull first)
14:32:30    ERROR -  ! [remote rejected] GECKO90_2011121217_RELBRANCH -> GECKO90_2011121217_RELBRANCH (non-fast-forward)
14:32:30    ERROR -  error: failed to push some refs to '/opt/vcs2vcs/build/target/beagle/.git'
14:32:30    ERROR - Return code: 256
14:32:30    ERROR - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
14:32:30    ERROR - This was a test push that failed; not proceeding any further with mozilla-beta!
14:32:30    ERROR - Unable to push mozilla-beta. failed.
14:32:30    FATAL - Unable to push these repos:
14:32:30    FATAL - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
14:32:30    FATAL - This was a test push that failed; not proceeding any further with mozilla-beta!
14:32:30    FATAL -
14:32:30    FATAL - Running post_fatal callback...
</pre>
 
This is a failure email involving [https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo#How_to_deal_with_GECKO90_2011121217_RELBRANCH this issue].  Luckily, we have a fix for that one already.


=== How to add a repo to gecko-dev or gecko-git ===
Failure emails will be sent for every failure currently, so will be very spammy until they're fixed.


This shouldn't be done lightly.  Gecko.git is our partner-oriented repo and should be treated with kid gloves.  gecko-dev is our release-trains and inbound-branches only.
== Maintenance ==
=== How to add a repo to gecko-git ===
'''''CAUTION:''' production gecko.git is still handled by legacy vcs-sync - see [http://people.mozilla.org/~hwine/tmp/vcs2vcs/cookbook.html#adding-a-branch-to-gecko-git legacy docs].''<br />
This shouldn't be done lightly.  Gecko.git is our partner-oriented repo and should be treated with kid gloves.   
=== How to add another repo to mirror for b2g OS ===
Mirroring repositories for purpose of b2g os builds is handled by the legacy docs. See [http://people.mozilla.org/~hwine/tmp/vcs2vcs/cookbook.html#mirroring-a-git-repository here].
=== How to add a repo to gecko-dev ===
gecko-dev is our developer focused repository with all gecko branches related to release-trains and inbound-branches only.


You should follow the example [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l96 here].  Annotated:
You should follow the example [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l96 here].  Annotated:
Line 59: Line 254:


     "target_dest": "gitmo-beagle",                      # This is a location specified in the remote targets
     "target_dest": "gitmo-beagle",                      # This is a location specified in the remote targets
    "vcs": "git",                                        # http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l332
                                                        # http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l332
}, {
}, {


     "target_dest": "github-beagle",                      # Also in remote_targets
     "target_dest": "github-beagle",                      # Also in remote_targets
    "vcs": "git",
}],
}],


"bare_checkout": True,                                  # This is always set to True; we can probably obsolete
"vcs": "hg",                                            # The "vcs" settings are kind of glossed over at the moment,
"vcs": "hg",                                            # The "vcs" settings are kind of glossed over at the moment,
                                                         # but when we get git<->git and git->hg going it'll matter.
                                                         # but when we get git<->git and git->hg going it'll matter.
Line 93: Line 286:


If the project isn't under hg.m.o/projects, add it [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/project-branches.py#l32 here].
If the project isn't under hg.m.o/projects, add it [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/project-branches.py#l32 here].
=== How to add a project cloning to a custom github account/project ===
* Key setup
** Generate an ssh key for pushing to the github project (no passphrase)
** Add the key to github/org-name/project-name as a deploy key
** Add the key to the vcs2vcs account on the vcssync server
* Add the project to mapper for hg to git hash mapping
** Get a token from https://mozilla-releng.net/tokens
** POST to /project-name on https://mapper.mozilla-releng.net
* Filesystem setup on the vcssync server
** Clone the source hg repo to /opt/vcs2vcs/vcs_sync_build/build/stage_source/
** Clone the hg repo from stage_source to conversion <pre>hg clone /opt/vcs2vcs/vcs_sync_build/build/stage_source/$PROJECT /opt/vcs2vcs/vcs_sync_build/build/conversion/$PROJECT
</pre>
* Add to a vcs-sync configuration file. For example: https://hg.mozilla.org/build/mozharness/rev/57a550a9e307863ab125f1cd1c381de974caf79d
** Set the source location
** Set the destination org/project
* Monitor results and fix/tweak
** Join and watch the error emails in https://groups.google.com/a/mozilla.com/forum/?hl=en&pli=1*!forum/releng-ops-trial
** Check the github project for the commits to appear
** Compare the github commits with the hg commits


=== How to add a locale to l10n conversion ===
=== How to add a locale to l10n conversion ===
Line 112: Line 326:
If <code>skip_empty_messages</code> is set to True, no emails without a message body will be sent (these should only be successful noop runs with no warnings).
If <code>skip_empty_messages</code> is set to True, no emails without a message body will be sent (these should only be successful noop runs with no warnings).


Allowing both of these to be False will result in a lot of email, approximately one per minute.  Just setting <code>skip_empty_messages</code> to True will send email per successful push, any sort of warning, or failures.
Allowing both of these to be False will result in a lot of email, approximately one per minute per process.  Just setting <code>skip_empty_messages</code> to True will send email per successful push, any sort of warning, or failures.


=== How to force the process to pull/bookmark/convert/push a repo, even if nothing's changed ===
=== How to force the process to pull/bookmark/convert/push a repo, even if nothing's changed ===
Line 121: Line 335:


This can also be set in the config file per-repository, as <code>check_incoming</code> inside each of the specific <code>conversion_repos</code> where we want to skip the incoming check.
This can also be set in the config file per-repository, as <code>check_incoming</code> inside each of the specific <code>conversion_repos</code> where we want to skip the incoming check.
To force a push manually:
* cd into project-branches jobs top level directory
* wait for projects.lock to disappear (from cron job)
* touch projects.lock # grab lock to avoid race with automation
** or: <code>touch my_user_name.lock; while ! ln my_user_name.lock projects.lock ; do sleep 10 ; done</code>
** wait for return of command prompt
* get the command line from the run_*.sh file: grep python run_*.sh
* add the '--no-check-incoming' option to the command
* wait for processing to complete
** '''NOTE''': the log output is only to your screen. The usual summary, "<code>projects.log</code>" is only updated via the <code>run_*</code> script. ({{bug|1279008|see also}})
*** The official logs from the last run are in the "<code>./logs/</code>" directory.
*** Older logs are in the "<code>./build/upload/logs/</code>" directory.
* rm 'projects.lock' # let normal processing resume
=== Where to find the keys ===
The keys are the same as the keys on vcs2vcs@gd{0..4}, and has been shared on Google Drive to various RelEngers... except the passphrase has been stripped from them: <code>ssh-keygen -p -f FILE</code>
=== How to move a VCS Sync process ===
* pause the cron job
* move the work dir (or rsync, if across machines).
** You need to move build/conversion or you'll start from scratch, which is definitely a no-no for mozilla-central based repos (see https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo#How_to_start_a_mozilla-central_based_repo_conversion_from_scratch)
** Moving build/stage_source will save a lot of cloning time, especially for processes with a lot of large repos like the project-branches process.
** If the path changes, you need to delete (or not move) build/venv.
** If you're trying to move it cleanly, don't move build/target or build/upload.  If you want to preserve the old history, move them.
* If you're on a new machine, you need to make sure it's set up properly.  (Should be easier once it's puppetized)
* make sure the cron job script is pointing at the right location
* restart the cron job


=== How to start a mozilla-central based repo conversion from scratch ===
=== How to start a mozilla-central based repo conversion from scratch ===
Line 168: Line 412:
=== How to deal with project branch reset ===
=== How to deal with project branch reset ===


To some degree, the integration-gecko-projects repo should be self-healing.  If the incoming changesets to a repo are incompatible (so an <code>hg pull</code> doesn't work), the script should blow away the source repo and reclone.  The integration-gecko-projects target has <code>force_push</code> set to True, so it should be pushing with <code>git push -f</code>.
To some degree, the integration-gecko-projects repo should be self-healing.  If the incoming changesets to a repo are incompatible (so an <code>hg pull</code> doesn't work), the script should blow away the source repo and reclone.  The integration-gecko-projects target will shortly have <code>force_push</code> set to True, so it should be pushing with <code>git push -f</code>.


However, project branch resets may become an issue; time will tell.
However, project branch resets may become an issue; time will tell.
Line 239: Line 483:
hg out
hg out
hg push
hg push
</pre>
Then update https://wiki.mozilla.org/ReleaseEngineering/VCSSync/History#Relbranch_issues .
==== Hints ====
<pre>
hg glog 0..REVISION
</pre>
</pre>
Garbas
14

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/728866...1196481"