MozillaWiki

ReleaseEngineering/How To/VCSSync: Difference between revisions

Page Discussion

ReleaseEngineering/How To/VCSSync (view source)

Revision as of 02:12, 30 June 2018

7,588 bytes added ,  30 June 2018
new mapper url
(new mapper url)
 
(19 intermediate revisions by 5 users not shown)
Line 3: Line 3:
This is documentation on the mozharness-based vcs sync processes.
This is documentation on the mozharness-based vcs sync processes.


<b>See [http://people.mozilla.org/~hwine/tmp/vcs2vcs/ Hal's docs] for the legacy process.</b>
<div id="legacy"></div>
<b>See [http://people.mozilla.org/~hwine/tmp/vcs2vcs/ legacy docs] for the legacy (aka "old") process.</b>


= Quick notes =
= Quick notes =
== What's running so far ==


* beagle, which is https://github.com/mozilla/integration-gecko-dev and http://git.mozilla.org/?p=integration/gecko-dev.git;a=summary
"VCS-sync" is a system which has evolved from it's original purpose to being the tool used whenever "this repository needs to be publicly visible 'over here'", perhaps with modifications, and a few exceptions on "publicly visible". The current (2016 Q1) business needs it addresses are:
* gecko-projects, which is https://github.com/mozilla/integration-gecko-projects


Everything else is currently being run in Hal's legacy processes.
# Support for gecko developers who prefer to work with git:
#* All gecko primary development and release branches are merged into a single git repository (with CVS history) in "gecko-dev.git".
#* All gecko twig and misc branches are merged into a single git repository (with CVS history) in "gecko-projects.git".
# Support for Mozilla development tools and processes which (for operational needs (e.g. travis-ci), or developer preference, should appear in both Mercurial and Git repositories.
#* These are "mirrors" from one system to the other.
#* One repo is always designated the RoR (Repository of Record), and is the only one where commits will "stick".
#* Refer to the system configurations for each mirror to determine the RoR. (It can, and has, changed over time for some repositories.)
# Support for Partner facing FxOS development processes.
#* The URL's for these repositories can not change, as they are referenced in manifest files which comprise the delivery to Partners.
#* These repositories on git.mozilla.org satisfy various contractual needs with FxOS partners.
#* These repositories must not contain either object deletions, or non-fast-forward commits.
#* These repositories only contain content that was "delivered" to Partners, with branches named according to FxOS conventions.
#* These repositories include any "Mozilla Contributions" to FxOS, which includes code beyond gecko.git and gaia.git.
# Support for internal FxOS development processes.
#* Many Android repositories have been mirrored to git.mozilla.org for the purpose of having a known-good-repository for the build systems to clone from.
#* Some tests still require translation from Git &rarr; Mercurial format. Each such gaia.git branch is mapped to a distinct Mercurial repository.
 
 
== What's running where ==
 
Anything not listed below is probably running in the new system. The system pushing the production content is listed first. If the other system is listed, we're in parallel operations mode and changes should be made to both systems.
 
=== Repositories needed to Build Mozilla Products ===
{| class="fullwidth-table sortable"
| style="background:#cccccc" | '''Repository'''
| style="background:#cccccc" | '''Old/New'''
| style="background:#cccccc" | '''Web views at'''
| style="background:#cccccc" | '''Notes'''
|-
| gecko-dev.git
| new
| [https://github.com/mozilla/gecko-dev.git github], [http://git.mozilla.org/?p=integration/gecko-dev.git;a=summary Moz]
|
|-
| gecko-projects.git
| new
| [https://github.com/mozilla/gecko-projects.git github], [http://git.mozilla.org/?p=integration/gecko-projects.git;a=summary Moz]
|
|-
| releases/gecko.git
| old, new
| [http://git.mozilla.org/?p=releases/gecko.git;a=summary Moz]
| Partner facing FxOS <nowiki>[1]</nowiki>
|-
| releases/gaia.git
| old, new
| [http://git.mozilla.org/?p=releases/gaia.git;a=summary Moz]
| Partner facing FxOS <nowiki>[1]</nowiki>
|-
| releases/l10n/<locale>/{gecko,gaia}.git
| old, new
| [http://git.mozilla.org/ Moz] (browse from there)
| Partner facing FxOS <nowiki>[1]</nowiki>
|-
| integration/gaia*
| old
| [http://hg.mozilla.org/integration/ Moz] (browse from there)
| Hg versions of gaia branches to support older tooling
|-
| b2g mirrors
| old
| [http://git.mozilla.org/ Moz] browse <tt>external</tt> and <tt>b2g</tt> from there (r/o)
| Local copies to ensure availability during builds.
|}
 
Notes:
# Partner facing repositories are configured to:
#* not accept non fast forward commits
#* not accept deletes
#* only contain branches related to shipping versions of FxOS
 
 
=== Repositories Supporting Mozilla Web Products ===
{| class="fullwidth-table sortable"
| style="background:#cccccc" | '''Repository'''
| style="background:#cccccc" | '''Old/New'''
| style="background:#cccccc" | '''Web views at'''
| style="background:#cccccc" | '''Notes'''
|-
| bugzilla-{bugzilla,qa}.git
| old
| [http://git.mozilla.org/ Moz] (r/w), [https://github.com/bugzilla/ github] (r/o)
| Only the bugzilla.git & qa.git repos are mirrored to github
|-
| webtools-bmo-{bugzilla,qa}.git
| old
| [http://git.mozilla.org/ Moz] (r/w), [https://github.com/mozilla/ webtools-bmo-* on github] (r/o)
|
|}
 
=== Repositories Supporting Infrastructure, Tooling, and Miscellaneous ===
See also [[ReleaseEngineering/Repositories|full list of RelEng repositories]].
{| class="fullwidth-table sortable"
| style="background:#cccccc" | '''Repository'''
| style="background:#cccccc" | '''Old/New'''
| style="background:#cccccc" | '''Web views at'''
| style="background:#cccccc" | '''Notes'''
|-
| build-releng-api.git
| old
| [http://github.com/mozilla/build-releng-api.git github] (r/o), [http://hg.mozilla.org/build/releng-api Moz] (r/o)
|-
| automation/fuzz-tools.git
| old
| [http://git.mozilla.org/?p=automation/fuzz-tools.git Moz] (r/w), [https://github.com/mozilla/automation-fuzz-tools.git github] (r/o)
|
|-
| qa/mozmill-tests
| old
| [http://hg.mozilla.org/qa/mozmill-tests Moz] (r/w), [https://github.com/mozilla/qa-mozmill-tests github] (r/o)
|
|}


== Machines ==
== Machines ==
Line 32: Line 142:
| vcs2vcs
| vcs2vcs
|-
|-
| vcssync-dev
| github-sync2.dmz.scl3.mozilla.com
| staging
| build repos (autoland, buildapi, buildbot-configs, buildbotcustom, cloud-tools, mozharness, opsi-package-sources, partner-repacks, preproduction, puppet, puppet-manifests, rpm-sources, talos, tools)
| /opt/vcs2vcs
| /opt/vcs2vcs
| vcs2vcs
| vcs2vcs
|-
| github-sync2.dmz.scl3.mozilla.com
| staging (initial conversion, gecko-git, l10n)
| /home/asasaki/{initial2,gecko,l10n}
| asasaki
|-
|-
| github-sync4.dmz.scl3.mozilla.com
| github-sync4.dmz.scl3.mozilla.com
| staging (project branches)
| gecko l10n and gaia l10n
| /home/asasaki/projects
| /opt/vcs2vcs
| asasaki
| vcs2vcs
|}
|}


Plus a [http://people.mozilla.org/~hwine/tmp/vcs2vcs/trouble_shooting.html#accessing-the-machines few other github-sync machines] for Hal's legacy processes.
Plus a [http://people.mozilla.org/~hwine/tmp/vcs2vcs/trouble_shooting.html#accessing-the-machines few other github-sync machines] for legacy processes.


== Branch ==
== Branch ==
Line 102: Line 207:
mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
This was a test push that failed; not proceeding any further with mozilla-beta!
This was a test push that failed; not proceeding any further with mozilla-beta!
Summary is non-zero:
error - Unable to push mozilla-beta. failed.


14:32:30    ERROR -  remote: error: denying non-fast-forward refs/heads/GECKO90_2011121217_RELBRANCH (you should pull first)
14:32:30    ERROR -  remote: error: denying non-fast-forward refs/heads/GECKO90_2011121217_RELBRANCH (you should pull first)
Line 115: Line 225:
14:32:30    FATAL -  
14:32:30    FATAL -  
14:32:30    FATAL - Running post_fatal callback...
14:32:30    FATAL - Running post_fatal callback...
Summary is non-zero:
error - Unable to push mozilla-beta. failed.
</pre>
</pre>


Line 126: Line 232:


== Maintenance ==
== Maintenance ==
 
=== How to add a repo to gecko-git ===
=== How to add a repo to gecko-dev or gecko-git ===
'''''CAUTION:''' production gecko.git is still handled by legacy vcs-sync - see [http://people.mozilla.org/~hwine/tmp/vcs2vcs/cookbook.html#adding-a-branch-to-gecko-git legacy docs].''<br />
 
This shouldn't be done lightly.  Gecko.git is our partner-oriented repo and should be treated with kid gloves.   
This shouldn't be done lightly.  Gecko.git is our partner-oriented repo and should be treated with kid gloves.  gecko-dev is our release-trains and inbound-branches only.
=== How to add another repo to mirror for b2g OS ===
Mirroring repositories for purpose of b2g os builds is handled by the legacy docs. See [http://people.mozilla.org/~hwine/tmp/vcs2vcs/cookbook.html#mirroring-a-git-repository here].
=== How to add a repo to gecko-dev ===
gecko-dev is our developer focused repository with all gecko branches related to release-trains and inbound-branches only.


You should follow the example [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l96 here].  Annotated:
You should follow the example [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l96 here].  Annotated:
Line 151: Line 260:
}],
}],


"bare_checkout": True,                                  # This is always set to True; we can probably obsolete
"vcs": "hg",                                            # The "vcs" settings are kind of glossed over at the moment,
"vcs": "hg",                                            # The "vcs" settings are kind of glossed over at the moment,
                                                         # but when we get git<->git and git->hg going it'll matter.
                                                         # but when we get git<->git and git->hg going it'll matter.
Line 178: Line 286:


If the project isn't under hg.m.o/projects, add it [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/project-branches.py#l32 here].
If the project isn't under hg.m.o/projects, add it [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/project-branches.py#l32 here].
=== How to add a project cloning to a custom github account/project ===
* Key setup
** Generate an ssh key for pushing to the github project (no passphrase)
** Add the key to github/org-name/project-name as a deploy key
** Add the key to the vcs2vcs account on the vcssync server
* Add the project to mapper for hg to git hash mapping
** Get a token from https://mozilla-releng.net/tokens
** POST to /project-name on https://mapper.mozilla-releng.net
* Filesystem setup on the vcssync server
** Clone the source hg repo to /opt/vcs2vcs/vcs_sync_build/build/stage_source/
** Clone the hg repo from stage_source to conversion <pre>hg clone /opt/vcs2vcs/vcs_sync_build/build/stage_source/$PROJECT /opt/vcs2vcs/vcs_sync_build/build/conversion/$PROJECT
</pre>
* Add to a vcs-sync configuration file. For example: https://hg.mozilla.org/build/mozharness/rev/57a550a9e307863ab125f1cd1c381de974caf79d
** Set the source location
** Set the destination org/project
* Monitor results and fix/tweak
** Join and watch the error emails in https://groups.google.com/a/mozilla.com/forum/?hl=en&pli=1*!forum/releng-ops-trial
** Check the github project for the commits to appear
** Compare the github commits with the hg commits


=== How to add a locale to l10n conversion ===
=== How to add a locale to l10n conversion ===
Line 206: Line 335:


This can also be set in the config file per-repository, as <code>check_incoming</code> inside each of the specific <code>conversion_repos</code> where we want to skip the incoming check.
This can also be set in the config file per-repository, as <code>check_incoming</code> inside each of the specific <code>conversion_repos</code> where we want to skip the incoming check.
To force a push manually:
* cd into project-branches jobs top level directory
* wait for projects.lock to disappear (from cron job)
* touch projects.lock # grab lock to avoid race with automation
** or: <code>touch my_user_name.lock; while ! ln my_user_name.lock projects.lock ; do sleep 10 ; done</code>
** wait for return of command prompt
* get the command line from the run_*.sh file: grep python run_*.sh
* add the '--no-check-incoming' option to the command
* wait for processing to complete
** '''NOTE''': the log output is only to your screen. The usual summary, "<code>projects.log</code>" is only updated via the <code>run_*</code> script. ({{bug|1279008|see also}})
*** The official logs from the last run are in the "<code>./logs/</code>" directory.
*** Older logs are in the "<code>./build/upload/logs/</code>" directory.
* rm 'projects.lock' # let normal processing resume


=== Where to find the keys ===
=== Where to find the keys ===
Garbas
14

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/763551...1196481"