Personal tools

ReleaseEngineering/VCSSync/HowTo

From MozillaWiki

Jump to: navigation, search

This is documentation on the mozharness-based vcs sync processes.

See Hal's docs for the legacy process.

Contents

Quick notes

What's running so far

Everything else is currently being run in Hal's legacy processes.

Machines

2013.10.14: We're currently using the following machines:

Machine Process Location User
vcssync1.srv.releng.usw2.mozilla.com beagle (gecko-dev) /opt/vcs2vcs vcs2vcs
vcssync2.srv.releng.usw2.mozilla.com project-branches (gecko-projects) /opt/vcs2vcs vcs2vcs
vcssync-dev staging /opt/vcs2vcs vcs2vcs
github-sync2.dmz.scl3.mozilla.com staging (initial conversion, gecko-git, l10n) /home/asasaki/{initial2,gecko,l10n} asasaki
github-sync2.dmz.scl3.mozilla.com build repos (autoland, buildapi, buildbot-configs, buildbotcustom, cloud-tools, mozharness, opsi-package-sources, partner-repacks, preproduction, puppet, puppet-manifests, rpm-sources, talos, tools) /opt/vcs2vcs vcs2vcs
github-sync4.dmz.scl3.mozilla.com staging (project branches) /home/asasaki/projects asasaki

Plus a few other github-sync machines for Hal's legacy processes.

Branch

2013.10.14: We're based off the mozharness production branch.

Specific instructions

Emails

Successful noop emails

From: vcs2vcs@vcssync2.srv.releng.usw2.mozilla.com
Subject: [vcs2vcs] Successful conversion for project-branches (12s) <EOM>

These are pretty much there to tell you that the job's running, everything's fine, and the noop time was (in this case) 12 seconds. The <EOM> tells you that the body of the email is empty (end of message).

The noop time varies depending on hg.m.o load, as well as log upload times.

See here for enabling/disabling these... they can be pretty spammy.

Successful emails with messages

From: vcs2vcs@vcssync1.srv.releng.usw2.mozilla.com
Subject: [vcs2vcs] Successful conversion for beagle (72s)


Summary is non-zero:

info - Successfully pushed mozilla-aurora.
error - Error getting changes for mozilla-inbound; skipping!


02:16:08    ERROR -  abort: HTTP Error 500: Internal Server Error
02:16:08    ERROR -  Automation Error: hg not responding
02:16:08    ERROR - Return code: 65280
02:16:08    ERROR - Error getting changes for mozilla-inbound; skipping!

Whenever the summary of a job is non-zero, or the error log is non-zero, we'll get a message body in the success email. This is usually a successful push notification, but can include the contents of the error log or any other information we put in the summary. (At some future point we may add l10n repo creation on git.m.o in here).

In this case, the ISE from hg.m.o is an intermittent issue. In general, the script should auto-fix by clobbering+recloning, and the conversion will happen later, though it'll be delayed by this action.

Failure emails

From: vcs2vcs@vcssync-dev.build.mozilla.com
Subject: [vcs2vcs] Failed conversion for beagle

Unable to push these repos:
mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
This was a test push that failed; not proceeding any further with mozilla-beta!

Summary is non-zero:

error - Unable to push mozilla-beta. failed.


14:32:30    ERROR -  remote: error: denying non-fast-forward refs/heads/GECKO90_2011121217_RELBRANCH (you should pull first)
14:32:30    ERROR -   ! [remote rejected] GECKO90_2011121217_RELBRANCH -> GECKO90_2011121217_RELBRANCH (non-fast-forward)
14:32:30    ERROR -  error: failed to push some refs to '/opt/vcs2vcs/build/target/beagle/.git'
14:32:30    ERROR - Return code: 256
14:32:30    ERROR - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
14:32:30    ERROR - This was a test push that failed; not proceeding any further with mozilla-beta!
14:32:30    ERROR - Unable to push mozilla-beta. failed.
14:32:30    FATAL - Unable to push these repos:
14:32:30    FATAL - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
14:32:30    FATAL - This was a test push that failed; not proceeding any further with mozilla-beta!
14:32:30    FATAL - 
14:32:30    FATAL - Running post_fatal callback...

This is a failure email involving this issue. Luckily, we have a fix for that one already.

Failure emails will be sent for every failure currently, so will be very spammy until they're fixed.

Maintenance

How to add a repo to gecko-dev or gecko-git

This shouldn't be done lightly. Gecko.git is our partner-oriented repo and should be treated with kid gloves. gecko-dev is our release-trains and inbound-branches only.

You should follow the example here. Annotated:

"repo": "https://hg.mozilla.org/releases/mozilla-beta",  # where we should pull the repo from.
"revision": "default",                                   # what revision to pull
"repo_name": "mozilla-beta",                             # the name we'll know this repo by.  This should be unique within this job.

"targets": [{                                            
    "target_dest": "beagle/.git",                        # This is a location on disk (test_push is True)
    "vcs": "git",                                        # This was used for non-fastforward detection; might be overkill at this point.
    "test_push": True,
}, {

    "target_dest": "gitmo-beagle",                       # This is a location specified in the remote targets
                                                         # http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l332
}, {

    "target_dest": "github-beagle",                      # Also in remote_targets
}],

"vcs": "hg",                                             # The "vcs" settings are kind of glossed over at the moment,
                                                         # but when we get git<->git and git->hg going it'll matter.

"branch_config": {
    "branches": {
        "default": "beta",                               # the 'default' hg branch should become a 'beta' git branch.
                                                         # This is really needed by the all-in-one repos where there are multiple
                                                         # 'default' branches from hg that need to each be their own unique branch.
    },
    "branch_regexes": [
        "^GECKO[0-9_]*RELBRANCH$",                       # Whitelist the gecko and mobile release branches without having to
        "^MOBILE[0-9_]*RELBRANCH$",                      # list them all by name.
    ],
},
"tag_config": {
    "tag_regexes": [
        "^(B2G|RELEASE_BASE)_",                          # Whitelist the B2G and RELEASE_BASE tags.
                                                         # Explicitly avoid the release tags since they currently move in hg-land.
    ],
},

How to add a project to project-branch conversion

If the project is under hg.m.o/projects, add it here.

If the project isn't under hg.m.o/projects, add it here.

How to add a locale to l10n conversion

L10n conversions are based on this config file.

These point at gecko all-locales and gaia languages_dev.json files. To add a locale, update the appropriate all-locales or languages_dev.json file.

If the b2g versions are changing against gecko trains (as they will every 6 weeks), you need to update this section.

How to adjust email notifications

The notify_config's are in each config file, like this.

This is a list-of-dictionaries. Each dictionary has a "to" email address.

If failure_only is set to True, no emails with a successful outcome will be sent to this address.

If skip_empty_messages is set to True, no emails without a message body will be sent (these should only be successful noop runs with no warnings).

Allowing both of these to be False will result in a lot of email, approximately one per minute per process. Just setting skip_empty_messages to True will send email per successful push, any sort of warning, or failures.

How to force the process to pull/bookmark/convert/push a repo, even if nothing's changed

There is a --no-check-incoming commandline option to the vcs_sync.py script.

This can also be set in the config file, as check_incoming set to True.

This can also be set in the config file per-repository, as check_incoming inside each of the specific conversion_repos where we want to skip the incoming check.

Where to find the keys

The keys are the same as the keys on vcs2vcs@gd{0..4}, and has been shared on Google Drive to various RelEngers... except the passphrase has been stripped from them: ssh-keygen -p -f FILE

How to move a VCS Sync process

  • pause the cron job
  • move the work dir (or rsync, if across machines).
  • If you're on a new machine, you need to make sure it's set up properly. (Should be easier once it's puppetized)
  • make sure the cron job script is pointing at the right location
  • restart the cron job

How to start a mozilla-central based repo conversion from scratch

  • First, you need a copy of initial3.tar.bz2. This is mozilla-central-with-cvs-history, already converted to git for you, up to Spring, 2013. Without this file, you will need to run the initial conversion with this config file (or beagle) and let it run for about a week.
    • The sha512 sum is 0a3243fe5a6c8ffa4e47131e0eb0243e1f5676ea3cacd535d11b424f8f601511130a2e941670950b63c0b00726dafa9bd30bf7c3d040752fce824158021ef014
    • This file exists on vcssync{1,2} at /opt/vcs2vcs/initial3.tar.bz2 , on github-sync{2,4} at /home/asasaki/initial3.tar.bz2 , and on Aki's laptop.
  • Second, you need a config file for the conversion type. This is probably covered with the beagle, gecko-git, and project-branches config files here.
  • Clone the appropriate mozharness repo + branch (see here, unless you're working off a separate development branch). This will go into the base_work_dir. For instance, if you want to use /opt/vcs2vcs as the base_work_dir, this will be /opt/vcs2vcs/mozharness .
  • Extract the contents of initial3.tar.bz2 into the appropriate location. For instance, if I want to use /opt/vcs2vcs/ as my base_work_dir, and I'm using this config file (which specifies "gecko-git" as my conversion_dir), I'll need to move some files around.
cd /opt/vcs2vcs
tar xjvf initial3.tar.bz2  # this creates conversion/beagle
mkdir -p build/conversion
# move the extracted directory from initial3.tar.bz2 to where
# the mozharness script expects the gecko-git conversion dir to be
mv conversion/beagle build/conversion/gecko-git
rmdir conversion
  • Run the conversion once if you want to make sure.
cd /opt/vcs2vcs
# For example,
python mozharness/scripts/vcs-sync/vcs_sync.py -c mozharness/configs/vcs-sync/CONFIGFILE.py  # [--no-push] [--no-upload] [--no-notify] if desired
    • If you want to avoid certain actions, you can specify that you want to skip them. --no-push, --no-upload, and --no-notify might be some options you want to use, depending on the situation. --help and --list-actions should be helpful as well.
  • Run in cron like the other boxen.

Troubleshooting

How to deal with non-ffwd

  1. Determine what happened. This can generally be found with
    hg heads
    . If there are multiple heads for a branch, you'll need to follow the below instructions.
  2. Determine if you want to discard one of the heads, or merge them. (Do you want the changes from both heads, or do you want one head's changes to stick and discard the other one?)
    • If you want to merge the two heads, you can do so with hg merge, hg commit, and hg push
    • If we want to discard one,
# we're on good revision X
# we have old [bad] revision Y, which is a different head 
hg debugsetparents X Y; hg commit -m "merging X and Y via debugsetparents" # Create a new changeset that has parents X and Y
hg glog | more # see that the history has merged
hg diff -r X # see that the code is the same as revision X, no diffs
hg push

If you hit a relbranch issue you may need to follow these instructions instead.

How to deal with project branch reset

To some degree, the integration-gecko-projects repo should be self-healing. If the incoming changesets to a repo are incompatible (so an hg pull doesn't work), the script should blow away the source repo and reclone. The integration-gecko-projects target will shortly have force_push set to True, so it should be pushing with git push -f.

However, project branch resets may become an issue; time will tell.

We can probably:

and it'll clone a new copy of the branch, convert, and push.

If there are still issues, we can debug further to try to get it to work. A final nuke-the-site-from-orbit solution is documented here, but may take the repo down for a day or so.

How to deal with completely resetting gecko-projects

  1. Make the decision that this is the solution to the problem.
  2. Make sure people are aware this is going to happen.
  3. Stop the conversion of gecko-projects. This is currently on vcs2vcs@vcssync2, and is run via cron.
  4. Save a copy of https://raw.github.com/mozilla/integration-gecko-projects/master/README.md
  5. Delete the repo! This option is available in the "Danger Zone" area of https://github.com/mozilla/integration-gecko-projects/settings
  6. Recreate the repo. There's a "Create a new repo" icon in the top right. This will need the same name, should be public, and you want to initialize the repo with a README. (gecko-projects is the only repo where we want the README: essentially, we want an empty master branch because github doesn't allow you to delete the first populated branch in a repo.)
  7. Edit the README with the contents you downloaded in step 4.
  8. Start the conversion back up. You may only need to delete the build/target and build/stage_source/BAD_REPO directories. Or you may need to blow away the whole thing from orbit to be sure, in which case you should follow the docs for how to start a mozilla-central based repo conversion from scratch.
    • Note, just cloning the hg repos for the projects repo can take many many hours, so this will take the gecko-projects repo down for a long time.

How to deal with GECKO90_2011121217_RELBRANCH

We were getting email with the following failure:

11:14:01    ERROR -  remote: error: denying non-fast-forward refs/heads/GECKO90_2011121217_RELBRANCH (you should pull first)
11:14:01    ERROR -   ! [remote rejected] GECKO90_2011121217_RELBRANCH -> GECKO90_2011121217_RELBRANCH (non-fast-forward)
11:14:01    ERROR -  error: failed to push some refs to '/opt/vcs2vcs/build/target/beagle/.git'
11:14:02    ERROR - Return code: 256
11:14:02    ERROR - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
11:14:02    ERROR - This was a test push that failed; not proceeding any further with mozilla-beta!

To debug:

# vcs2vcs@vcssync1
cd /opt/vcs2vcs/build/target/beagle
git show GECKO90_2011121217_RELBRANCH  # this gave 1003ec79451969335008880ad82e305d93b89642
cd /opt/vcs2vcs/build/conversion/beagle
git show GECKO90_2011121217_RELBRANCH  # this gave fac7279c040d643fb4c35105fa85b9335ba2c2f9
git merge-base fac7279c040d643fb4c35105fa85b9335ba2c2f9 1003ec79451969335008880ad82e305d93b89642

The merge-base returned fac7279c040d643fb4c35105fa85b9335ba2c2f9. That means that fac727 is a parent to 1003ec. No sha divergence, only stale history.

I noticed that http://hg.mozilla.org/releases/mozilla-beta/rev/GECKO90_2011121217_RELBRANCH pointed at a9221b332d8a (dec 15) and http://hg.mozilla.org/releases/mozilla-release/rev/GECKO90_2011121217_RELBRANCH pointed at 4e309e63c279 (jan 3); Calendar must have done a release off the relbranch off mozilla-release. Since the branch name is shared across mozilla-beta and mozilla-release, converting the branch from mozilla-beta effectively does a non-fastforward push and is rejected.

To fix, I exported the two revisions from mozilla-release to mozilla-beta:

cd mozilla-release
hg export -r 79350 > ../cal2
hg export -r 79349 > ../cal1
# Doublecheck to make sure those look good
cd ../mozilla-beta
hg pull -u
hg up -r GECKO90_2011121217_RELBRANCH
hg import ../cal1
hg ident  # sha matched m-r
hg import ../cal2
hg ident  # sha matched m-r
hg out
hg push

Then update https://wiki.mozilla.org/ReleaseEngineering/VCSSync/History#Relbranch_issues .

Hints

hg glog 0..REVISION