Release:Release Automation on Mercurial:Staging Specific Notes

From MozillaWiki
Jump to: navigation, search

Release automation in staging is mostly the same as in production, but does have a few differences you should know about:

  • Use the config files with the prefix "staging_". These have many values already set correctly for staging.
  • All uploading is done to and the files are visible at Thunderbird uses for uploads, same downloads from the same host but different candidates directory.
  • It can take a few tries to get the repos created properly. This is because hgweb sometimes returns a 500 (internal server error) when we query about a locale. Just retry the failing repos.
  • Staging doesn't have any default set of slaves, you'll need to make loan requests. You can prevent your master from running dep builds using limit_branches in master_config.json.

How to start a staging release

Setup staging repos

You can use the stage-ffxbld repos or your own hg user account for the repos. If you use stage-ffxbld you most likely need to run this script on a build slave with staging ssh keys available. For your own user repos just make sure you have your own key in ssh-agent (or equivalent).

This is what you'd do for testing a X.0b1 release ahead of time:


for repo in compare-locales tools buildbotcustom buildbot buildbot-configs; do
    echo "deleting $repo"
    ssh edit $repo delete YES
    echo "cloning $repo"
    ssh clone $repo build/$repo

# substitute mozilla-aurora --> mozilla-beta if you want to do a beta from the current cycle, or release before merging from beta
for l in `wget -q -O- |grep -v en-US | awk '{print $1}'`; do
    echo "deleting locale $l"
    ssh edit $l delete YES
    echo "cloning locale $l"
    ssh clone $l releases/l10n/mozilla-aurora/$l

# these may change if you're testing mozilla-release
ssh edit mozilla-beta delete YES
ssh clone mozilla-beta releases/mozilla-beta

Running the merge scripts

You'll need this if you're doing a b1 before we actually merge. This is based on the main docs for migration.

# checkout mozharness from the gecko tree using archiver
python mozharness --repo mozilla-central --rev default --debug

Add a patch like this to suppress tagging of the production aurora repo

diff -ru mozharness.base/configs/merge_day/ mozharness/configs/merge_day/
--- mozharness.base/configs/merge_day/	2016-02-25 14:32:51.000000000 -0800
+++ mozharness/configs/merge_day/	2016-02-25 16:44:44.000000000 -0800
@@ -8,8 +8,8 @@

     "tools_repo_url": "",
     "tools_repo_revision": "default",
-    "from_repo_url": "ssh://",
-    "to_repo_url": "ssh://",
+    "from_repo_url": "",
+    "to_repo_url": "ssh://",

     "base_tag": "FIREFOX_BETA_%(major_version)s_BASE",
     "end_tag": "FIREFOX_BETA_%(major_version)s_END",
diff -ru mozharness.base/scripts/merge_day/ mozharness/scripts/merge_day/
--- mozharness.base/scripts/merge_day/	2016-02-25 14:32:51.000000000 -0800
+++ mozharness/scripts/merge_day/	2016-02-25 18:05:34.000000000 -0800
@@ -174,7 +174,7 @@

     def query_push_dirs(self):
         dirs = self.query_abs_dirs()
-        return dirs.get('abs_from_dir'), dirs.get('abs_to_dir')
+        return dirs.get('abs_to_dir')

     def query_push_args(self, cwd):
         if cwd == self.query_abs_dirs()['abs_to_dir'] and \
# run the script
python2.7 mozharness/scripts/merge_day/ \
    -c mozharness/configs/merge_day/ \
    -c mozharness/configs/merge_day/

Verify changes are as expected with

hg -R build/mozilla-beta diff | less

Then commit and push

python2.7 mozharness/scripts/merge_day/ \
  -c mozharness/configs/merge_day/ \
  -c mozharness/configs/merge_day/ \
  --create-virtualenv --commit-changes --push

Buildbot configs

When working in staging you'll need to swap out the buildbot-configs, build/tools, buildbotcustom, and compare-locales repos for either stage-ffxbld ones, or your own. If you're using your own, you also need to change some "root" variables that calculated repos are based on. Here's an example of the changes you'll need:

diff --git a/mozilla/ b/mozilla/
--- a/mozilla/
+++ b/mozilla/
@@ -34,3 +34,3 @@ GLOBAL_VARS = {
     'graph_selector': '/server/collect.cgi',
-    'compare_locales_repo_path': 'build/compare-locales',
+    'compare_locales_repo_path': 'users/',
     'compare_locales_tag': 'RELEASE_AUTOMATION',
@@ -2569,3 +2569,3 @@ BRANCHES['mozilla-release']['platforms']
 ######## mozilla-beta
-BRANCHES['mozilla-beta']['repo_path'] = 'releases/mozilla-beta'
+BRANCHES['mozilla-beta']['repo_path'] = 'users/'
 BRANCHES['mozilla-beta']['l10n_repo_path'] = 'releases/l10n/mozilla-beta'
diff --git a/mozilla/ b/mozilla/
--- a/mozilla/
+++ b/mozilla/
@@ -31,4 +31,4 @@ GLOBAL_VARS = {
     'staging': True,
-    'config_repo_path': 'users/stage-ffxbld/buildbot-configs',
-    'buildbotcustom_repo_path': 'users/stage-ffxbld/buildbotcustom',
+    'config_repo_path': 'users/',
+    'buildbotcustom_repo_path': 'users/',
     'stage_server': '',
diff --git a/mozilla/ b/mozilla/
--- a/mozilla/
+++ b/mozilla/
@@ -6,3 +6,3 @@
 # you're starting a release without Release Kickoff. You have been warned.
@@ -58,3 +58,3 @@ releaseConfig['sourceRepositories']  = {
 releaseConfig['l10nRelbranch']       = None
-releaseConfig['l10nRepoPath']        = 'users/stage-ffxbld'
+releaseConfig['l10nRepoPath']        = 'users/'
 releaseConfig['l10nRevisionFile']    = 'l10n-changesets_mobile-beta.json'
@@ -63,4 +63,4 @@ releaseConfig['l10nJsonFile']        = r
 releaseConfig['otherReposToTag']     = {
-    'users/stage-ffxbld/compare-locales': 'RELEASE_AUTOMATION',
-    'users/stage-ffxbld/buildbot': 'production-0.8',
+    'users/': 'RELEASE_AUTOMATION',
+    'users/': 'production-0.8',
@@ -94,3 +94,3 @@ releaseConfig['ausServerUrl']        = '
 releaseConfig['doPartnerRepacks']       = False
-releaseConfig['partnersRepoPath']       = 'users/stage-ffxbld/partner-repacks'
+releaseConfig['partnersRepoPath']       = 'users/'
 releaseConfig['partnerRepackPlatforms'] = ()
@@ -133,3 +133,3 @@ releaseConfig['single_locale_options'] =
         '--tag-override', '%s_RELEASE' % releaseConfig['baseTag'],
-        '--user-repo-override', 'users/stage-ffxbld',
+        '--user-repo-override', 'users/',
         '--cfg', 'single_locale/',
@@ -141,3 +141,3 @@ releaseConfig['single_locale_options'] =
         '--tag-override', '%s_RELEASE' % releaseConfig['baseTag'],
-        '--user-repo-override', 'users/stage-ffxbld',
+        '--user-repo-override', 'users/',
         '--cfg', 'single_locale/',
@@ -158,3 +158,3 @@ releaseConfig['multilocale_config'] = {
         '--tag-override=%s_RELEASE' % releaseConfig['baseTag'],
-        '--user-repo-override=users/stage-ffxbld',
+        '--user-repo-override=users/',
@@ -167,3 +167,3 @@ releaseConfig['multilocale_config'] = {
 # Staging config
-releaseConfig['build_tools_repo_path'] = "users/stage-ffxbld/tools"
+releaseConfig['build_tools_repo_path'] = "users/"
 releaseConfig['enableSigningAtBuildTime'] = True
diff --git a/mozilla/ b/mozilla/
--- a/mozilla/
+++ b/mozilla/
@@ -6,3 +6,3 @@
 # you're starting a release without Release Kickoff. You have been warned.
@@ -72,3 +72,3 @@ releaseConfig['sourceRepositories']  = {
 releaseConfig['l10nRelbranch']       = None
-releaseConfig['l10nRepoPath']        = 'users/stage-ffxbld'
+releaseConfig['l10nRepoPath']        = 'users/'
 releaseConfig['l10nRevisionFile']    = 'l10n-changesets_mozilla-beta'
@@ -76,4 +76,4 @@ releaseConfig['l10nRevisionFile']    = '
 releaseConfig['otherReposToTag']     = {
-    'users/stage-ffxbld/compare-locales': 'RELEASE_AUTOMATION',
-    'users/stage-ffxbld/buildbot': 'production-0.8',
+    'users/': 'RELEASE_AUTOMATION',
+    'users/': 'production-0.8',
@@ -164,3 +164,3 @@ releaseConfig['bouncer_submitter_config'
 # Misc configuration
-releaseConfig['build_tools_repo_path'] = "users/stage-ffxbld/tools"
+releaseConfig['build_tools_repo_path'] = "users/"
 releaseConfig['enableAutomaticPushToMirrors'] = True


Linux update verify will not obey certain settings from To work around this, make a change like the following to scripts/release/updates/

diff --git a/scripts/release/updates/ b/scripts/release/updates/
--- a/scripts/release/updates/
+++ b/scripts/release/updates/
@@ -24,5 +24,5 @@ if $JSONTOOL -k properties.NO_BBCONFIG $
 if [ -z "$NO_BBCONFIG" -a -z "$BUILDBOT_CONFIGS" ]; then
-    export BUILDBOT_CONFIGS=""
+    export BUILDBOT_CONFIGS=""
 if [ -n "$PROPERTIES_FILE" -a -f "$PROPERTIES_FILE" ]; then

Setup Ship It

A proper staging release should be done through a staging instance of Ship It regardless of whether or not you're making changes to it. There's two parts to doing this, both of which can be run on dev-master01 and are described below.

The web app

First off, you'll need an instance of the ship it web application to submit your release to. It's recommended to run this on dev-master2. To get it running, start a screen or tmux session and run the following:

git clone
cd release-kickoff
virtualenv venv
venv/bin/pip install -r requirements/dev.txt -r requirements/compiled.txt
venv/bin/python -d sqlite:///kickoff.db -u shipit -p shipitgood --host= &> log &

If you get any errors about ports being in use you can use the "--port" option to choose a different one.

Once that's running, test it out by browsing to (or the port you specified) to make sure it's working. The default username/password are shipit:shipitgood (from the cmdline above)

Release runner

You'll also need release runner, which will process releases submitted to your ship it instance and kick them off on your master.

You should also make sure you have mq and rebase enabled in your hgrc, in case release runner needs either of them. Eg:


Your dev master also needs to be set-up correctly. Make sure the following is true:

  • <your_master>/Makefile should be a copy of buildbot-configs/Makefile.master, not a symlink. You need to comment out the call to in the update target (this will fail because your dev master won't appear in the production master list).
  • <your_master>/{buildbot-configs,buildbotcustom,tools} must be Mercurial repositories (NOT Git). Release runner will be pushing and pulling from them.

Release runner needs a few configuration files to function correctly, below are samples from bhearsum. Note that the json file is the same content as the one created by the buildbot master Makefile, but encased in "[]". Also note that the repo paths in release-runner.ini are _local_ paths, not http:// - this is important.

  • release-runner.ini:
username: shipit
password: shipitgood

notify_from: Release Eng <>
smtp_server: localhost
hg_ssh_key: dummy
buildbot_configs_branch: production
buildbotcustom_branch: production-0.8
tools_branch: default
masters_json: nthomas-master.json
staging: true
verbose: true
sleeptime: 30
ssh_username: nthomas

[symlinks] {buildbot_configs}/mozilla/ {buildbot_configs}/mozilla/
  • nthomas-master.json:
  "basedir": "/builds/buildbot/nthomas/build1",
  "bbconfigs_branch": "production",
  "bbconfigs_dir": "/builds/buildbot/nthomas/build1/buildbot-configs",
  "bbcustom_branch": "production-0.8",
  "bbcustom_dir": "/builds/buildbot/nthomas/build1/buildbotcustom",
  "buildbot_bin": "/builds/buildbot/nthomas/build1/bin/buildbot",
  "buildbot_branch": "production-0.8",
  "buildbot_python": "/builds/buildbot/nthomas/build1/bin/python",
  "buildbot_setup": "/builds/buildbot/nthomas/build1/buildbot/master/",
  "buildbot_version": "0.8.2",
  "datacentre": "scl1",
  "db_name": "",
  "enabled": true,
  "environment": "staging",
  "hostname": "",
  "http_port": 8710,
  "master_dir": "/builds/buildbot/nthomas/build1/master",
  "limit_branches": [],
  "limit_tb_branches": [],
  "limit_b2g_branches": [],
  "limit_projects": [],
  "limit_branch_projects": [],
  "mobile_release_branches": ["mozilla-beta"],
  "name": "nthomas",
  "pb_port": 9710,
  "release_branches": [
  "role": "build",
  "ssh_port": 7710,
  "thunderbird_release_branches": ["comm-esr38"],
  "tools_branch": "default",
  "tools_dir": "/builds/buildbot/nthomas/build1/tools",
  "queuedir": "/builds/buildbot/nthomas/common-queue"

You'll need to run release runner from somewhere with your SSH key in your keychain. Eg, run it from your laptop or forward the the key to dev-master2 If you're forwarding the key, you should connect to dev-master1 as follows:

ssh-add ~/.ssh/id_rsa
ssh -A

When you're ready, you can run release runner with the following commands:

hg clone
cd tools/buildfarm/release
# default branch is release runner for promotion, use the old branch
hg up old-release-runner
virtualenv venv
venv/bin/pip install buildbot==0.8.2 fabric simplejson
PATH=venv/bin:$PATH venv/bin/python -c /path/to/your/release-runner.ini &> log &

If everything is working correctly you should output similar to the following:

2013-03-04 05:54:24,047 - DEBUG - Fetching release requests
2013-03-04 05:54:24,168 - INFO - No new releases: {'releases': []}
2013-03-04 05:54:24,168 - DEBUG - Sleeping for 30 seconds before polling again

Start the release

Using your staging Ship It instance, submit a release! Make sure you use your own repository for "branch" instead of the production one.

Known Issues

Update verify fails

When mar signature verification is enabled on each platform update verify starts to break. This is because we use the dep cert in the staging release, but try to apply updates using the updater which expects the release cert. bug 997732 covers this.

Windows was broken from Firefox 30.0 (if not earlier), Mac from 40.0, Linux from 42.0.

To work around, manually trigger the "ready for release" builder after all update verify builders have completed.