User:Bhearsum/Release Automation Documentation

From MozillaWiki
Jump to: navigation, search

Notes and Definitions

  • 'Staging server' typically means stage.mozilla.org for production (read: real) releases and staging-stage.build.mozilla.org for staging (read: test) releases.
  • 'Candidates directory' refers to the directory on the staging server which builds are deposited into. It almost always resolves to /home/ftp/pub/$product/nightly/$version-candidates/build$buildNumber - resolve the variables appropriately. For example: /home/ftp/pub/firefox/nightly/3.1b3-candidates/build1.
  • Unless otherwise specified any references to directories on the staging server should be taken as relative to the candidates directory.

Overview

Mercurial Release Automation is a collection of Buildbot BuildFactory's which perform most of the tasks which are done as part of a release.

Code is stored in a few different places:

Of particular note is process/release.py, which instantiates all of the Schedulers, Builders, ChangeSources, and Status modules that releases use.

All steps are affected or fed in some way by the configuration files, but not all of them use code from the other locations. The subsequent sections contain links to the code that each step relies on.

Final Deliverables

The most important deliverables are the packages, installers, and updates. This includes (for each locale)

  • Linux tarball, Mac dmg, Windows installer (exe)
  • Complete and partial updates for all builds/locales
  • Update snippets for test and live channels

All of the above are handed off to and tested by QA during the release process.

Other, less important deliverables include:

  • Source tarball and bundle
  • Langpacks per platform per locale
  • Release branch and tags on appropriate repositories
  • Checksums for released files

Source packages are used by a small number of individuals and some Linux distributions to do their own builds of Firefox. Release branches and tags are very important to Release Automation and also used by developers when a release must be respun.

Individual steps may have additional deliverables which only serve to feed later steps. These will be discussed in the sections below.

Getting things going

Configuration

Where possible (and where it makes sense to), the release configuration reuses variables from the branches' dep/nightly config. These will not be discussed here. This MXR query will show you all of the places in the release automation set-up that branch variables are referenced.

The release configs themselves have a number of variables and options in them. These are discussed below.

sourceRepositories

This variable is information dense dictionary and mostly contains information only relevant to Tagging. Each key of sourceRepositories (eg. "mozilla"), has a value of a dict with the following structure:

  • name - The name of the source repository, without any leading directories
  • clonePath - (staging only) Where to clone this repository from in repo_setup
  • path - Where to clone this repository for everything else (tagging, builds, repacks, et. al)
  • revision - The revision to base this release on. If relbranch is None or the provided one does not exist, the automation will update to this revision before creating the relbranch.
  • relbranch - The name of the relbranch to use for this release. If set to None, it will be generated automatically. If provided and exists, the automation will update to it before doing any bumping or tagging. If provided and doesn't exist, it will be created.
  • bumpFiles - This is another dict, whose keys are the paths in the sourceRepo, and whose values are *another* dict, with the following structure.
    • version - The version number to use in this file on the relbranch
    • nextVersion - The version number to use in this file on the default branch

Here's a concrete example of sourceRepositories for a Firefox build. Note that the bump file versions are typically factored out to earlier variables to avoid repetition. 

releaseConfig['sourceRepositories'] = {
  'mozilla': {
    'name': 'mozilla-central',
    'path': 'mozilla-central',
    'revision': 'aaaaabbbbcccc',
    'relbranch': None
    'bumpFiles': {
      'browser/config/version.txt': {
        'version': releaseConfig['appVersion'],
        'nextVersion': releaseConfig['nextAppVersion'],
      },
      'config/milestone.txt': {
        'version': releaseConfig['milestone'],
        'nextVersion': releaseConfig['nextMilestone'],
      },
    }
  }
}

Others

Variable Description
hgUsername The account name used when pushing tags, version bumps, and other configuration file updates back to repositories hosting on hg.mozilla.org.
hgSshKey The path to the ssh key to used when pushing things back to hg.mozilla.org. The username in hgUsername must be setup to accept this key in order for the automation to work correctly. Path is generally in the ~/ form in order to be platform independent.
l10nRepoClonePath This is exactly like sourceRepoClonePath, but is the root of the l10n repository directory used for cloning locale repos in repo_setup.
l10nRepoPath This is exactly like sourceRepoPath, but for locale repositories. http://hgHost/l10nRepoPath/ab-CD should be a cloneable repository.
l10nRevisionFile This is the path, relative to the master's directory, which contains a listing of all of the locales to build, and their changesets. This file is read in at buildmaster start/reconfig time and the information is passed along to ReleaseTaggingFactory, which tags all of the repositories for this release. Any subsequent steps which require a list of locales will pull them from shipped-locales. This is a bug that may or may not be fixed.
l10nRelbranch The relbranch to use for l10n repositories. See "relbranch" in the preceding section for mechanics. Generally should be set to the same thing as the sourceRepositories relbranches.
shippedLocalesPath The path, relative to the root of the sourceRepo, to shipped-locales.
mergeLocales When True, merging will be enable when running compare-locales.
l10nChunks The number of l10n builders to be used, per platform. If not present, defaults to DEFAULT_L10N_CHUNKS set in process/release.py.
cvsroot The cvsroot to use for pulling MozBuild, patcher tools, and other things from CVS. It is also used to check in an update patcher config file and as such must be an account with write access.
productName The lowercase product brand name. Eg, 'firefox', 'thunderbird', etc.
appName The lowercase application name. Eg, 'browser', 'mailnews', etc.
binaryName The name which will be used in the filenames of installers, packages, and MARs. Usually is the same as productName.
oldBinaryName Same as above, but for the oldVersion.
version/appVersion Sometimes we need the application version to be different from what we "call" the build, eg public release candidates for a major release (3.1 RC1). appVersion and oldAppVersion are optional definitions used in places that# don't care about what we call it. Eg, when version bumping we will bump to appVersion, not version. As a concrete example, when we produce the first release candidate for 3.1, version will be '3.1rc1' and appVersion will be '3.1'.
milestone The milestone version of the current platform release. Eg, 1.9.1b2
nextAppVersion/nextMilestone These variables control what the appVersion and milestone will be bumped to on the "default" branch.
buildNumber The build number of the current release. 1 if it's the first try for this release. May be 2, 3, or higher if respins were required.
baseTag The base tag to use when tagging repositories. _BUILD$buildNumber and _RELEASE will be appending to this where necessary.
oldVersion/oldAppVersion Same thing as version/appVersion, but applies to the previous release. When 3.1rc1 is shipped oldVersion and oldAppVersion will both be 3.1b3. When we ship 3.1.1 oldVersion will be, eg, 3.1rc2 and oldAppVersion will be 3.1.
oldBuildNumber The build number of the previous release. This should be the build number which was actually shipped.
oldBaseTag Same as baseTag, but for the previous release.
enUSPlatforms This is a tuple containing all of the platforms that require an en-US build as part of the release. These platforms must exist in config.py for the branch being built.
l10nPlatforms A tuple containing all of the platforms that require l10n repacks. Any platform listed here must also be listed in enUSPlatforms.
talosTestPlatforms A tuple containing all of the platforms that require talos test runs to be requested. Platforms listed here must also be listed in enUSPlatforms.
unittestPlatforms A tuple containing all of the platforms that require unit test runs to be requested. Platforms listed here must also be listed in enUSPlatforms.
xulrunnerPlatforms A tuple containing all of the platforms that require XULRunner builds to be generated.
patcherConfig The name of the patcher config file to bump and pass to patcher2.pl.
patcherToolsTag The tag to update both the tools and source repository to before running patcher2.pl.
ftpServer The ftp server to use for 'beta' channel snippets when useBetaChannel is '1'. Regardless of what useBetaChannel is set to this must be passed.
stagingServer The stage server to use for 'betatest' snippets and most verification tests.
bouncerServer The bouncer server to use for 'release' and 'releasetest' channel snippets.
ausServerUrl The AUS URL to aus for update verify tests.
ausUser The username to use when uploading snippets to the AUS server
ausSshKey The SSH key to use when uploading snippets to the AUS server
releaseNotesUrl URL of release notes. If None use the default URL. Introduced in bug 553059.
useBetaChannel This variable controls which channel is used as the final release channel. When 0, 'beta' channel snippets will be considered the final release channel, and will point to mirrors. In this case, 'release' channel snippets will NOT be generated. When 1, 'release' channel snippets will be generated and will be considered the channel for final release. In this case, beta snippets will still be generated and will point to ftpServer.
*VerifyConfig The configuration file names for the update verify configuration files for each platform.
majorUpdateRepoPath The relative path repository that contains the shipped-locales file for the major update "to" release. If None, the rest of the majorUpdate* variables are not necessary.
majorUpdateToVersion, majorUpdateAppVersion, majorUpdateBuildNumber, majorUpdateBaseTag These variables contain information about the release you are generating a major update to. See version/appVersion/buildNumber/baseTag for descriptions of them.
majorUpdateReleaseNotesUrl Major Update release notes URL. If None use the default URL. Introduced in bug 553059.
majorUpdatePatcherConfig The Patcher config file to use for the major update. This can be an existing file, in which case it will be overwritten and committed, or a non-existant file, in which case it will be created and committed.
majorUpdateVerifyConfigs See updateVerifyConfigs description above.
testOlderPartials Applicable only to the "updates" step. When true, the update verify bumping step will mark the n-2 release to have both its partial and complete updates tested. When false, it will rely on the default behaviour of update verify (As of bug 514040, this is "test complete only for n-2 and earlier builds). Generally, should be True on 1.9.1 based releases and earlier and False on anything more recent.
doPartnerRepacks Whether or not to create and schedule partner repack builds as part of this release.
partnersRepoPath The repoPath of the repository containing the partner repack scripts and configs. Generally set to 'build/partner-repacks'. Only necessary when doPartnerRepacks is True.
AllRecipients List of email addresses to send step-by-step email notification of release progress, Defaults to release@
PassRecipients List of email addresses to send step-by-step email notification of release progress if and only if the step is successful, Defaults to release@, although we may want to send directly to r-d@
ReleaseTemplates Location of the message bodies to use to formulate the messages sent for the notifications
tuxedoConfig A configuration file of tuxedo-add.py, which contains FTP path templates for the deliverables (installers and MARs). Examples: stable version, alpha (devpreview) version.
tuxedoServerUrl Tuxedo server's API URL prefix. Trailing slash is important.

Preparation

There are a couple of things you can do prior to the "go to build" to hasten things when you do receive. Generally, we do the following ahead of time:

  • Post updated configs and any other patches you'll need for review, using 'FILLMEOUT' or similar as the revision.
  • Mark a clobberer for "Any master", $branch, "Any builder" with clobberer. This will instruct any slave that did the last release on $branch to clobberer their release directories.

If you are doing a test release in staging there is additional setup work to do, and the clobberer URL changes.

Starting the automation

Generally, the workflow for kicking off Release Automation is as follows:

  • Update the configuration files
  • Tag buildbotcustom, buildbot-configs, and tools with the correct _RELEASE and _BUILDn tags.
  • Reconfig the Buildbot Master doing the release
  • Run the pre-flight sanity check. This is done in the master directory (watch out for PYTHONPATH). For example:
python release_sanity.py -u bhearsum -V 3.6.12 --branch mozilla-1.9.2 --build-number 1 --release-config release-firefox-mozilla-1.9.2.py --dryrun localhost:9010
  • Start the automation by running the sanity script again, without --dryrun. For example:
python release_sanity.py -u bhearsum -V 3.6.12 --branch mozilla-1.9.2 --build-number 1 --release-config release-firefox-mozilla-1.9.2.py localhost:9010

If you're working in staging you must make sure to pass in the correct staging release config (staging_release-firefox-<branch name>.py)

If for some reason you need to perform the sendchange manually, here's a sample. Note that the branch name is the repo path to the source repository. In staging, this will be a user repo. All parameters are required:

buildbot sendchange --username bhearsum --master localhost:9010 --branch releases/mozilla-1.9.2 -p script_repo_revision:FIREFOX_3_6_12_RELEASE -p release_config:mozilla/release-firefox-mozilla-1.9.2.py release_build

Starting Signing

The next step is to start the signing automation, which will download builds as they become available, and start the signing process once all builds are ready. For more details on starting the signing automation, see the Combined Signing page.

Steps

Tag

Code used:

Deliverables:

  • Release branch and tags on source and l10n repositories

This steps creates an in-repo branch and tags in both source and locale repositories. For example, if you started Release Automation for Firefox 3.1b3 on January 17th, 2009 (building out of the mozilla-1.9.1 repository) you would end with the following in mozilla-1.9.1 and all appropriate l10n repositories:

  • GECKO191_20090117_RELBRANCH named branch
  • FIREFOX_3_1b3_BUILD1 tag
  • FIREFOX_3_1b3_RELEASE tag

For source repositories, after creating the branch Release Automation will perform a version bump to ensure that the release will be versioned correctly. After commiting this, it will be tagged with both of the FIREFOX_* tags and pushed back from whence it came.

For l10n repositories the exact revisions give in l10n-changesets are used as the branch point *and* tagged with the FIREFOX_* tags. They do not require a version bump.

Source / XULRunner Source

Code:

Deliverables:

  • Source bundle and tarball uploaded to the staging server.

Creates a ready-to-be-built tarball and ready-to-be-used Mercurial bundle of the sourceRepo. These files will be uploaded to the candidates directory in the 'source' subdirectory.

Build

Code:

Deliverables:

  • en-US build for each platform (tarball for Linux, dmg for Mac, exe and zip for Windows) uploaded to the staging server
  • en-US complete MAR files for each platform uploaded to the staging server
  • *_info.txt files containing BuildID uploaded to the staging server
  • .checksums files containing sha512 sums of all files uploaded
  • debugging symbols uploaded to the symbol server

This step happens per item in enUSPlatforms and follows the nightly build process very closely. Notable differences are:

  • Updates to the _RELEASE tag prior to building
  • Does not run codesighs
  • Does not generate update snippets
  • Uses a different mozconfig

XULRunner Build

Code:

Deliverables:

  • XULRunner runtime (tarball for Linux, dmg for Mac, zip for Windows) and SDK per platform, uploaded to the staging server

This step happens per item in xulrunnerPlatforms and follows the nightly XULRunner build process almost identically. Notable differences are:

  • Updates to the _RELEASE tag prior to building
  • Uses a different mozconfig

L10n Repack

Code:

Deliverables:

  • A build for each platform/locale combination
  • A complete MAR file for each platform/locale combination
  • A langpack (.xpi) file for each platform/locale combination
  • .checksums file containing sha512 sums for all uploaded files

Repacks are done across a set of Builders per platform. The number of Builders per platform is controlled by the 'l10nChunks' release config variable, with a default set as DEFAULT_L10N_CHUNKS in process/release.py. Each Builder performs one Build; each Build repacks (totalLocales / l10nChunks) locales.

Standalone Repack Builders

We sometimes run into the case where individual locales need to be re-spun. The easiest way to do this without re-spinning the entire chunk that failed is to use the standalone repack builder through the 'force build' form. You will need to enter the following fields and properties for the builder to succeed:

  • branch
  • revision (you should use the _RELEASE tag here)
  • properties:
    • locale (A colon separated list of locales to respin
    • release_config (usually mozilla/release-mozilla-firefox-<branch name>.py)
    • script_repo_revision (you should also use the _RELEASE tag here)

Partner Repack

Code:

Deliverables:

  • A full set of unsigned partner repacks, uploaded to the staging server

This step is only done as part of releases on the latest stable branch. It uses unsigned en-US and l10n builds to create partner repacks.

Signing

Code used:

Deliverables:

  • Signed Windows builds (internal binaries and installers), including partner repacks (where applicable)
  • Detached GPG signatures for all builds and source packages
  • MD5SUMS and SHA1SUMS files containing checksums for all builds, source packages, langpacks, and some MARs

This is a semi-automated process. The instructions for initiating and finishing the process are documented on the Combined Signing page.

The final step of signing is uploading the signing log. An FtpPoller watches for it to appear on the ftp server and triggers the appropriate builders when it appears.

L10n Verify

Code:

Deliverables:

  • l10n verify logs, per platform

As the name implies this is strictly a verification step. For both the current and previous release it compares the strings in the localized build with the strings in the en-US build. This diff is dumped to a file. Once this is done for every locale for the current and previous release a metadiff of $locale.diff for the current and previous release is done. If that diff has no output, no strings have changed.

Log Analysis

There are a few common things to watch for:

  • In the "compare current/previous"
    • Anything about being unable to unpack or mount a package
      • Usually indicates a corrupt package
      • On Mac, can mean that '7z' is not installed or not in PATH.
  • In the "Only in..." log:
    • Locale(s) only in the previous release:
      • Locale may have failed to build (make sure the locale exists in the candidates directory)
      • Locale may have been removed in this release (compare shipped-locales from previous and current release)
    • Locale(s) only in the current release:
      • Locale newly added in this release (compare shipped-locales from previous and current release)

The metadiff is a bit trickier and less clean cut. In a typical stable release (eg. 3.1.3) there are no string changes, the l10n metadiff log should be empty. During beta cycles there are lots of string changes, and therefore lots of output in the metadiff log. Generally, no analysis is done on the metadiff for Betas, because of the high rate of change.

Update/snippet Generation

Code used:

Deliverables:

  • Updated patcher config file, checked into CVS
  • Partial mar file per-platform per-locale
  • Update snippets on test & live channels
  • Snippet comparison log

After checking out the necessary modules and tools this builder goes to work doing the following:

  • Bumping & checking in a patcher config file
  • Cloning sourceRepo & building update-packaging tools
  • Creating & uploading partial MARs
  • Creating & uploading update snippets
  • When applicable, creating snippets for previous builds of the release
  • Pushing test snippets live
  • When applicable, doing a comparison of live channel snippets vs. releasetest

This step is generally pretty failsafe but in the event that it needs to be run a second time or restarted at later step it's necessary to comment out the patcher config bump step (see below for details on how). Until bug 466999 FIXED is resolved the patcher config file can only be bumped once.

Note that both the sourceRepo and the tools repo will be updated to the patcherToolsTag. If the update-packaging tools or patcher config script has been updated since the last release it is a good idea (and sometimes necessary) to create a new UPDATE_PACKAGING_R# tag before starting the release.

Snippet Comparison Log Analysis

Once updates is completed there will be a log from the snippet comparison step. This step compares the contents of the releasetest channel snippets to the contents of the live channel (beta or release) snippets. There's a few things to note about this log:

  • It will usually contain warnings about missing snippets for alphas and betas. These are expected and ignorable.
  • When there are warnings about other snippets missing, a 'Warnings' log will be created and the step will be orange. These must be investigated and explained/fixed before any live snippets are pushed.
  • If any snippets differ in contents between the test and live channels a 'Diffs' log will be created and the step will be orange. The Diffs log will list the files which differ; the full diffs can be viewed in the stdio log. Any differences must be investigated and explained/fixed before any live snippets are pushed.

Update Verify

Code used:

Deliverables:

  • Update verify logs per platform. Details on them are below.

This step verifies both the MAR files and does quick tests on the update snippets. Firstly, it tests both MAR files by applying them against the previous version, and then comparing the result against the current version's full package. Once this is done for every locale it then ensures that an update snippet is present for older versions (that is, older than the previous version), and that the complete MAR the snippet points to is downloadable. Note that it does NOT attempt to apply it.


This step takes approximately 15 minutes on Linux, 45 minutes on Mac, and 30 minutes on Windows.

Log Analysis

This step is pretty good at reporting errors. Most errors will turn the step red, and be pretty clearly visible in the log. Nonetheless it's a good idea to have a scan through the log for anything abnormal (searching for FAIL and WARN are helpful here).

Here's some common failures:

  • "Binary files source/bin/freebl3.chk and target/bin/freebl3.chk differ" (or softokn3.chk) on win32 in the complete MAR
    • This is normal in the win32 logs and is caused by bug 404340. Once this bug is fixed it should go away.
  • "CRC check failed" and/or "LoadSourceFile failed" on a partial MAR
    • This typically happens when the previous release's complete MAR differs from the complete package. Because partial MARs are generated by diffing the previous complete MAR with the current complete MAR the partial will fail the CRC check when this happens. There's nothing that can be done about this, because the problem is with an already-release-build. The correct thing to do here is inform release-drivers that there will not be partials available on this platform, trim them out of the candidates directory, and remove the partial snippets.
  • "FAIL: partial from xxxxx wrong size", "FAIL: actual size: 0"
    • This is usually caused by missing MARs. You should first check to see if the MAR in question exists in the candidates directory. If it doesn't, go back and look at the logs to see if there was a problem generating or uploading it. If it does exist in the candidates directory make sure the permissions are set properly (ffxbld:firefox, 664).
  • "FAIL: Could not download source xxx"
    • Usually indicates missing packages or bad permissions in the candidates directory. Take the same action as above.
  • "FAIL: no partial update found for xxx" or "FAIL: no complete update found for xxx"
    • Check the logs from the previous step to make sure 'pushsnip' was run successfully. This error usually indicates that snippets weren't pushed.

Major Update

Code:

As part of any release that is on a less than current branch we generally create and push a Major Update as part of it. For example, when we did a 3.5.8 release we generated a major update for 3.5.8 -> 3.6 alongside it. Note that major updates happen as part of the "from" release, not the "to" release.

Major update generation is not kicked off automatically because it depends on both the "to" and "from" releases having had their builds and repacks signed. Once they are at that point "Force Build" can be used to initiate the Major Update generation.

If either of the "from" or "to" releases respin at any point you must regenerate the major update. If the "to" release respins, majorUpdateBuildNumber needs to be updated.

At the end of the factory snippets will be uploaded to the AUS server and the test snippets will be pushed live. After that, the major update verify builders will be triggered.

Major Update Verify

Everything in the previous "update verify" section applies here.

Generally, major updates generate more harmless warnings than other releases. Analysis on these is done when testing a new major update and noted on release notes. When verifying a major update it's always helpful to look at the previous one's notes to find a list of the harmless warnings. If you find warnings that are NOT in the previous major update's notes they must be investigated.

It's rare to see major update verify builders go green because of all of the harmless warnings.

Pre Push to Mirrors

Code: scripts/release/push-to-mirrors.py

Check Permissions

This builder:

  1. Checks if the FTP directories and files have proper permissions and modes
  2. Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

This builder should be run second time before pushing file to mirrors by hitting "Rebuild" button or forcing the builder with 2 properties set:

  • Property 1 name: script_repo_revision: use FIREFOX_*_RELEASE tag here
  • Property 2 name -> release_config: mozilla/release-firefox-mozilla-2.0.py

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

Antivirus check

This builder:

  1. Runs antivirus
  2. Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

This builder should be run second time before pushing files to mirrors by hitting "Rebuild" button or forcing the builder with 2 properties set:

  • Property 1 name: script_repo_revision use FIREFOX_*_RELEASE tag here
  • Property 2 name -> release_config: mozilla/release-firefox-mozilla-2.0.py

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

Push to Mirrors

Code: scripts/release/push-to-mirrors.py

This builder:

  1. Runs rsync and pushes files to the release directory
  2. Triggers mirror uptake monitoring poller.

To trigger this builder you need to specify some build properties:

  • Property 1 name: script_repo_revision use FIREFOX_*_RELEASE tag here
  • Property 2 name -> release_config: mozilla/release-firefox-mozilla-2.0.py

Press 'Force Build' in the Buildbot waterfall.

No reconfig needed for this builder.

Final Verification

Code:

Once mirror uptake has reached 50 or 60% this builder can be started with 'Force Build' from the Buildbot waterfall.

This is a simple test which is run after mirrors are propagated to ensure there's no (major) problems with them. It does the same thing as the second part of Update Verify: for every locale/platform/version combination it checks for an update snippet, and then performs an HTTP HEAD request on the MAR the snippet points to.

The step will turn red if there are http codes other than 200 or 302, or if FAIL is found in the log. If it does this, you should have a look in the log, and find out which mirror is failing. It's a good idea to keep an eye on that mirror for awhile to make sure it finishes getting all of the files.

Searching the log for errors is made easier by downloading it and running the following locally: 

grep HTTP/1 text | grep -v 302 | grep -v 200
grep FAIL text

Bouncer Submitter

Code:

Configuration:

  • firefox-tuxedo.ini for stable versions, and firefox-devpreview-tuxedo.ini for Alpha releases.
  • BuildSlaves.py file should contain the following variables:
    • tuxedoUsername: Tuxedo username with appropriate permissions for adding new entries via API
    • tuxedoPassword: password for this user

This builder can be started any time with 'Force Build' from the Buildbot waterfall.

This step adds adds bouncer entries for installers, complete and partial updates for all platforms listed in enUSPlatforms and l10nPlatforms and locales specified in shipped-locales (en-US is being added in any case).

The step will turn red if tuxedo-add.py exits non zero. There are 2 major error sources:

  • tuxedoUsername has not enough privileges to add entries
  • Tuxedo is not aware of new locales added in the current version. Usually the log contains "HTTPError: HTTP Error 400: BAD REQUEST". See this link for an example workflow on how to resolve this problem.

Common Problems & Resolutions

Restarting the automation from a specific point

The automation has the ability to easily skip tagging, source generation, or en-US builds in the event that you want to restart from source, en-US, or l10n repacks. To make use of this, you need to add skip_$builder flags to the release config. For example, to restart from l10n repacks you will need to add the following lines to the release config: 

releaseConfig['skip_tag'] = 1
releaseConfig['skip_source'] = 1
releaseConfig['skip_build'] = 1

Only these three steps (and repo_setup, if you're in staging) are skippable. If you need to restart in later parts of the automation, "Force Build" is the supported method of doing so.

Restarting failed builders without patching the config

It's often easier to use Force Build or buildbot sendchange to restart specific things than it is to patch the configs, check in, update the master, etc. If you need to restart tag, source, an en-US build, l10nverify, updates, or update_verify you can use "Force Build" from the Buildbot waterfall.

If you need to restart a single l10n build (that is, one locale) on a single platform you may also use "Force Build", but be sure to use en_revision, l10n_revision, and locale as properties. The revision properties should be set to _RELEASE tags, and the locale should be obvious.

If you need to restart many or all locales you'll have to fill out the web interface multiple times, until bug 518589 is resolved.

Restarting a builder from a certain point

In some cases only part of a Builder will successfully complete. For example, if the 'updates' builder was able to build partial MARs successfully but failed to upload them you will want to figure out what the problem is, and then restart it from the upload step. To do that, open up /tools/buildbotcustom/buildbotcustom/process/factory.py and comment out all steps before uploading. You should also find out which slave it was started on, and update the 'slavenames' for 'updates' in release_master.py to ensure the job gets started on the same slave.

Depending on how much of a builder is left to run it can be faster and/or easier to simply run the steps manually and continue on. For example, if backupsnip or pushsnip in 'updates' failed, it's probably easier to run those steps manually rather than go to the trouble of starting another job.

In other cases, it may make sense just to start from the beginning. Eg, if the build tools repository fails to clone, it's generally easier just to start from scratch.

Use your judgment here.

Tagging failed out part way through

If this happens, you'll need to continue it on but get it to skip locales which it has already tagged. To do so, delete any locales which have been tagged from l10n-changesets and reconfig. If the source repository has already been tagged you should pass 'l10n_repositories' to ReleaseTaggingFactory instead of 'repositories. Note that the rest of the release automation uses shipped-locales so removing things from l10n-changesets doesn't cause locales not to build.

Re-spinning a single locale

We sometimes run into the case where individual locales need to be re-spun. Some reasons this might happen include network timeouts or build slave failures. When this happens, you can follow the following steps to recover and re-spin the missing locale(s):

  1. Manually re-tag the locale's repository (if necessary). You can find the appropriate revision to use for tagging from the shipped-locales file, or from the l10n shipping dashboard
  2. Delete the current build of the locale and cleanup l10n build dirs on build slaves.
  3. Manually force the repack on each of the "$platform_standalone_repack" builders. See the Standalone_Repack_Builders section for details.
  4. Manually sign it and update the *SUMS files
    • You need to download the new locale builds to the signing machine, but you also need the SUM files, en-US Windows build (used for caching) and the zh-TW builds (monitoring tools check for this locale to know when the directory has changed).
    • Run sign-files on the new builds.
    • Manually update the SUMS files with new md5/sha1 sums.
    • Remove the .asc file for the en-US Windows build
    • Push the signed builds back to stage.
    • The Build Notes for 3.6b4 show an example of how this is done.
  5. Manually create a partial MAR for the locale
    • use an appropriate patcher_config file for your release.
    • on a linux slave (preferably a fast one), download the builds with patcher2.pl
    • use patcher2.pl to create the update MAR files and snippets.
    • ensure file (755) and directory (644) modes are correct for your created files.
    • transfer the MAR files to stage
    • transfer the update snippets to the aus2 server(s) <- there may be more than one
      • it is good practice to use a new directory name on the aus2 server to mark the new snippets as part of a distinct respin, e.g. 20091125-Firefox-3.6b4-fr-respin-test. Please also add that new directory name to the list of directories to be run through backupsnip/pushnip in the build notes.
    • The Build Notes for 3.6b4 show an example of how this is done.
  6. Re-run the update verify builder from the waterfall.

Staging Specific Notes

Release automation in staging is mostly the same as in production, but does have a few differences you should known about:

  • All uploading (builds, snippets, symbols) is done to staging-stage.build.mozilla.org
  • It can take a few tries to get repo_setup to run properly. This is because hgweb sometimes returns a 500 (internal server error) when we query about a locale. The best solution is just to start the automation from scratch until it works, to make sure you get a clean run. If this is too frustrating for you, you can manually clone the repositories you care about and start automation from tag (see Restarting from a specific point).
  • Staging doesn't have a lot of slaves, and you may need to go around and stop running builds to get your automation run to happen in a reasonable period of time.

Staging specific preparation

Download the previous release

Because we point the staging releases at staging-stage.build.mozilla.org the previous release must be downloaded to it. This is done by the "release_downloader" builder which is fired by a sendchange sent by release_sanity.py. It automatically removes any candidates or relase dir on staging-stage which already exists.

The builder could be disabled by setting releaseConfig['skip_release_download'] = True

Buildbot configs

When working in staging you'll need to swap out the buildbot-configs, build/tools, buildbotcustom, and compare-locales repos for either stage-ffxbld ones, or your own. Here's an example of the changes you'll need: 

diff --git a/mozilla/config.py b/mozilla/config.py
--- a/mozilla/config.py
+++ b/mozilla/config.py
@@ -32,17 +32,17 @@ GLOBAL_VARS = {
-    'compare_locales_repo_path': 'build/compare-locales',
+    'compare_locales_repo_path': 'users/bhearsum_mozilla.com/compare-locales',
diff --git a/mozilla/staging_config.py b/mozilla/staging_config.py
--- a/mozilla/staging_config.py
+++ b/mozilla/staging_config.py
@@ -25,24 +25,24 @@ TRY_SLAVES = SLAVES
-    'config_repo_path': 'build/buildbot-configs',
-    'buildbotcustom_repo_path': 'build/buildbotcustom',
+    'config_repo_path': 'users/bhearsum_mozilla.com/buildbot-configs',
+    'buildbotcustom_repo_path': 'users/bhearsum_mozilla.com/buildbotcustom',
-    'build_tools_repo_path': 'users/stage-ffxbld/tools',
+    'build_tools_repo_path': 'users/bhearsum_mozilla.com/tools',
--- a/mozilla/staging_release-firefox-mozilla-1.9.2.py
+++ b/mozilla/staging_release-firefox-mozilla-1.9.2.py
-releaseConfig['partnersRepoPath']    = 'build/partner-repacks'
+releaseConfig['partnersRepoPath']    = 'users/armenzg_mozilla.com/partner-repacks'

As with production releases, you must tag these repositories with the _RELEASE tag prior to starting the release.


It is possible to use your own HG repository (long term release tests, parallel run, etc). Instead of running repo_setup builder, you can use the following script (run it on your laptop, using LDAP credentials): 

#!/bin/bash

for repo in compare-locales tools buildbotcustom buildbot buildbot-configs partner-repacks; do
    echo "deleting $repo"
    ssh hg.mozilla.org edit $repo delete YES
    echo "cloning $repo"
    ssh hg.mozilla.org clone $repo build/$repo
done
 
for l in `wget -q -O- http://hg.mozilla.org/mozilla-central/raw-file/tip/browser/locales/shipped-locales |grep -v en-US`; do
    echo "deleting locale $l"
    ssh hg.mozilla.org edit $l delete YES
    echo "cloning locale $l"
    ssh hg.mozilla.org clone $l l10n-central/$l
done

ssh hg.mozilla.org edit mozilla-central delete YES
ssh hg.mozilla.org clone mozilla-central mozilla-central

Doing a test run with a limited number of locales

To run a test with a limited number of locales, do the following:

  1. Modify l10n-changesets to include whichever locales you want.
  2. Reconfig Buildbot and start the automation as normal
  3. Once the tag builder is finished and before the end of the first en-US build, clone the staging source repository (eg, http://hg.mozilla.org/users/stage-ffxbld/mozilla-1.9.1) do the following:
hg up -C GECKO191_20090115_RELBRANCH # replace the relbranch appropriately, of course
# modify browser/locales/shipped-locales to include the same locales as l10n-changesets
hg commit -m "Reduce number of locales for this test run"
hg tag -f FIREFOX_3_1b3_RELEASE # using the right version number in the tag
hg push ssh://hg.mozilla.org/users/stage-ffxbld/mozilla-1.9.1

If you don't want to clone the full repo you can do this on the slave that did the tagging.

How to sign in staging

Signing with the staging keys

When you're specifically testing something related to signing or doing a full end to end run it's best to sign the builds on the staging signing server. Doing so is very similar to signing production builds and fully documented in the CombinedSigning doc.

Faking it out

If you're not looking to test signing you can speed up your staging run a bit by shuffling files around so post-signing steps can find them. To do this, log onto staging-stage.build.mozilla.org and do the following: 

# ffxbld@staging-stage
VERSION=3.5rc1
BUILD=1
cd /home/ftp/pub/firefox/nightly/${VERSION}-candidates/build${BUILD}
mkdir win32 update/win32
rsync -av --exclude=*.zip unsigned/win32/ win32/
rsync -av unsigned/update/win32/ update/win32/
rsync -av unsigned/win32_info.txt .
echo "faked" > win32_signing_build${BUILD}.log

We purposely make copies here rather than symlinking for a couple of reasons: L10n verify scripts barf when they get zip files (hence the --exclude above), 'updates' factory will blow away complete MARs upon upload if update/win32 is a symlink. The echo creates the log the automation looks for, in order to continue to l10n verify and updates.

Creating a CVS mirror for patcher and configs

If you need a new or modified patcher config, which shouldn't be checked into production CVS, then modify the following method: 

WHO=yournamehere
# cltbld@staging-stage
cd /builds/cvsmirrors
mkdir -p ${WHO}/cvsroot.clean/mozilla/tools/
rsync -av --exclude=CVSROOT/config --exclude=CVSROOT/loginfo cvs-mirror.mozilla.org::mozilla/CVSROOT /builds/cvsmirrors/${WHO}/cvsroot.clean/
rsync -av cvs-mirror.mozilla.org::mozilla/mozilla/tools/patcher-configs /builds/cvsmirrors/${WHO}/cvsroot.clean/mozilla/tools/
rsync -av cvs-mirror.mozilla.org::mozilla/mozilla/tools/patcher /builds/cvsmirrors/${WHO}/cvsroot.clean/mozilla/tools/
rsync -av cvs-mirror.mozilla.org::mozilla/mozilla/tools/release /builds/cvsmirrors/${WHO}/cvsroot.clean/mozilla/tools/
rsync -a --delete-after /builds/cvsmirrors/${WHO}/cvsroot{.clean,}/

To make changes check out using 

cvs -d staging-stage.build.mozilla.org:/builds/cvsmirrors/${WHO}/cvsroot co mozilla/
and specify a cvsroot of :ext:cltbld@staging-stage.build.mozilla.org:/builds/cvsmirrors/${WHO}/cvsroot in the config for the release automation.

Retrieved from "https://wiki.mozilla.org/index.php?title=User:Bhearsum/Release_Automation_Documentation&oldid=286953"