Release:Release Automation on Mercurial:Updates through Shipping: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(clarify manual restart operations)
Line 209: Line 209:
* [https://play.google.com/store/apps/details?id=org.mozilla.firefox_beta Beta]
* [https://play.google.com/store/apps/details?id=org.mozilla.firefox_beta Beta]
* [https://play.google.com/store/apps/details?id=org.mozilla.firefox GA]
* [https://play.google.com/store/apps/details?id=org.mozilla.firefox GA]
It usually takes about 75 minutes for the new product to appear as available in Google Play.


= Push snippets =
= Push snippets =

Revision as of 17:21, 14 August 2012

<< Documentation

Update/snippet Generation

Code used:

Deliverables:

  • Updated patcher config file, checked into CVS
  • Partial mar file per-platform per-locale
  • Update snippets on test & live channels
  • Snippet comparison log

After checking out the necessary modules and tools this builder goes to work doing the following:

  • Bumping & checking in a patcher config file
  • Cloning sourceRepo & building update-packaging tools
  • Creating & uploading partial MARs
  • Creating & uploading update snippets
  • When applicable, creating snippets for previous builds of the release
  • Pushing test snippets live
  • When applicable, doing a comparison of live channel snippets vs. releasetest

This step is generally pretty failsafe but in the event that it needs to be run a second time or restarted at later step it's necessary to comment out the patcher config bump step (see below for details on how). Until bug 466999 FIXED is resolved the patcher config file can only be bumped once.

Note that both the sourceRepo and the tools repo will be updated to the patcherToolsTag. If the update-packaging tools or patcher config script has been updated since the last release it is a good idea (and sometimes necessary) to create a new UPDATE_PACKAGING_R# tag before starting the release. ReleaseEngineering/PatcherTags documents the tag history, please keep it up to date.

Snippet Comparison Log Analysis

Once updates is completed there will be a log from the snippet comparison step. This step compares the contents of the releasetest channel snippets to the contents of the live channel (beta or release) snippets. There's a few things to note about this log:

  • It will usually contain warnings about missing snippets for alphas and betas. These are expected and ignorable.
  • When there are warnings about other snippets missing, a 'Warnings' log will be created and the step will be orange. These must be investigated and explained/fixed before any live snippets are pushed.
  • If any snippets differ in contents between the test and live channels a 'Diffs' log will be created and the step will be orange. The Diffs log will list the files which differ; the full diffs can be viewed in the stdio log. Any differences must be investigated and explained/fixed before any live snippets are pushed.

Update Verify

Code used:

Deliverables:

  • Update verify logs per platform. Details on them are below.

Update verify, in its current form, verifies a few different things:

  • That the "updater" binary works
  • That all MAR files can be applied, and result in a functionally equivalent build to the installer.
  • That all previous versions (across all locales), have snippets available

It does this in two different types of tests:

  • Full update verify unpacks a previous version's installer, pings AUS, downloads the MAR it finds, applies it, and diffs the result against the current version's installer. Differences between the two are reported in the log.
    • Run for all locales in the oldVersion
    • Run for select locales ("de", "en-US", "ru" at the time of writing) for all other previous versions
    • Completes and partials (if present) are both tested
  • Quick update verify does a subset of the above: it pings AUS, downloads the MAR it finds, and ensures that it is non-zero in size
    • Run against previous versions for all locales not tested as part of the full update verify.

Update verify is split into multiple chunks per platform (10, by default). The full and quick checks are spaced evenly throughout the chunks giving each one roughly equivalent run time.

Log Analysis

This step is pretty good at reporting errors. Most errors will turn the step red, and be pretty clearly visible in the log. Nonetheless it's a good idea to have a scan through the log for anything abnormal (searching for FAIL and WARN are helpful here).

Here's some common failures:

  • "Binary files source/bin/freebl3.chk and target/bin/freebl3.chk differ" (or softokn3.chk) on win32 in the complete MAR
    • This is normal in the win32 logs and is caused by bug 404340. Once this bug is fixed it should go away.
  • "CRC check failed" and/or "LoadSourceFile failed" on a partial MAR
    • This typically happens when the previous release's complete MAR differs from the complete package. Because partial MARs are generated by diffing the previous complete MAR with the current complete MAR the partial will fail the CRC check when this happens. There's nothing that can be done about this, because the problem is with an already-release-build. The correct thing to do here is inform release-drivers that there will not be partials available on this platform, trim them out of the candidates directory, and remove the partial snippets.
  • "FAIL: partial from xxxxx wrong size", "FAIL: actual size: 0"
    • This is usually caused by missing MARs. You should first check to see if the MAR in question exists in the candidates directory. If it doesn't, go back and look at the logs to see if there was a problem generating or uploading it. If it does exist in the candidates directory make sure the permissions are set properly (ffxbld:firefox, 664).
  • "FAIL: Could not download source xxx"
    • Usually indicates missing packages or bad permissions in the candidates directory. Take the same action as above.
  • "FAIL: no partial update found for xxx" or "FAIL: no complete update found for xxx"
    • Check the logs from the previous step to make sure 'pushsnip' was run successfully. This error usually indicates that snippets weren't pushed.

Check Permissions

This builder:

  1. Checks if the FTP directories and files have proper permissions and modes
  2. Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

Antivirus check

This builder:

  1. Runs antivirus locally ← report these results in build notes
  2. Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

If a long time has passed since the first run, this builder should be run a second time before pushing files to mirrors by hitting "Rebuild" button or forcing the builder with 2 properties set:

  • Property 1 name: script_repo_revision use FIREFOX_*_RELEASE tag here
  • Property 2 name -> release_config: mozilla/release-firefox-mozilla-2.0.py

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

Check Redirects

Thunderbird start page/releasenotes/whatsnew redirects are stored in SVN at https://svn.mozilla.org/mozillamessaging.com/sites/live.mozillamessaging.com/trunk/conf/live.mozillamessaging.com.conf.

Check that the redirects for the current Thunderbird release are in place before releasing. Beta redirects are only needed for Beta1 as they are not specific to a beta number.

Run Backupsnip

Code: /cvsroot/mozilla/tools/release/bin/backupsnip

This script backs up all the snippets that will be overwritten by the corresponding pushsnip. This should be run prior to doing any pushsnip. If possible, running it ahead of time is good because it speeds things up when we're ready to push. The following is an example of running backupsnip: 

# As ffxbld or tbirdbld@aus3-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-13.0-build1

Push to releases directory / internal mirrors

Code: scripts/release/push-to-mirrors.py

This builder:

  1. Runs rsync and pushes files to the release directory
  2. Triggers mirror uptake monitoring poller.

When enableAutomaticPushToMirrors is True, this is triggered automatically after the antivirus scan is done. When False, it must be manually triggered with Force Build.

Dealing with index.html files

For high-visibility releases (generally, .0 Firefox releases) we try to prevent direct linking to the files by pushing index files to avoid directory listings. 

# ffxbld@stage
version=14.0
cd /pub/mozilla.org/firefox/releases/$version
wget --no-check-certificate -O index.html https://bugzilla.mozilla.org/attachment.cgi?id=631778
sed -i -e "s/13/$version/g" index.html 
for dir in `find . -mindepth 1 -type d `; do cp -pv index.html $dir/; done

These need to be removed after shipping. Release Management should be consulted before removing them. 

version=14.0
cd /pub/mozilla.org/firefox/releases/$version
find . -name index.html -exec rm {} \;

Push to external mirrors

If this is a chemspill you will want to ping justdave (or other sysadmin) in #release-drivers and ask them to enable CDN.

  • Remove previously added rsync exclusion from stage.mozilla.org:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude
  • Replace old version with the current one in stage.mozilla.org:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude

See details.

If the CDN needs to be enabled for the release (check with Release Management if you're unsure), it can be done like this:

  • Set rating of CDN to 45k and enable it

Once uptake for the release hits 100,000, the CDN should be disabled again.

Final Verification

Code:

Once mirror uptake has reached 50 or 60% this builder should be automatically started by the uptake monitoring process. If these don't get fired automatically, manually verify uptake on bouncer, then start the builder for each platform with 'Force Build' (no parameters needed) from the Buildbot waterfall.

This is a simple test which is run after mirrors are propagated to ensure there's no (major) problems with them. It does the same thing as the second part of Update Verify: for every locale/platform/version combination it checks for an update snippet, and then performs an HTTP HEAD request on the MAR the snippet points to.

The step will turn red if there are http codes other than 200 or 302, or if FAIL is found in the log. If it does this, you should have a look in the log, and find out which mirror is failing. It's a good idea to keep an eye on that mirror for awhile to make sure it finishes getting all of the files.

Searching the log for errors is made easier by downloading it and running the following locally: 

grep HTTP/1 text | grep -v 302 | grep -v 200
grep FAIL text

Bouncer Submitter

Code:

Configuration:

  • firefox-tuxedo.ini for stable versions, and firefox-devpreview-tuxedo.ini for Alpha releases.
  • BuildSlaves.py file should contain the following variables:
    • tuxedoUsername: Tuxedo username with appropriate permissions for adding new entries via API
    • tuxedoPassword: password for this user

This builder can be started any time with 'Force Build' from the Buildbot waterfall.

This step adds adds bouncer entries for installers, complete and partial updates for all platforms listed in enUSPlatforms and l10nPlatforms and locales specified in shipped-locales (en-US is being added in any case).

The step will turn red if tuxedo-add.py exits non zero. There are 2 major error sources:

  • tuxedoUsername has not enough privileges to add entries
  • Tuxedo is not aware of new locales added in the current version. Usually the log contains "HTTPError: HTTP Error 400: BAD REQUEST". See this link for an example workflow on how to resolve this problem.

Publish Fennec

Push the files

Play Store

IMPORTANT - Make sure to follow instructions!

  • download both multi apks. These have the same name so save them to separate directories.
  • visit https://market.android.com/publish
  • choose "Firefox Beta" or "Firefox"
  • select the "APK Files" tab and choose "Upload APK"
    • choose the android multi apk that you downloaded and hit "upload"
      • NOTE: the progress bar does not seem to show progress (at least on Google Chrome)
    • when it finishes uploading, verify VersionCode is the build date (almost anyway, the market reports an hour later than our buildID)
    • hit "Save"
  • If you're also uploading XUL Fennec, repeat the same steps for the android-xul multilocale apk
  • activate new android apk (You should see an Error message since both apk's are active)
  • deactivate old android apk (You should not see the Error message anymore). You can identify the two build types by
    • Native Fennec (phones) has Supported screens: small-large
    • XUL Fennec (tablets) has Supported screens: xlarge
  • If you're also activating XUL Fennec, repeat those steps for the android-xul multilocale apk.
  • hit "Save"
  • If this is a beta 1, go to "Product Details" tab -> Recent Changes, change the url to http://www.mozilla.com/en-US/mobile/{VERSION}beta/releasenotes/
    • and hit "Save"

Note: you can verify that the correct version code is being offered to device owners by viewing the main page for the product:

It usually takes about 75 minutes for the new product to appear as available in Google Play.

Push snippets

Prerequisite: push to mirrors (beta releases only go to internal-mirrors).

This happens after QA signs off and we get an explicit "go" from Release Management. Do not push snippets unless you're certain you should. If you're unsure, ask someone. For example: 

# As ffxbld or tbirdbld@aus3-staging
cd /opt/aus2/snippets/staging
~/bin/pushsnip Firefox-13.0-build1

Check Throttling

See http://people.mozilla.com/~nthomas/update-watch/release/ for example AUS links

Some links to check:

Unlock Build Slaves

Once you are done with the build machines that you locked to your buildbot master, unlock them in slavealloc and clear the comments field.

Update XULRunner wiki page

Update the links on:

Update Symlinks

Some lines of code have symlinks that point to the latest version. For example, there's a "latest" symlink for both Firefox and Thunderbird that points to the latest rapid release. After a release is shipped these symlinks need to be updated. Here's an example of how to update the "latest" snippet: 

# ffxbld or tbirdbld@stage.mozilla.org
version=13.0
product=thunderbird
cd /pub/mozilla.org/$product
rm latest && ln -s $version latest

If you don't know whether or not the release you're working on has a symlink, check the releases directory for your product or ask someone.

Update the Browser Choice Website

As part of the Choice Campaign we need to make sure that the Browser Choice website is always pointing at the latest version of Firefox. IT manages the website, and they simply need to be poked to update it after each release ships. Use this Bugzilla template to file the bug, making sure to fill in the proper version in the summary and description.

Retrieved from "https://wiki.mozilla.org/index.php?title=Release:Release_Automation_on_Mercurial:Updates_through_Shipping&oldid=460839"