UpdateGeneration: Difference between revisions

Jump to navigation Jump to search

UpdateGeneration (view source)

Revision as of 04:48, 5 July 2012

2,191 bytes removed ,  5 July 2012
no edit summary
No edit summary
Line 5: Line 5:
== Current processes ==
== Current processes ==


Currently we use two different sets of processes and tools to generate updates for nightly updates and releases.  
Currently we use two different sets of processes and tools to generate updates for nightly updates and releases.


=== Nightly updates ===
=== Nightly updates during code development ===


We make updates available for official nightly builds for all current code branches. This helps ensure that our nightly testers are always running the most recent code with the least possible effort. These updates are offered on the 'nightly' update channel.
We make updates available for official nightly builds for all current code branches. This helps ensure that our nightly testers are always running the most recent code with the least possible effort. These updates are offered on the 'nightly' update channels, or derivates like 'nightly-ux', and for 'aurora'.


Both complete and partial updates are offered. If the user is running the most recent previous nightly build, the partial update is served. Otherwise, the complete update is always served. This greatly simplifies/expedites the update generation matrix.  
Both complete and partial updates are offered. If the user is running the most recent previous nightly build, the partial update is served. Otherwise, the complete update is always served. This greatly simplifies/expedites the update generation matrix.  


Steps for nightly update generation (3.5 branch and earlier):
Steps for nightly update generation:
* nightly builders generate complete [[Software_Update:MAR|MAR]] using Makefile
* nightly builder generate complete MAR using [http://hg.mozilla.org/mozilla-central/file/default/tools/update-packaging/Makefile.in Makefile]
* upload complete MAR to staging along with build
* nightly builder downloads the previous complete MAR from the latest dir on the staging server ([http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/ eg])
* upload complete snippet to AUS2 server
* prometheus-vm runs cron script every 5 minutes (patch-packager-cron.sh) that calls patch-packager.pl. In the simplest terms, this script does the following:
*# Read in all of the available builds.
*# Compute all of the patches we could offer.
*# Read in all of the offered patches.
*# Compute which patches we aren't offering which we could.
*# Create all of the patches we could offer.
*# Upload them.
 
As of Firefox 3.6, we've stopped funneling all partial updates through prometheus-vm in favor of generating the partial updates directly on the builders. This gives us much quicker turnaround on update generation.
 
Steps for nightly update generation (3.6 branch, project branches, and later):
* nightly builder generate complete MAR using Makefile
* nightly builder downloads the previous complete MAR from the latest dir on the staging server
* nightly builder generates partial MAR containing the differences between the two complete MARS using Makefile
* nightly builder generates partial MAR containing the differences between the two complete MARS using Makefile
* nightly builder generates snippets for both the complete and partial updates
* nightly builder generates snippets for both the complete and partial updates
Line 36: Line 22:
For more information about how the update MAR files are actually created, please see the section below entitled [[#What_the_Makefiles_do.2C_or_How_to_make_your_own_updates|How to make your own updates]].
For more information about how the update MAR files are actually created, please see the section below entitled [[#What_the_Makefiles_do.2C_or_How_to_make_your_own_updates|How to make your own updates]].


=== Updates for security releases ===
=== Updates for end users ===


We create both partial and complete updates for security releases (aka as 'dot' releases, e.g. 3.5.x). These updates are only ever served to users on the same product branch.  
We create both partial and complete updates for the 'release' and 'beta' channel too.


Legitimate partial updates, i.e. binary diffs, are generated for migrating users from the previous security release to the new security release, e.g. 3.5.1->3.5.2. Complete mars are served to those users on older security releases, e.g. 3.5->3.5.2. Because the updater checks for partial updates before complete updates, the update system serves the complete update for a security release as a partial update to all users on previous security releases. This avoids presenting the user with a failed check for a partial update that, while harmless, might be confusing.
Legitimate partial updates, i.e. binary diffs, are generated for migrating users from the previous security release to the new security release, e.g. 13.0->13.0.1. Complete mars are served to those users on older security releases, e.g. 12.0.


Steps for security release update generation:
Steps for end-user update generation:
* partial MARs and AUS configuration data are generated using the patcher2 tool: http://mxr.mozilla.org/mozilla/source/tools/patcher/
* similar to the nightly case, the build slave generates both a complete and a partial update
* a release engineer puts the appropriate URLs, version numbers, and build IDs into the patcher configuration file: http://mxr.mozilla.org/mozilla/source/tools/patcher-configs/
* snippets are generated using the patcher2 tool: http://mxr.mozilla.org/mozilla/source/tools/patcher/
* patcher downloads the complete MARs, creates partial diffs (using mbsdiff), and generates partial MAR files as well as AUS configuration files ("snippet" files).
* patcher generates both production ('release'/'beta') and testing ('releasetest'/'betatest') snippets, for QA to verify
* patcher generates both production ("release") and testing ("releasetest") snippets, release snippets point to bouncer while releasetest snippets point to stage.
* partial MAR files are copied to the candidates directory, and the releasetest snippets are loaded into AUS.


=== Updates between major branches (aka major update) ===
=== Updates between major branches (aka major update) ===
 
Prior to Firefox 4.0, there were long lived branches that produced 3.5.x and 3.6.x releases. An update that converted users from 3.5.x to 3.6.x, or from 3.6.x to the latest release was known as a major update. They were characterised by displaying a dialog offering the user the new branch (ie new features) and asking them to accept the update. In the post-4.0 world these are not used as everyone gets a background update to the latest release, modulo OS compatibility. The only exceptions have been attempts to get people on older releases to update (eg Fx10 is current, offer to 6.0 and older).
=== Which machines do what  ===
 
{| cellpadding="5" border="1"
|-
| '''Machine Name'''
| '''Function'''
|-
| any build slave
| build generation<br/>complete patch(MAR) generation<br/>partial patch(MAR) generation (3.6+)<br/>XML snippet generation (3.6+)
|-
| prometheus-vm
| partial patch(MAR) generation (3.5)<br/>XML snippet generation (3.5)
|-
| aus2-staging.mozilla.com
| store/serve snippets
|}


== What the Makefiles do, or How to make your own updates ==
== What the Makefiles do, or How to make your own updates ==
Line 77: Line 45:
* [http://ftp.mozilla.org/pub/mozilla.org/xulrunner/mar-generation-tools/mar-generation-tools-macosx.zip mar-generation-tools-macosx.zip]
* [http://ftp.mozilla.org/pub/mozilla.org/xulrunner/mar-generation-tools/mar-generation-tools-macosx.zip mar-generation-tools-macosx.zip]
* [http://ftp.mozilla.org/pub/mozilla.org/xulrunner/mar-generation-tools/mar-generation-tools-win32.zip mar-generation-tools-win32.zip]
* [http://ftp.mozilla.org/pub/mozilla.org/xulrunner/mar-generation-tools/mar-generation-tools-win32.zip mar-generation-tools-win32.zip]
XXX: These need updating for the maintenance service changes


=== Creating a complete MAR ===
=== Creating a complete MAR ===
Line 105: Line 74:
  mkdir current
  mkdir current
  pushd current
  pushd current
  ../tools/update-packaging/unwrap_full_update.sh current.complete.mar
  ../tools/update-packaging/unwrap_full_update.sh /path/to/current.complete.mar
  popd
  popd
  mkdir previous
  mkdir previous
  pushd previous
  pushd previous
  ../tools/update-packaging/unwrap_full_update.sh previous.complete.mar
  ../tools/update-packaging/unwrap_full_update.sh /path/to/previous.complete.mar
  popd
  popd


Line 158: Line 127:


We then upload these snippets to the Mozilla [[AUS|AUS2]] server. The AUS2 server has internal logic to determine the most recent complete and partial updates available for a given product/branch/version, so that when the client updater requests an update from the server, it's able to redirect to the correct MAR file.
We then upload these snippets to the Mozilla [[AUS|AUS2]] server. The AUS2 server has internal logic to determine the most recent complete and partial updates available for a given product/branch/version, so that when the client updater requests an update from the server, it's able to redirect to the correct MAR file.
= Staging environment for nightly updates =
The staging environment for nightly updates consists of two machines:
* staging-nightly-updates: this machine mirrors prometheus-vm in production, running the patch-packager scripts to generate partial mars based on update snippets published to...
* staging-stage: used like stage.m.o in production to house builds and mars, but also holds the aus2 snippets used in update generation and publishing
== Common problems and resolutions with updates in the staging environment ==
=== Downloaded MAR file doesn't match the hash in the snippet ===
This can happen when two successful "nightly" builds start in the same hour (this shouldn't ever happen in production FWIW). The result is two different snippet files for two different mars, but only one mar in the corresponding build dir on staging-stage (these dirnames are only accurate down to the hour).
Removing the orphaned snippet should fix things up.
NThomas
Confirmed users
4,293

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/447704"

Navigation menu