Confirmed users
4,293
edits
Software Update:Nightly update infrastructure: Difference between revisions
Software Update:Nightly update infrastructure (view source)
Revision as of 09:30, 19 August 2010
, 19 August 2010no edit summary
(AUS info) |
No edit summary |
||
| (24 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
<font color=red>'''This document is out of date. Tinderbox is no longer in use, partials are generated during nightly builds instead of on a dedicated box, and there's a var=value format for snippets.'''</font> | |||
=Overview= | =Overview= | ||
Updates are available for nightly builds of Firefox | Updates are available for nightly builds of Firefox, Thunderbird and Sunbird. Keeping this system running requires the coordination of tinderbox configuration, partial update generation, and the AUS server that provides update information to the client applications. This document describes how that all works, to the extent that publicly available information allows. Hopefully the build crew and Mike Morgan can fill out the details to make this page more complete. | ||
Supported app versions: | |||
* Firefox & Thunderbird on the branches for 1.5.0.x and 2.0.0.y, and on the trunk | |||
* Sunbird on the Mozilla1.8 branch and trunk | |||
Supported platforms: | |||
* the three Tier-1 platforms of Windows, Linux, and Macintosh Universal (PPC+Intel) | |||
Supported locales: | |||
* en-US only | |||
* en-US with a language pack (ab-CD.xpi) will update the app, but not the language pack (bugs [https://bugzilla.mozilla.org/show_bug.cgi?id=334136 334136], [https://bugzilla.mozilla.org/show_bug.cgi?id=313453 313453]). Mozilla1.8 branch and trunk only | |||
=Tinderbox= | =Tinderbox= | ||
The tinderbox produces the nightly build & | The tinderbox produces the nightly build & mar file for a complete update, and publishes them to the FTP. It also creates a complete-update snippet, which describes the update, and copies that to AUS. | ||
==mozconfig== | ==mozconfig== | ||
The update channel is set to "default" (no updates) unless this line is in the mozconfig: | The update channel is set to "default" (no updates) unless this line is in the mozconfig: | ||
ac_add_options --enable-update-channel=nightly | ac_add_options --enable-update-channel=nightly | ||
For 1.5 the beta releases shipped with a "beta" channel, while the release candidates and final shipped with "release". | |||
For Firefox 1.5, the beta releases shipped with a "beta" channel, while the release candidates and final shipped with "release". | |||
To pull the code for complete update generation you need: | To pull the code for complete update generation you need: | ||
mk_add_options MOZ_CO_MODULE="mozilla/tools/update-packaging" | mk_add_options MOZ_CO_MODULE="mozilla/tools/update-packaging" | ||
This line is also used: | |||
ac_add_options --enable-update-packaging | ac_add_options --enable-update-packaging | ||
The mar executable is built by default. | |||
This option ensures that the update-packaging and (m)bsdiff Makefile's are created. | |||
The mar executable (mozilla/modules/libmar/tool) is built by default and copied into $objdir/dist/host/bin. | |||
The (m)bsdiff executable creates binary diffs and is needed for partial updates only. It, too, is copied to $objdir/dist/host/bin. | |||
==Snippet configuration== | ==Snippet configuration== | ||
| Line 24: | Line 44: | ||
A handful of variables in tinder-config.pl control the update and snippet generation on the tinderbox. The code is [http://lxr.mozilla.org/mozilla/source/tools/tinderbox/post-mozilla-rel.pl#464 mozilla/tools/post-mozilla-rel.pl::update_create_package] and runs after the successful completion of the build and any tests. | A handful of variables in tinder-config.pl control the update and snippet generation on the tinderbox. The code is [http://lxr.mozilla.org/mozilla/source/tools/tinderbox/post-mozilla-rel.pl#464 mozilla/tools/post-mozilla-rel.pl::update_create_package] and runs after the successful completion of the build and any tests. | ||
{| | {| class="fullwidth-table" | ||
|- | |- | ||
| Variable || Example value || Use | | Variable || Example value || Use | ||
|- | |- | ||
| $update_package || 1 || Whether to create a complete update | | $update_package || 1 || Whether to create a complete update (other conditions apply too) | ||
|- | |- | ||
| $update_pushinfo || 1 || Whether to push the update snippet to AUS | | $update_pushinfo || 1 || Whether to push the update snippet to AUS | ||
| Line 34: | Line 54: | ||
| $update_product || "Firefox" || Product identifier for AUS | | $update_product || "Firefox" || Product identifier for AUS | ||
|- | |- | ||
| $update_version || " | | $update_version || "2.0" || "Branch" identifier for AUS | ||
|- | |- | ||
| $update_platform || "WINNT_x86-msvc" || Platform ABI for AUS | | $update_platform || "WINNT_x86-msvc" || Platform ABI for AUS | ||
| Line 40: | Line 60: | ||
| $update_hash || "SHA1" || Hash function for download verfication | | $update_hash || "SHA1" || Hash function for download verfication | ||
|- | |- | ||
| $update_filehost || "mozilla | | $update_filehost || "ftp.mozilla.org" || The server that provides the mar files to the client | ||
|- | |- | ||
| $update_appv || " | | $update_appv || "2.0a2" || Version that appears in the update notification dialog | ||
|- | |- | ||
| $update_extv || " | | $update_extv || "2.0a2" || Version used to check extension compatibility (this seems to not work properly all the time) | ||
|- | |||
| $update_ver_file || "browser/config/version.txt" || Optional - A file (relative to $topsrcdir) to look up the version from if $update_appv and/or $update_extv are not defined ([https://bugzilla.mozilla.org/show_bug.cgi?id=334011 Bug 334011]) | |||
|} | |} | ||
The examples reflect current real world values rather than those in tinder-defaults.pl. | |||
The snippet is scp'ed to | The snippet is scp'ed to | ||
aus2-staging.mozilla.org:/opt/aus2/build/0/$update_product/... | |||
$update_version/$update_platform/$buildid/$locale/complete.txt | $update_version/$update_platform/$buildid/$locale/complete.txt | ||
A complete-snippet for a Firefox build from the Mozilla1.8 branch looks like: | |||
complete | |||
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2006-05-10-02-mozilla1.8/firefox-2.0a2.en-US.win32.mar | |||
SHA1 | |||
202f8c4a5c6af9bd00569ed087e245dec87d67fb | |||
6925417 | |||
2006051002 | |||
2.0a2 | |||
2.0a2 | |||
And here is the relevant part of the tinderbox log (from almost the very end): | |||
Gathering complete update info... | Gathering complete update info... | ||
Got build ID | Got build ID 2006051002. | ||
Pushing third-gen update info... | Pushing third-gen update info... | ||
ssh -i /home/cltbld/.ssh/aus cltbld@ | ssh -i /home/cltbld/.ssh/aus cltbld@aus2-staging.mozilla.org mkdir -p /opt/aus2/build/0/Firefox/2.0/WINNT_x86-msvc/2006051002/en-US | ||
scp -i /home/cltbld/.ssh/aus | scp -i /home/cltbld/.ssh/aus /cygdrive/c/builds/tinderbox/Fx-Mozilla1.8/WINNT_5.2_Depend/mozilla/dist/update/update.snippet.1 cltbld@aus2-staging.mozilla.org:/opt/aus2/build/0/Firefox/2.0/WINNT_x86-msvc/2006051002/en-US/complete.txt | ||
Completed pushing update info... | Completed pushing update info... | ||
Update build completed. | Update build completed. | ||
First and second generation update info is deprecated and no longer in the tinderbox code ([https://bugzilla.mozilla.org/show_bug.cgi?id=341752 bug 341752]). Third generation update info is only sent to AUS if the $update_pushinfo variable is set. | |||
=Partial Update Generation= | =Partial Update Generation= | ||
The code for patcher2 lives [http://lxr.mozilla.org/mozilla/source/tools/patcher here]. This is used for releases, and generates the update files and info for AUS. | |||
* the work is done on | |||
The nightly system is a more ad-hoc solution: | |||
* the | * the work is done on the tinderbox prometheus, using cron to start it every 5 minutes | ||
* Chase Phillips wrote the code; Paul Reed is | * the script [http://lxr.mozilla.org/mozilla/source/tools/update-packaging/unwrap_full_update.sh unwrap_full_update.sh] is used to unpack two complete MAR files. Then [http://lxr.mozilla.org/mozilla/source/tools/update-packaging/make_incremental_update.sh make_incremental_update.sh] is run against the two resulting directories, and that calls (m)bsdiff to create binary diff's for files that have changed in the later build. The resulting patches are combined with a manifest to create the partial MAR file. | ||
* it pushes partial update to the FTP server (into the <i>from</i> directory of any <i>from</i>--><i>to</i> update), and a partial snippet to AUS | |||
* Chase Phillips wrote the code; Paul Reed (preed) is the current build engineer who knows how to drive it | |||
=AUS= | =AUS= | ||
This server returns update information to the client, based on the snippets it has from the build systems. It was written in PHP by Mike Morgan. | This server returns update information to the client, based on the snippets it has from the build systems. It was written in PHP by Mike Morgan (morgamic). The source is now available [http://lxr.mozilla.org/mozilla/source/webtools/aus/ from the Mozilla CVS server]. | ||
==Application update request== | ==Application update request== | ||
The application makes a request | The application makes a request to the url specified by the global preference app.update.url (unless app.update.url.override is defined, and see below for recent changes), which is: | ||
https://aus2.mozilla.org/update/1/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/update.xml | https://aus2.mozilla.org/update/1/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/update.xml | ||
This is preprocessed to make the required substitutions. Continuing the example above of a Firefox Mozilla1.8 branch nightly, this would be: | |||
https://aus2.mozilla.org/update/1/Firefox/2.0a2/2006050905/WINNT_x86-msvc/en-US/nightly/update.xml | https://aus2.mozilla.org/update/1/Firefox/2.0a2/2006050905/WINNT_x86-msvc/en-US/nightly/update.xml | ||
which returns | which returns | ||
| Line 93: | Line 129: | ||
</update> | </update> | ||
</updates> | </updates> | ||
The full list of %BUILD_TARGET%/$update_platform combinations is | |||
* Windows: WINNT_x86-msvc | |||
* Linux: Linux_x86-gcc3 | |||
* Mac Universal: Darwin_Universal-gcc3 | |||
* Mac PPC: Darwin_ppc-gcc3 | |||
From version 2.0a1 of Thunderbird, and 2.0b1 of Firefox, the preference app.update.url was changed to | |||
https://aus2.mozilla.org/update/2/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/update.xml | |||
ie, there is a new %OS_VERSION% ([https://bugzilla.mozilla.org/show_bug.cgi?id=343689 bug 343689 and dependents]) which will allow the next generation of toolkit apps (Firefox/Thunderbird 3) to be offered only to supported operating systems (eg: Windows 2000 or later and not Windows 9x). | |||
From version 3.0a9 of Firefox, the preference app.update.url was changed to: | |||
https://aus2.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml | |||
The schema version was bumped to 3, and distribution and distribution version fields were added. These additional fields are optional, and will be used by AUS to be able to deliver updates to customized distributions of Firefox. See [https://bugzilla.mozilla.org/show_bug.cgi?id=392496 bug 392496] for details. | |||
==AUS itself== | ==AUS itself== | ||
To | To provide this content, AUS maps the client version (%VERSION%) from the request to $update_version of snippet store. This was the mapping on Oct 02 2007, which uses wildcard matches since [https://bugzilla.mozilla.org/show_bug.cgi?id=364628 bug 364628]: | ||
'Firefox' => array( | |||
'1.5.0.*' => '1.5.0.x', | |||
'2.0*' => '2.0', | |||
'3.0a*' => 'trunk' | |||
), | |||
'Thunderbird' => array( | |||
'1.5.0.*' => '1.5.0.x', | |||
'2.0*' => '2.0', | |||
'3.0a*' => 'trunk' | |||
), | |||
'Sunbird' => array( | |||
'0.7*' => 'branch', | |||
'0.6a1' => 'trunk' | |||
) | |||
This is an abbreviated version with only the versions still in use (except Firefox 1.5.0.x, which is beyond end-of-life), the current config is [http://lxr.mozilla.org/mozilla/source/webtools/aus/xml/inc/config-dist.php#83 here]. | |||
There is a many to one relationship between versions from a | There is a many-to-one relationship between application versions from a code branch to an AUS disk location. Eg: The Mozilla1.8 branch is being used for development of v2.0 and will have versions 2.0a1, a2, a3, b1,b2, RC1, 2.0 final and all the 2.0.0.x security releases. | ||