Build:Release Automation:Trunk

Release Automation on Trunk

Items with a strikethrough have been fixed in the staging setup.

How to run the production slaves using the staging mirrors

In order to test the tinderbox and other setup of the production slaves it is sometimes necessary to run them without touching the production servers (eg, cvs.m.o, stage.m.o, etc). Here's how:

1. Copy the staging bootstrap.cfg to the /builds/buildbot/trunk-automation-master/bootstrap.cfg on production-trunk-automation.
2. Manually sync the staging-trunk-automation cvsmirror to a clean state (trailing slashes are important!):
rsync -av /builds/cvsmirror.clean/cvsroot/ /builds/cvsmirror/cvsroot/ rsync -av /builds/cvsmirror.clean/l10n/ /builds/cvsmirror/l10n/ rsync -av /builds/cvsmirror.clean/mofo/ /builds/cvsmirror/mofo/
3. Manually clean-up staging-trunk-automation and fx-linux-1.9-slave1: On staging-trunk-automation:
rm -rf /builds/config/* /builds/tags/* /builds/release/logs/* /builds/updates/* /builds/verify/* /home/ftp/pub/firefox/nightly/2007* /home/ftp/pub/firefox/nightly/3.0b2-candidates /data/cltbld/firefox-3.0b2 On fx-linux-1.9-slave1: rm -rf /home/ftp/pub/firefox/nightly/2007* /home/ftp/pub/firefox/nightly/3.0b2-candidates /data/cltbld/firefox-3.0b2
4. Cleanup the tagging area on production-trunk-automation
rm -rf /builds/tags/*
5. Comment out the "clean staging area" and "create staging area" BuildSteps on production-trunk-automation's master.cfg.

At this point you should be able to start a run on production-trunk-automation that doesn't touch any production servers.

General Notes

• There is a set of patches in ~cltbld/local-patches that must be applied to the /builds/cvsmirror.clean whenever it is updated from the main CVS mirror
• CVS on windows requires '-z3' or it will not work correctly. (OR follow the "msys package upgrade" steps below)
• Commented out the SYMBOL_SERVER_* and $crashreporter_* variables in mozilla/tools/tinderbox-configs/firefox/*/tinder-config.pl on the 'release' and 'l10n_release' tags.
• Bootstrap has it's own special cvsmirror in /builds/cvsmirror.bootstrap/cvsroot. master.cfg was updated to reflect this.
• Some msys packages were upgraded on trunk. Here's how to do it:

1. Run this script:
#!/bin/sh files="http://easynews.dl.sourceforge.net/sourceforge/mingw/zlib-1.2.3-MSYS-1.0.11.tar.bz2 http://easynews.dl.sourceforge.net/sourceforge/mingw/openssl-0.9.8e-3-MSYS1.0.11.tar.bz2 http://easynews.dl.sourceforge.net/sourceforge/mingw/openssh-4.6p1-MSYS-1.0.11.tar.bz2 http://easynews.dl.sourceforge.net/sourceforge/mingw/minires-1.01-1-MSYS-1.0.11.tar.bz2" for file in $files; do wget $file done for file in *.tar.bz2; do tar -jvxf $file done
2. When that's done, make a backup of your existing msys insntallation, just in case.
3. Copy usr/bin/* to your existing msys bin directory. You will have to overwrite some files.

• Make sure all the tinder-config.pl's (release and l10n_release branches) have ReleaseToNightly=0 and ReleaseToDated=1 set.
• Had to install 7z on OS X slave.
• Created a moz-trunk-patcher2.cfg and checked in to local mofo mirror:

<app>
    <Firefox>
        <current-update>
            to   3.0b1
            testchannel   betatest releasetest
            <rc>
                betatest   1
                beta   1
            </rc>
            from   3.0a9
            channel   beta release
            details   http://www.mozilla.com/firefox/releases/3.0a9.html
            <complete>
                beta-url   http://staging-trunk-automation.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b1-candidates/rc3/firefox-3.0b1.%locale%.%platform%.complete.mar
                url   http://staging-trunk-automation.build.mozilla.org/?product=firefox-3.0b1-complete&os=%bouncer-platform%&lang=%locale%
                path   firefox/nightly/3.0b1-candidates/rc3/firefox-3.0b1.%locale%.%platform%.complete.mar
                betatest-url   http://fx-linux-1.9-slave1.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b1-candidates/rc3/firefox-3.0b1.%locale%.%platform%.complete.mar
            </complete>
        </current-update>
        past-update   3.0a9 betatest releasetest beta release
        <release>
            <3.0a9>
                <platforms>
                    win32   2007110904
                    linux-i686   2007110903
                    mac   2007110903
                </platforms>
                <exceptions>
                    gu-IN   mac, linux-i686
                    eu   linux-i686 win32
                </exceptions>
                version   3.0a9
                completemarurl   http://staging-trunk-automation.build.mozilla.org/pub/mozilla.org/firefox/releases/3.0a9/firefox-3.0a9.%locale%.%platform%.complete.mar
                locales   be ca cs de el en-US es-AR es-ES eu fi fr fy-NL ga-IE gu-IN he ja-JP-mac ja ka ko ku lt nb-NO nl pa-IN pl ru sk sv-SE tr uk zh-CN
            </3.0a9>
            <3.0b1>
                <platforms>
                    linux-i686   2007110903
                    win32   2007110904
                    mac   2007110903
                </platforms>
                <exceptions>
                    ja-JP-mac    mac
                    ja    linux-i686, win32
                    gu-IN   linux-i686, mac
                </exceptions>
                version   3.0b1
                locales   be cs de el en-US es-ES fi fr fy-NL gu-IN ja-JP-mac ja ka ko lt nl pl ru sk sv-SE uk zh-CN
                completemarurl   http://fx-linux-1.9-slave1.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b1-candidates/rc3/firefox-3.0b1.%locale%.%platform%.complete.mar
                schema   1
                extension-version   3.0b1
            </3.0b1>
        </release>
    </Firefox>
</app>

• Had to add moz-trunk-firefox-${platform}.cfg files in mozilla/testing/release/updates. These files were empty to begin with.

Problems & Solutions

• 'Tag substep Bump died: None of ^WIN32_MODULE_PRODUCTVERSION_STRING=3.0a9pre$ found in file /builds/tags/FIREFOX_3_0_A9_RC1/cvsroot/mozilla/browser/app/module.ver: at Bootstrap/Step/Tag/Bump.pm line 149.'
  • Your cvsmirror is out-of-date. Run the Tag Builder with the cvsmirror steps enabled to resolve this. (double check this)
• Python errors on OS X (cannot import os module, etc)
  • configure seems to be finding the wrong version of Python. Worked around by moving /usr/bin/python symlink to point to /tools/python/bin/python
• CVS sometimes hangs on windows
  • Upgrade ssh or use -z3 (see above)
• Build fails on an nsinstall of scriptaculous unit tests
  • This is caused by bug 396187. Workaround by using /e/fx19rel and /e/fx19l10nrel as the tinderbox directories
    • Make sure to change bootstrap.cfg to use these new directories.
• pushDir not found at end of Build
  • last-built is not being removed at the start of a new build. remove it manually before doing a Build.
  • On Windows Tinderbox/Bootstrap seem to use different names for the build platform...'WINNT_5.2_Depend' seems to be a safe choice.
• wget errors in Repack::Execute()

wget -nv --output-document "/builds/tinderbox/Fx-Mozilla1.9-l10n-Release/Linux_2.6.18-8.el5_Depend/firefox.tar.gz" http://staging-trunk-automation.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b2-candidates/rc1/firefox-3.0b2.en-US.linux-i686.tar.gz
End:   Mon Nov 12 10:41:20 2007
Error: wget failed or timed out.

• Make sure the HTTP server on staging-trunk-automation is running
• On Linux, this may occur because your tinder-config.pl is hardcoded to look for firefox.tar.gz. This should be updated to firefox.tar.bz2. (The '# CONFIG' lines, that is).

• mar failing out, returning 255
  • one of the existing mar files is corrupt. you should delete the mall (/builds/updates/firefox-3.0b2/patcher/temp) and run Updates::Execute() again
• win32 Build dies citing a problem with firefox.ico.
  • The firefox.ico on GECKO190_20071106_RELBRANCH on cvs.mozilla.org needs to be landed on the local cvsmirror trunk (the clean and working copy). the pullDate also needs to be bumped up to make sure it gets this version.
• Build::Push() on win32 tries to push win32_info.txt to the wrong directory.
  • Make sure logs are going to /builds/logs NOT /e/builds/logs. The latter will not work with the Makefile.
• Step Stage died: msgCannot rename /data/cltbld/firefox-3.0b2/batch1/stage-unsigned/linux-xpi /data/cltbld/firefox-3.0b2/batch1/stage-unsigned/linux-i686/xpi: No such file or directory at Bootstrap/Step/Stage.pm line 373.
  • Create the linux-i686 directory. Why isn't this done by Bootstrap? How does it work in staging?
• Mac/Windows fail to download previous packages.
  • The Firefox 3.0b1 packages were grabbed from stage.m.o. These packages are named "Firefox 3.0 Beta 1....". The release automation uses "Firefox 3.0b1....".
    • Had to rename packages on the staging FTP server to the right format.
    • I don't _think_ this will be a problem in production. The same format _should_ be used all of the time.