ReferencePlatforms/Mac-10.5
Contents
- 1 Reference Image Info
- 2 Things done for you
- 2.1 Darwin Version
- 2.2 Initial Setup and Account Creation
- 2.3 Ref platform packages
- 2.4 Installing Xcode
- 2.5 Installing CHUD
- 2.6 Installing MacPorts
- 2.7 ssh
- 2.8 ENV setup
- 2.9 switch to role key
- 2.10 chown scripts
- 2.11 Establish initial ssh connections and accept keys
- 2.12 Mail config
- 2.13 ntp config
- 2.14 Tp2 config
- 2.15 nagios
- 2.16 Buildbot
- 2.17 Final Configuration
- 2.18 Misc loose ends
- 2.19 Update to Xcode 3.1
- 2.20 Installing Mercurial 1.2.1
- 2.21 Copy fonts from another Mac
- 2.22 Set dock size
- 2.23 Symlink for autoconf
- 2.24 "Install" 7z
- 2.25 Setup for verification on staging
- 2.26 Setup buildbot start on boot
- 2.27 Puppet client installation
- 2.28 Post-puppet packages
- 3 Manual setup after cloning
- 4 Puppet packages
Reference Image Info
We now have one image for xserve and one for minis:
macOSX-10.5.2-xserve-ref-image macOSX-10.5.2-mini-ref-v4 will be the current image. (image is up-to-date at the colo and 650 Castro 20091020)
You must do everything in the "Manual steps after cloning" section after a machine is freshly imaged, still.
Now that the reference image for minis and xserves has Puppet on it new images do not need to taken every time something is added, new machines that receive the current image will automatically update themselves when deployed.
Updating the images
As mentioned above, these images do not need to be updated every time a change happens. However, it is good to update them from time to time, to speed up new slave deployment, and make it cleanlier in general. Here's how to do it:
Updating the mini image
bm-mini-build-ref.build.mozilla.org
We've got a mac mini which is permanently dedicated to being a ref image. It will be kept up to date by Puppet. Therefore, the only thing to do to update it is file an IT bug requesting a new image be taken. Make sure you ask them to use the new image for all future deployments.
Updating the xserve image
Unlike the mini, we do not have a machine dedicated to being a ref image, which makes this process a little more involved.
First, choose an xserve and shut down the Buildbot process on it. Next, some prep work:
# as cltbld # backup the tac file and... # clobber the slave dir so buildbot.tac gets generated on new slaves, and the trees are fresh cd /builds/slave mv buildbot.tac ~cltbld rm -rf * # kill puppet so the ssh keys don't get replaced before we're done ps auxww | grep puppet kill -15 $puppetpid cd /Library/LaunchDaemons sudo mv com.reductivelabs.puppet.plist ~cltbld # switch to staging ssh keys cd ~ rm -rf .ssh sudo rsync -av /N/darwin9/.ssh-staging/ .ssh/ sudo chown -R cltbld:admin .ssh # change the hostname - the name is significant! sudo scutil --set HostName bm-xserve-build-ref.build.mozilla.org # ...and reboot sudo reboot
Once all of the above is done, it's time to file a bug and wait.
Once the new image has been taken the xserve needs to be returned to its original place:
# restore the tac file, hostname puppet, and get rid of the staging ssh keys cd ~cltbld mv buildbot.tac /builds/slave sudo mv com.reductivelabs.puppet.plist /Library/LaunchDaemons rm -rf .ssh sudo scutil --set HostName $originalhostname.build.mozilla.org
Now, reboot and Puppet will sync up the ssh keys and start Buildbot for you.
Things done for you
Darwin Version
These are the kernel versions we're starting with:
Intel: Darwin bm-xserve16.build.mozilla.org 9.2.0 Darwin Kernel Version 9.2.0: Tue Feb 5 16:13:22 PST 2008; root:xnu-1228.3.13~1/RELEASE_I386 i386
This is 10.5.2, up to date on 28/Feb/2008.
Initial Setup and Account Creation
- Change the resolution to 1024 x 768, 60 Hz, Millions of Colours, under System Preferences->Displays (this is changed later)
- Change the password for the administrator user to the standard build system root password.
- Create a new admin-equiv account for the cltbld user using the standard build password.
- Under System Preferences->Sharing->Screen Sharing->Computer Settings change the VNC screen control password to the standard build password. NOTE: your VNC session will drop when you do this. You'll need to reconnect with the new password.
- If the hostname is wrong, e.g. unused-*, use
sudo scutil --set HostName new.hostname.mozilla.orgto fix it. NOTE: you'll need to restart the machine for this to fully take effect. - Under System Preferences->Software Update, turn off automatic update checking.
- Using the Workgroup Manager, open the Preferences pane for the cltbld user. Under Software Update, set Manage: to Always, and set the Software Update server to use: as http://bm-xserve01.build.mozilla.org:8088/ - NEED TO FIGURE THIS OUT
Ref platform packages
Helpfully, OS X doesn't have CVS out of the box, so retrieve the Mac reference platform packages on another machine:
cvs -d :ext:cltbld@cvs.mozilla.org:/mofo co ref-platforms/mac/chud_4.5.0.dmg cvs -d :ext:cltbld@cvs.mozilla.org:/mofo co ref-platforms/mac/macports-10.5.tar.bz2 cvs -d :ext:cltbld@cvs.mozilla.org:/mofo co ref-platforms/mac/MacPorts-1.6.0-10.5-Leopard.dmg cvs -d :ext:cltbld@cvs.mozilla.org:/mofo co ref-platforms/mac/xcode_3.0.dmg
This should yield 4 files with md5sums:
3f02477e0df2a3bc53d02fe0824c0015 chud_4.5.0.dmg 514e818170147c3851820defb6151ee1 macports-10.5.tar.bz2 5cbae915b67f7d06576d544b83303714 MacPorts-1.6.0-10.5-Leopard.dmg e6ce732e203511c9d3102b027a8d89a8 xcode_3.0.dmg
Then transfer them to the machine you are setting up. (We switch to xcode 3.1 later)
Installing Xcode
- Mount the Xcode disk image, and begin installing the XcodeTools.mpkg.
- A standard install is fine (we don't need 10.3.9 support or WebObjects).
Now we have cvs.
Installing CHUD
- Mount the CHUD disk image, install the CHUD.mpkg
Installing MacPorts
- Install the MacPorts package from ref platform version 1.6-10.5; ignore the error about the postflight script
- Unpack the macports-10.5 tarball of source packages and move them into place:
cd /opt/local/var/macports sudo rm -rf * sudo tar jxf ~/ref-platforms/mac/macports-10.5.tar.bz2
- Next, you need to install a series of ports, all of which should already be contained (and the right version) in the macports tarball you just unpacked. If you see the port command hitting the network at all (-->Fetching...) while installing the needed ports, something has gone wrong.
sudo port install sqlite3 sudo port install autoconf213 cvs libidl subversion wget
There's a bug with sqlite3 where it fails to compile, depending on what was built before it, so we build it and it's deps first. The second command builds everything else. Afterwards, running port installed should show only the 21 ports listed below:
apr @1.2.12_1+darwin_9 (active) apr-util @1.2.12_0 (active) autoconf213 @2.13_0 (active) cvs @1.11.22_0 (active) db44 @4.4.20_1 (active) expat @2.0.1_0 (active) gawk @3.1.6_0 (active) gettext @0.17_3 (active) glib2 @2.14.6_0+darwin_9 (active) libiconv @1.12_0 (active) libidl @0.8.10_0 (active) ncurses @5.6_0 (active) ncursesw @5.6_1 (active) neon @0.26.4_0 (active) openssl @0.9.8g_0 (active) pkgconfig @0.23_0 (active) readline @5.2.007_0+darwin_9 (active) sqlite3 @3.5.6_0 (active) subversion @1.4.6_0 (active) wget @1.11_0 (active) zlib @1.2.3_1 (active)
ssh
scp the cltbld and appropriate role keys over directly from a machine that already has them installed. In .ssh, make sure the cltbld keys are called cltbld* and not id_dsa*, then symlink
ln -s cltbld_dsa id_dsa ln -s cltbld_dsa.pub id_dsa.pub
This is temporary to get hold of the standard .profile (we should move that file to public cvs).
ENV setup
cvs -d :ext:cltbld@cvs.mozilla.org:/mofo co release/tinderbox-configs/host ln -s release/tinderbox-configs/host/mac-osx-profile .profile
...at which point you'll probably want to restart your shell to use the new .profile.
switch to role key
In ~/.ssh,
rm id_dsa id_dsa.pub ln -s ffxbld_dsa id_dsa ln -s ffxbld_dsa.pub id_dsa.pub
or substitute ffxbld for whatever key is appropriate for what you will be building.
chown scripts
NOTE: This section is now taken care of by puppet (bug 528189).
After you've created your initial /builds/tinderbox directory structure, checkout and build the following tools that are also required for changing permissions while packaging on Macs:
cd ~ cvs -d ffxbld@cvs.mozilla.org:/cvsroot co mozilla/build/macosx/permissions cd mozilla/build/macosx/permissions gcc -o chown_revert chown_revert.c gcc -o chown_root chown_root.c mkdir ~/bin cp chown_revert chown_root ~/bin cd ~/bin chmod 4755 chown* sudo chown root chown* ln -s chown_revert revert_root
Establish initial ssh connections and accept keys
Execute the following simple ssh calls to connect to the ftp server (stage), symbol server (if different from ftp), and update server so that the various pieces of the build can be uploaded automatically by the build scripts when needed:
ssh ffxbld@stage.mozilla.org pwd ssh ffxbld@aus2-staging.mozilla.org pwd ssh ffxbld@hg.mozilla.org ssh ffxbld@dm-symbolpush01.mozilla.org pwd
Mail config
NB: Postfix config is tinderbox client specific, all the buildbot information goes over a socket to the master and is posted to the tinderbox server from there.
You'll need to update the postfix config to make sure that myhostname is set correctly and that large messages can be sent. The tail of /etc/postfix/main.cf should look similar to:
myhostname = bm-xserve16.build.mozilla.org mailbox_transport = cyrus # Allow large messages (unlimited size, required for tinderbox) message_size_limit = 0
You'll then want to restart postfix to pick up the changes.
Also make sure that Mail.app is installed.
ntp config
sudo su -
echo "server ntp1.build.mozilla.org iburst" > /etc/ntp.conf
kill -2 `ps x | grep ntp | grep -v grep | awk '{print $1}'`
exit
Tp2 config
NB: This is deprecated by talos.
Set up the server by doing to Application->Server->Server Preferences. Disable File Sharing, iCal, and iChat. Then go to Web, and uncheck "Enable group Wikis", "Webmail", and "User blogs"; set "Home Page" to "Server Home Page".
To setup Tp2:
cd /builds/tinderbox cvs -d ffxbld@cvs.mozilla.org:/cvsroot mozilla/tools/performance/pageload cd /Library/WebServer/Documents ln -s /builds/tinderbox/mozilla/tools/performance/pageload .
nagios
- Copy over the appropriate package from bm-admin01, either nrpe-darwin-i386.tar.gz or nrpe-darwin-powerpc.tar.gz depending on your platform.
- Create a nagios user and group
sudo dscl localhost -create /Local/Default/Users/nagios sudo dscl localhost -create /Local/Default/Users/nagios UserShell /usr/bin/false sudo dscl localhost -create /Local/Default/Users/nagios UniqueID 510 sudo dscl localhost -create /Local/Default/Groups/nagios sudo dscl localhost -create /Local/Default/Groups/nagios UniqueID 1010
- Unpack the package in /usr/local. Eg,
cd /usr/local && sudo tar xfz ~cltbld/nrpe-darwin-i386.tar.gz
- Edit/checkout nrpe.cfg. The one in mozilla/tools/nagios/nrpe-mac.cfg may be useful.
- Run enablenrpe to turn it on
sudo /usr/local/nagios/sbin/enablenrpe
Ask IT to setup the suite of checks.
Buildbot
Make directories
mkdir -p /tools/dist/logs mkdir -p /builds/logs
Install Python 2.5.2
Python 2.5 doesn't compile on 10.5, and we'll take the latest version rather than the 2.5.1 that ships with the OS.
mkdir -p /tools/dist cd /tools/dist wget http://www.python.org/ftp/python/2.5.2/Python-2.5.2.tar.bz2 tar jxvf Python-2.5.2.tar.bz2 cd Python-2.5.2 ./configure --prefix=/tools/python make make test make install export PYTHONHOME="/tools/python" export PATH="/tools/python/bin:$PATH"
Installing Zope Interface
cd /tools/dist/ wget http://www.zope.org/Products/ZopeInterface/3.3.0/zope.interface-3.3.0.tar.gz tar xfvz zope.interface-3.3.0.tar.gz cd zope.interface-3.3.0/ python setup.py install --prefix=/tools/zope-interface-3.3.0 cd /tools; ln -s zope-interface-3.3.0/ zope-interface export PYTHONPATH="/tools/zope-interface/lib/python2.5/site-packages/:$PYTHONPATH"
Installing Twisted
cd /tools/dist/ wget http://tmrc.mit.edu/mirror/twisted/Twisted/2.4/TwistedCore-2.4.0.tar.bz2 wget http://tmrc.mit.edu/mirror/twisted/Twisted/2.4/Twisted-2.4.0.tar.bz2 tar xfvj TwistedCore-2.4.0.tar.bz2 tar xfvj Twisted-2.4.0.tar.bz2 cd TwistedCore-2.4.0 python setup.py install --prefix=/tools/twisted-core-2.4.0 pushd /tools/; ln -s twisted-core-2.4.0/ twisted-core; popd export PYTHONPATH="/tools/twisted-core/lib/python2.5/site-packages:$PYTHONPATH" cd /tools/dist/Twisted-2.4.0/ python setup.py install --prefix=/tools/twisted-2.4.0 cd /tools; ln -s twisted-2.4.0 twisted export PYTHONPATH="/tools/twisted/lib/python2.5/site-packages/:$PYTHONPATH"
Final Configuration
Add the appropriate paths to the Buildbot user's .bash_profile
echo 'export PYTHONHOME="/tools/python"' >> ~/.profile echo 'export PYTHONPATH="/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-corelib/python2.5/site-packages/:/tools/zope-interface/lib/python2.5/site-packages/"' >> ~/.profile echo 'export PATH="/opt/local/bin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:$PYTHONHOME/bin:$PATH"' >> ~/.profile
Misc loose ends
We used to do the following manually, but this change is now done on the ref image, to reduce manual setup.
- Change screen resolution to 1280 x 1024, Millions of colours
- Set the background to a Solid Colour (better for slower network connections)
- Remove crud from doc, leaving Finder, System Preferences, Terminal, Activity Monitor (not essential, just tidier)
- have /usr/local/bin/autoconf-2.13 symlink to /opt/local/bin/autoconf-2.13
Update to Xcode 3.1
- Check out the XCode disk image from cvs (will need to adjust ~/.ssh/id_dsa)
cvs -d :ext:cltbld@cvs.mozilla.org:/mofo co -d xcode ref-platforms/mac/xcode_3.1.dmg
- Check sha1 sum is right
openssl dgst -sha1 xcode_3.1.dmg; \ echo 'SHA1(xcode_3.1.dmg)= e90f21f262dec5f1dae983e8fb53a613f2cc72b0'
- Mount the Xcode disk image, and begin installing the XcodeTools.mpkg.
- A standard install is fine (we don't need 10.3.9 support or WebObjects).
Installing Mercurial 1.2.1
cd /tools/dist wget http://www.selenic.com/mercurial/release/mercurial-1.2.1.tar.gz # SHA512(mercurial-1.2.1.tar.gz)= dcadfc731e5e1afae2cfc3eb1ac46875c26d5f029228ea5f397061c05e0524bdf651e3443d8936ec78f51c8d92a904d89c1a45ca941ee8908a99ad5c075096c2 tar xfvz mercurial-1.2.1.tar.gz cd mercurial-1.2.1 python setup.py install
Copy fonts from another Mac
cd /Library/Fonts sudo rsync -av cltbld@bm-xserve16:/Library/Fonts/ ./
Set dock size
defaults write com.apple.dock tilesize -int 49; killall Dock
(until a fix is in for bug 470420)
Symlink for autoconf
Due to calling in RepackFactory to command=['bash', '-c', 'autoconf-2.13'] to fix this we have to add a symlink to autoconf213
sudo ln -s /opt/local/bin/autoconf213 /opt/local/bin/autoconf-2.13
"Install" 7z
We need 7zip for the l10n verification step of releases
sudo scp -p cltbld@bm-xserve16:/usr/local/bin/7z* /usr/local/bin/
Setup for verification on staging
Move ~/.ssh to ~/ssh_prod, and scp staging keys in from a staging box. Patch master config to allow new clone to connect and get jobs.
Setup buildbot start on boot
cd /Library/LaunchAgents sudo wget --no-check-certificate -Obuildbot.start.slave.plist https://bug428124.bugzilla.mozilla.org/attachment.cgi?id=371801 sudo chown root:wheel buildbot.start.slave.plist
- Ensure the the correct user and slave directory are set in the script (cltbld and /builds/slave by default)
From VNC:
- Make sure the resolution is set to 1280x1024. (this is not in the xserve ref image)
- System Prefs -> Accounts -> Login Options
- Set 'Automatic Login' to 'cltbld', enter the password when prompted.
Reboot (after setting up the slave, see following step in this doc). NB: The LaunchAgent will try to start buildbot every 10 minutes, be wary of leaving a working buildbot.tac in place during setup
(See bug 428124 for details.)
Puppet client installation
# as root curl http://reductivelabs.com/downloads/gems/facter-1.5.6.gem > facter-1.5.6.gem md5 facter-1.5.6.gem # should be e42990b6f40ade3c07e9ee91f052f717 curl http://projects.reductivelabs.com/attachments/download/584/puppet-0.24.8.gem > puppet-0.24.8.gem md5 puppet-0.24.8.gem # should be ad85a0aa6d63a91fbef6e578e3309bee gem install facter-1.5.6.gem puppet-0.24.8.gem mkdir /N mount 10.2.71.136:/export/buildlogs/puppet-files /N cp /N/darwin9/com.reductivelabs.puppet.plist /Library/LaunchDaemons cp /N/darwin9/sleep-and-run-puppet.sh /usr/local/bin chown root:wheel /Library/LaunchDaemons/com.reductivelabs.puppet.plist chmod 644 /Library/LaunchDaemons/com.reductivelabs.puppet.plist mkdir -p /var/puppet/log sudo launchctl load -w /Library/LaunchDaemons/com.reductivelabs.puppet.plist
After the next reboot Puppet will be starting on boot.
Post-puppet packages
Manual setup after cloning
If you used the new ref images then you should only need to do all the bullet points and sections below
- Set the hostname if required (IT probably did this, but see above if not)
- Double check that the display resolution is 1280 x 1024
- Double check that Bluetooth Setup Assistant is turned off (System Preferences -> Bluetooth -> Advanced -> uncheck "Open Setup Assistant...")
- Verify that screen saver is off - check bug 562629
- Verify that spotlight is disabled
- * For a try builder, be sure to wipe all ssh keys and copy over trybld keys from another try builder
Add slave(s) to configuration files
The following files need to be updated with new slaves (note that the production Buildbot config is explicitly not updated here, it will be done later):
- buildbot-configs/mozilla2-staging/config.py
- puppet-manifests/site-production.pp (or site-staging.pp for slaves that will be in staging permanently).
- For a permanent staging slave - edit /Library/LaunchDaemons/com.reductivelabs.puppet.plist to staging-puppet
Once your patches have landed, you need to update the following checkouts:
- On staging-master:
# as cltbld cd /builds/buildbot/user-configs/clean-configs hg pull && hg up cd ../../user-configs2/clean-configs hg pull && hg up
- On production-puppet (or staging-puppet if you updated site-staging.pp):
# as root cd /etc/puppet/manifests hg pull && hg up
- If a machine has been recloned/replaced then you also do this as root@production-puppet
puppetca --clean moz2-darwin8-slaveNN.build.mozilla.org
Setup buildbot slave
The buildbot.tac file will automatically be generated to send the slave to the correct staging master (staging-master or sm-staging-try-master).
Run the slave through staging
see ReleaseEngineering:BuildSlaveSetup
Puppet packages
Disable bluetooth
TODO: Document this.
See bug 570843.
