ReleaseEngineering/How To/Update Mercurial

Mercurial is one of our core binaries that we need to install everywhere.

Philosophy

We install Mercurial into /tools/python27-mercurial/ everywhere* (* - On windows we override the mozilla-build version). This is due to wanting a single, common, place for mercurial, and to avoid needing to update system packages that may depend on mercurial.

This document is designed so it can/should be followed top-down, but with self-contained sections to allow easy skimming/read-through.

Future Issues

There are some forseen reasons this doc could be incorrect for your needs. Below are those reasons, and suggested solutions.

• We change the way PuppetAgain goes about doing packaging
  • Follow the new documentation to achieve a similar result, this page is informative not normative for how we do things.
• We deployed PuppetAgain on Windows (adding a need for Windows to the puppet code)
  • We'll need to do this from scratch once, and update these docs.
• OSX 10.6 will soon no longer be used in our infra
  • Delete the 10.6 instructions from this wiki page when that happens

Prepare for the Upgrade

• File a bug for the upgrade, (if not already filed)
• Send an e-mail to Dev Services <developer-services@mozilla.org> asking if there are any concerns (set a deadline, so you are not indefinitely blocked on replies)

Create the Packages

CentOS

6.x

(This is an informative description of The PuppetAgain How to Build RPMs)

• In us-east AWS console find and start rpmpackager1.srv.releng.use1.mozilla.com
• Connect

USER=`whomai`
ssh $USER@rpmpackager1.srv.releng.use1.mozilla.com

• Get Old RPM, Spec from puppet, and extract spec from rpm, Also get new sources

NEWHGVER=4.4.1  # Set this appropriately
NEWHGREL=1 #Set this too
OLDHGVER=3.9.1
OLDHGREL=1
OLDHGFULL=$OLDHGVER-$OLDHGREL
NEWHGFULL=$NEWHGVER-$NEWHGREL
PUPPETAGAIN=http://puppetagain.pub.build.mozilla.org
HGREPO=data/repos/yum/custom/mozilla-mercurial/
mkdir mercurial-$NEWHGVER; cd mercurial-$NEWHGVER
wget $PUPPETAGAIN/$HGREPO/x86_64/mozilla-python27-mercurial-$OLDHGFULL.el6.x86_64.rpm
wget $PUPPETAGAIN/$HGREPO/i386/mozilla-python27-mercurial-$OLDHGFULL.el6.i686.rpm
wget $PUPPETAGAIN/$HGREPO/x86_64/mozilla-python27-mercurial-$OLDHGFULL.el6.src.rpm
mkdir old-extracted; cd old-extracted
rpm2cpio ../mozilla-python27-mercurial-$OLDHGFULL.el6.src.rpm | cpio -ivd
cd ..
wget https://www.mercurial-scm.org/release/mercurial-$NEWHGVER.tar.gz

• Check for any unexpected differences between puppet and old version

diff py27_mercurial.spec old-extracted/py27_mercurial.spec

• Update spec for versions

sed -i "s/\(Version:.*\)$OLDHGVER/\1$NEWHGVER/" py27_mercurial.spec
sed "s/\(Release:.*\)$OLDHGREL/\1$NEWHGREL/" py27_mercurial.spec

• Build the src rpm

mock -r puppetagain-centos6.5-64 --buildsrpm --sources $PWD --spec py27_mercurial.spec 

• Copy output to a safe place

mkdir output
cp /var/lib/mock/epel-6-x86_64/result/mozilla-python27-mercurial-$NEWHGFULL.el6.src.rpm output/

• Now build the actual rpms (and clean caches first)

sudo rm -rf /var/cache/mock/epel-6-x86_64/ccache
sudo rm -rf /var/cache/mock/epel-6-i386/ccache/
mock --rebuild -r puppetagain-centos6.5-64 output/mozilla-python27-mercurial-$NEWHGFULL.el6.src.rpm
mock --rebuild -r puppetagain-centos6.5-32 output/mozilla-python27-mercurial-$NEWHGFULL.el6.src.rpm

• Copy those out (if nothing failed):

cp /var/lib/mock/epel-6-x86_64/result/*.x86_64.rpm output/
cp /var/lib/mock/epel-6-x86_64/result/*.i?86.rpm output/ 

• IF there are any ERRORS
  • Edit the spec file to account for them and retry subsequent steps, you'll need to carry these spec file changes over into the Puppet Section of the work.
• scp the files in output to somewhere local, in prep for the #Deploy to PuppetAgain Section
• Halt the system (avoids need for aws console)

sudo su -
users # Check for other logged in users first, be nice!
halt

Ubuntu

Precise (12.04)

(This is an informative description of The PuppetAgain How to Build DEBs)

• In us-east AWS console find and start ubuntu64packager1.srv.releng.use1.mozilla.com
• Connect

USER=`whomai`
ssh $USER@ubuntu64packager1.srv.releng.use1.mozilla.com

• Get old from repo

NEWHGVER=4.4.1  # Set this appropriately
OLDHGVER=3.9.1
PUPPETAGAIN=http://puppetagain.pub.build.mozilla.org
HGREPO=data/repos/apt/custom/mozilla-mercurial/pool/precise
mkdir old
wget $PUPPETAGAIN/$HGREPO/mozilla-python27-mercurial_$OLDHGVER.debian.tar.gz
cd old
tar -xzf ../mozilla-python27-mercurial_$OLDHGVER.debian.tar.gz  
cd ..

• Get old deb folder from puppet

hg clone https://hg.mozilla.org/build/puppet repo-puppet
cp -r puppet/modules/packages/manifests/mozilla/mozilla-python27-mercurial-debian ./

• Check for surprising differences

diff -r ./mozilla-python27-mercurial-debian/ old/debian/

• Edit changelog (adding new entry)
  • Note DEB Changelogs follow a specific format, add to top, match the one below yours.

EDITOR=vim
$EDITOR ./mozilla-python27-mercurial-debian/changelog

• Get the new .tar.gz from selenic (and make sure its named correctly post-download)

wget https://www.mercurial-scm.org/release/mercurial-$NEWHGVER.tar.gz \
     -O mozilla-python27-mercurial_$NEWHGVER.orig.tar.gz

• Build it

puppetagain-build-deb precise amd64 mozilla-python27-mercurial-debian
puppetagain-build-deb precise i386 mozilla-python27-mercurial-debian

• Copy the artifacts out

cp /tmp/mozilla-python27-mercurial-precise-amd64/* output/
cp /tmp/mozilla-python27-mercurial-precise-i386/* output/

• scp the files in output to somewhere local, in prep for the #Deploy to PuppetAgain Section
• copy out any/all changes you have made to the puppet deb directory (the changelog edit above) - You'll include that with your puppet patch.
• Halt the system (avoids need for aws console)

sudo su -
users # Check for other logged in users first, be nice!
halt

OSX

(This is an informative description based on The PuppetAgain How to Build DMGs)

OSX is funky and needs a package built separately for each version, on the OS version its targeted for. This means we need to pull active slaves from rotation and install XCode on them via a puppet user environment.

I'm not repeating the puppet user environment instructions here, but I will detail what you need to change for these nodes.

10.7 (Lion)

• Disable a random production slave, commenting with whatever bug you need it for, (you're essentially doing a loan - but need puppet running)
  • There is no need for puppet environment for this, its a builder and already has xcode
• login to your host as cltbld
• Get the necessary file:

mkdir mercurial
cd mercurial
wget -O py27_mercurial-dmg.sh \
    http://hg.mozilla.org/build/puppet/raw-file/default/modules/packages/manifests/mozilla/py27_mercurial-dmg.sh
chmod 755 ./py27_mercurial-dmg.sh

• Update the script to be the correct version

OLDHGVER=3.9.1
NEWHGVER=4.4.1
sed -i "s/version=$OLDHGVER/version=$NEWHGVER/" ./py27_mercurial-dmg.sh

• Get the new tar.gz

wget https://www.mercurial-scm.org/release/mercurial-$NEWHGVER.tar.gz

• Run the script

./py27_mercurial-dmg.sh

• The script will output where the dmg is, copy it outside the host (storing it in a way that you know it was for 10.7)
• You now want to reimage the slave following this mana doc, and put it back in production in slavealloc

10.10 (Yosemite)

• Disable a random production slave, commenting with whatever bug you need it for, (you're essentially doing a loan - but need puppet running)
• Make sure you have a (clean/updated) puppetagain environment setup you can use (See The Docs)
• Add packagemaker and xcode to the node def for testers (applies to all OSX jobs)
  • Note This will break any tester puppet runs for non OSX (but is fixable by reverting this)

# linux64 and OS X
node /t.*-\d+\.test\.releng\.scl3\.mozilla\.com/ {
    # hosts starting with t and ending in -digit.test.releng.scl3.mozilla.com
    $slave_trustlevel = 'try'
    include toplevel::slave::releng::test::gpu
    # New Lines Here
    include packages::packagemaker
    include packages::xcode
}

• login to your host and run puppet
  • CHANGE THE PUPPET ARGS APPROPRIATELY'

SLAVE=talos-t-yosemite-r7-0089
ssh root@$SLAVE.test.releng.scl3.mozilla.com
USER=yourusername
puppet agent --server foo --test --environment $USER

• change to cltbld

su - cltbld

• Get the necessary file:

mkdir mercurial
cd mercurial
wget -O py27_mercurial-dmg.sh \
    http://hg.mozilla.org/build/puppet/raw-file/default/modules/packages/manifests/mozilla/py27_mercurial-dmg.sh
chmod 755 ./py27_mercurial-dmg.sh

• Update the script to be the correct version

OLDHGVER=3.9.1
NEWHGVER=4.4.1
sed -i "s/version=$OLDHGVER/version=$NEWHGVER/" ./py27_mercurial-dmg.sh

• Get the new tar.gz

wget http://mercurial.selenic.com/release/mercurial-$NEWHGVER.tar.gz

• Run the script

./py27_mercurial-dmg.sh

• The script will output where the dmg is, copy it outside the host (storing it in a way that you know it was for 10.10)
• You now want to reimage the slave following this mana doc, and put it back in production in slavealloc

Windows

As of this doc's writing; this is done via GPO installing to <mozilla-build>/hg as a blow-away-what-was-there install (on hardware machines) and via updating AMIs with the new hg version (on AWS machines). This needs to be done for both builders (c.f. bug 1302375) and testers (c.f. bug 1302376)

Package Deployment

CentOS

Note: This is an informative depiction of https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#CentOS:_Adding_New_Packages

CentOS needs the files on releng-puppet2 (the distinguished puppetmaster).

• Transfer the files from local to releng-puppet2 (your user dir)

USER=yourusername
scp ./*.rpm $USER@releng-puppet2.srv.releng.scl3.mozilla.com:mercurial/

• As root, copy the files over and fixup

**CHANGE THE USERNAME**
sudo su -
USER=yourusername
for i in $(ls /home/$USER/mercurial/mozilla-python27-mercurial-*.el6.{src,x86_64}.rpm);
  do
    cp $i /data/repos/yum/custom/mozilla-mercurial/x86_64/; 
done
for i in $(ls /home/$USER/mercurial/mozilla-python27-mercurial-*.el6.{src,i686}.rpm);
  do
    cp $i /data/repos/yum/custom/mozilla-mercurial/i386/; 
done
pushd /data/repos/yum/releng/public/CentOS/6/i386
puppetmaster-fixperms
popd
pushd /data/repos/yum/releng/public/CentOS/6/x86_64
puppetmaster-fixperms
popd

• Update repo metadata

cd /data/repos/yum/custom/mozilla-mercurial/x86_64/; createrepo --update .
cd /data/repos/yum/custom/mozilla-mercurial/i386/; createrepo --update .
puppetmaster-fixperms

Ubuntu

Note: This is an informative depiction of https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Ubuntu:_Adding_New_Packages

• Transfer the files from local to releng-puppet2 (your user dir)

USER=yourusername
scp ./* $USER@releng-puppet2.srv.releng.scl3.mozilla.com:mercurial-apt/

• As root, copy the files over and fixup

**CHANGE THE USERNAME**
sudo su -
USER=yourusername
cd /data/repos/apt/custom/mozilla-mercurial
cp -v /home/$USER/mercurial-apt/mozilla-python27-mercurial_*{_amd64.changes,_amd64.deb,_i386.changes,_i386.deb,.debian.tar.gz,.dsc,.orig.tar.gz} ./pool/precise/
./update.sh
puppetmaster-fixperms

OSX

Note: This is an informative depiction of https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Darwin

• Transfer the files from local to releng-puppet2 (your user dir)

USER=yourusername
scp ./*.dmg $USER@releng-puppet2.srv.releng.scl3.mozilla.com:mercurial-DMGs/
** This instruction assumes your DMGs are located at 10.X/ under that directory

• As root, copy the files over and fixup

**CHANGE THE USERNAME**
sudo su -
USER=yourusername
cd /data/repos/DMGs
cp -v ~$USER/mercurial-DMGs/10.6/* 10.6/
cp -v ~$USER/mercurial-DMGs/10.7/* 10.7/
cp -v ~$USER/mercurial-DMGs/10.10/* 10.10/
puppetmaster-fixperms

Create puppet patch

• replace the spec file for CentOS
  • The final spec file you used above for the CentOS build you should place at <path-to-local-puppet-repo>/modules/packages/manifests/mozilla/py27_mercurial.spec
• replace all files for the Debian packaging
  • The final files you use above for Ubuntu should be replace the entire contents of the directory at <path-to-local-puppet-repo>modules/packages/manifests/mozilla/mozilla-python27-mercurial-debian
• replace the dmg creation script
  • If you changed the package building script above for OSX, replace the one at <path-to-local-puppet-repo>modules/packages/manifests/mozilla/py27_mercurial-dmg.sh
• Update the mercurial versions in <path-to-local-puppet-repo>modules/packages/manifests/mozilla/py27_mercurial.pp
  • You can do this by hand too, be sure to change the versions specified

sed -i "s/3.7.3/3.9.1/" modules/packages/manifests/mozilla/py27_mercurial.pp

• Update the repoflag to cause the system package repository cache to refresh
  • Inside modules/packages/manifests/setup.pp you want to increase the repoflag var everwhere
• Post the patch to the upgrade bug, and request review. (testing first if desired, see next section)

Testing the patch

• On a puppetmaster you should apply the above created patch and run against one of each of the host types (centos builder, ubuntu testers both arch's, all vers of OSX)

Deploy the change

• Commit the reviewed patch to default branch of puppet
• Merge patch to production branch of puppet
• push patch
• Inform sheriffs of the deploy
• Allow 24 hours for AWS slaves to regen their "golden image" to ensure no breakage there.
• Patch should begin deploying to all server and "atboot" class machines within about 30 minutes of push.