ReleaseEngineering/How To/Update Mercurial

Mercurial is one of our core binaries that we need to install everywhere.

Philosophy

We install Mercurial into /tools/python27-mercurial/ everywhere* (* - On windows we override the mozilla-build version). This is due to wanting a single, common, place for mercurial, and to avoid needing to update system packages that may depend on mercurial.

This document is designed so it can/should be followed top-down, but with self-contained sections to allow easy skimming/read-through.

Future Issues

There are some forseen reasons this doc could be incorrect for your needs. Below are those reasons, and suggested solutions.

• We change the way PuppetAgain goes about doing packaging
  • Follow the new documentation to achieve a similar result, this page is informative not normative for how we do things.
• We deployed PuppetAgain on windows (adding a need for windows to the puppet code)
  • We'll need to do this from scratch once, and update these docs.
• OSX 10.9 is no longer used in our infra (bug 1110612)
  • Delete the 10.9 instructions from this wiki page, and rejoice

Prepare for the Upgrade

• File a bug for the upgrade, (if not already filed)
• Run the hgtools tests against the new mercurial version (TODO: Link to doc how)
• Send an e-mail to Dev Services <developer-services@mozilla.org> asking if there are any concerns (set a deadline, so you are not indefinitely blocked on replies)

Create the Packages

CentOS

6.x

(This is an informative description of The PuppetAgain How to Build RPMs)

• In us-east AWS console find and start rpmpackager1.srv.releng.use1.mozilla.com
• Connect

USER=`whomai`
ssh $USER@rpmpackager1.srv.releng.use1.mozilla.com

• Get Old RPM, Spec from puppet, and extract spec from rpm, Also get new sources

NEWHGVER=4.0  # Set this appropriately
NEWHGREL=1 #Set this too
OLDHGVER=3.2.1
OLDHGREL=1
OLDHGFULL=$OLDHGVER-$OLDHGREL
NEWHGFULL=$NEWHGVER-$NEWHGREL
PUPPETAGAIN=http://puppetagain.pub.build.mozilla.org
HGREPO=data/repos/yum/releng/public/CentOS/6
mkdir mercurial-$NEWHGVER; cd mercurial-$NEWHGVER
wget $PUPPETAGAIN/$HGREPO/x86_64/mozilla-python27-mercurial-$OLDHGFULL.el6.x86_64.rpm
wget $PUPPETAGAIN/$HGREPO/i386/mozilla-python27-mercurial-$OLDHGFULL.el6.i686.rpm
wget $PUPPETAGAIN/$HGREPO/x86_64/mozilla-python27-mercurial-$OLDHGFULL.el6.src.rpm
mkdir old-extracted; cd old-extracted
rpm2cpio ../mozilla-python27-mercurial-$OLDHGFULL.el6.src.rpm | cpio -ivd
cd ..
wget http://mercurial.selenic.com/release/mercurial-$NEWHGVER.tar.gz

• Check for any unexpected differences between puppet and old version

diff py27_mercurial.spec old-extracted/py27_mercurial.spec

• Update spec for versions

sed -i "s/\(Version:.*\)$OLDHGVER/\1$NEWHGVER/" py27_mercurial.spec
sed "s/\(Release:.*\)$OLDHGREL/\1$NEWHGREL/" py27_mercurial.spec

• Build the src rpm

mock -r puppetagain-centos6-64 --buildsrpm --sources $PWD --spec py27_mercurial.spec 

• Copy output to a safe place

mkdir output
cp /var/lib/mock/epel-6-x86_64/result/mozilla-python27-mercurial-$NEWHGFULL.el6.src.rpm output/

• Now build the actual rpms (and clean caches first)

sudo rm -rf /var/cache/mock/epel-6-x86_64/ccache
sudo rm -rf /var/cache/mock/epel-6-i386/ccache/
mock --rebuild -r puppetagain-centos6-64 output/mozilla-python27-mercurial-$NEWHGFULL.el6.src.rpm
mock --rebuild -r puppetagain-centos6-32 output/mozilla-python27-mercurial-$NEWHGFULL.el6.src.rpm

• Copy those out (if nothing failed):

cp /var/lib/mock/epel-6-x86_64/result/*.x86_64.rpm output/
cp /var/lib/mock/epel-6-x86_64/result/*.i?86.rpm output/ 

• IF there are any ERRORS
  • Edit the spec file to account for them and retry subsequent steps, you'll need to carry these spec file changes over into the Puppet Section of the work.
• scp the files in output to somewhere local, in prep for the #Deploy to PuppetAgain Section
• Halt the system (avoids need for aws console)

sudo su -
users # Check for other logged in users first, be nice!
halt

Ubuntu

Precise (12.04)

(This is an informative description of The PuppetAgain How to Build DEBs)

• In us-east AWS console find and start ubuntu64packager1.srv.releng.use1.mozilla.com
• Connect

USER=`whomai`
ssh $USER@ubuntu64packager1.srv.releng.use1.mozilla.com

• Get old from repo

NEWHGVER=4.0  # Set this appropriately
OLDHGVER=3.2.1
PUPPETAGAIN=http://puppetagain.pub.build.mozilla.org
HGREPO=data/repos/apt/custom/mozilla-mercurial/pool/precise
mkdir old
wget $PUPPETAGAIN/$HGREPO/mozilla-python27-mercurial_3.2.1.debian.tar.gz
cd old
tar -xzf ../mozilla-python27-mercurial_3.2.1.debian.tar.gz  
cd ..

• Get old deb folder from puppet

hg clone https://hg.mozilla.org/build/puppet repo-puppet
cp -r puppet/modules/packages/manifests/mozilla/mozilla-python27-mercurial-debian ./

• Check for surprising differences

diff -r ./mozilla-python27-mercurial-debian/ old/debian/

• Edit changelog (adding new entry)
  • Note DEB Changelogs follow a specific format, add to top, match the one below yours.

EDITOR=vim
$EDITOR ./mozilla-python27-mercurial-debian/changelog

• Get the new .tar.gz from selenic (and make sure its named correctly post-download)

wget http://mercurial.selenic.com/release/mercurial-$NEWHGVER.tar.gz \
     -O mozilla-python27-mercurial_$NEWHGVER.orig.tar.gz

• Build it

puppetagain-build-deb precise amd64 mozilla-python27-mercurial-debian
puppetagain-build-deb precise i386 mozilla-python27-mercurial-debian

• Copy the artifacts out

cp /tmp/mozilla-python27-mercurial-precise-amd64/* output/
cp /tmp/mozilla-python27-mercurial-precise-i386/* output/

• scp the files in output to somewhere local, in prep for the #Deploy to PuppetAgain Section
• copy out any/all changes you have made to the puppet deb directory (the changelog edit above) - You'll include that with your puppet patch.
• Halt the system (avoids need for aws console)

sudo su -
users # Check for other logged in users first, be nice!
halt

OSX

(This is an informative description based on The PuppetAgain How to Build DMGs)

OSX is funky and needs a package built separately for each version, on the OS version its targeted for. This means we need to pull active slaves from rotation and install XCode on them via a puppet user environment.

I'm not repeating the puppet user environment instructions here, but I will detail what you need to change for these nodes.

10.6 (Snow Leopard)

TODO MOVE THE OSX SECTION BELOW THE PACKAGE DEPLOY SECTIONS. WE NEED RPM ON PUPPETAGAIN FIRST

• Disable a random production slave, commenting with whatever bug you need it for, (you're essentially doing a loan - but need puppet running)
• Make sure you have a (clean/updated) puppetagain environment setup you can use (See The Docs)
• Add packagemaker and xcode to the node def for testers (applies to all OSX jobs)
  • Note This will break any tester puppet runs for non OSX (but is fixable by reverting this)

# linux64 and OS X
node /t.*-\d+\.test\.releng\.scl3\.mozilla\.com/ {
    # hosts starting with t and ending in -digit.test.releng.scl3.mozilla.com
    $slave_trustlevel = 'try'
    include toplevel::slave::releng::test::gpu
    # New Lines Here
    include packages::packagemaker
    include packages::xcode
}

• login to your host and run puppet
  • CHANGE THE PUPPET ARGS APPROPRIATELY'

ssh root@t-snow-r4-0000.build.mozilla.org
puppet agent --server foo --test --environment jwood

• change to cltbld

su - cltbld

• Get the necessary file:

mkdir mercurial
cd mercurial
wget -O py27_mercurial-dmg.sh \
    http://hg.mozilla.org/build/puppet/raw-file/default/modules/packages/manifests/mozilla/py27_mercurial-dmg.sh
chmod 755 ./py27_mercurial-dmg.sh

• Update the script to be the correct version

OLDHGVER=3.2.1
NEWHGVER=4.0
sed -i "s/version=$OLDHGVER/version=$NEWHGVER/" ./py27_mercurial-dmg.sh

• Get the new tar.gz

wget http://mercurial.selenic.com/release/mercurial-$NEWHGVER.tar.gz

• Run the script

./py27_mercurial-dmg.sh

• The script will output where the dmg is, copy it outside the host (storing it in a way that you know it was for 10.6)
• You now want to reimage the slave following this mana doc, and put it back in production in slavealloc

10.7 (Lion)

• Disable a random production slave, commenting with whatever bug you need it for, (you're essentially doing a loan - but need puppet running)
  • There is no need for puppet environment for this, its a builder and already has xcode
• login to your host as cltbld
• Get the necessary file:

mkdir mercurial
cd mercurial
wget -O py27_mercurial-dmg.sh \
    http://hg.mozilla.org/build/puppet/raw-file/default/modules/packages/manifests/mozilla/py27_mercurial-dmg.sh
chmod 755 ./py27_mercurial-dmg.sh

• Update the script to be the correct version

OLDHGVER=3.2.1
NEWHGVER=4.0
sed -i "s/version=$OLDHGVER/version=$NEWHGVER/" ./py27_mercurial-dmg.sh

• Get the new tar.gz

wget http://mercurial.selenic.com/release/mercurial-$NEWHGVER.tar.gz

• Run the script

./py27_mercurial-dmg.sh

• The script will output where the dmg is, copy it outside the host (storing it in a way that you know it was for 10.7)
• You now want to reimage the slave following this mana doc, and put it back in production in slavealloc

10.8 (Mountain Lion)

• Disable a random production slave, commenting with whatever bug you need it for, (you're essentially doing a loan - but need puppet running)
• Make sure you have a (clean/updated) puppetagain environment setup you can use (See The Docs)
• Add packagemaker and xcode to the node def for testers (applies to all OSX jobs)
  • Note This will break any tester puppet runs for non OSX (but is fixable by reverting this)

# linux64 and OS X
node /t.*-\d+\.test\.releng\.scl3\.mozilla\.com/ {
    # hosts starting with t and ending in -digit.test.releng.scl3.mozilla.com
    $slave_trustlevel = 'try'
    include toplevel::slave::releng::test::gpu
    # New Lines Here
    include packages::packagemaker
    include packages::xcode
}

• login to your host and run puppet
  • CHANGE THE PUPPET ARGS APPROPRIATELY'

ssh root@talos-mtnlion-r5-XXX.build.mozilla.org
puppet agent --server foo --test --environment jwood

• change to cltbld

su - cltbld

• Get the necessary file:

mkdir mercurial
cd mercurial
wget -O py27_mercurial-dmg.sh \
    http://hg.mozilla.org/build/puppet/raw-file/default/modules/packages/manifests/mozilla/py27_mercurial-dmg.sh
chmod 755 ./py27_mercurial-dmg.sh

• Update the script to be the correct version

OLDHGVER=3.2.1
NEWHGVER=4.0
sed -i "s/version=$OLDHGVER/version=$NEWHGVER/" ./py27_mercurial-dmg.sh

• Get the new tar.gz

wget http://mercurial.selenic.com/release/mercurial-$NEWHGVER.tar.gz

• Run the script

./py27_mercurial-dmg.sh

• The script will output where the dmg is, copy it outside the host (storing it in a way that you know it was for 10.8)
• You now want to reimage the slave following this mana doc, and put it back in production in slavealloc

10.9 (Mavericks)

• WARNING As of this writing we only have mac-v2-signing using mavericks, however we do need mercurial on these systems.
• The version built on 10.8 worked with mavericks as of this docs writing, so instead of giving full docs here for building a mavericks version, I'm hopeful next hg ver works on mavericks when built on 10.8 or we no longer have mavericks in production.
• Deployment section will do the deploy of the 10.8 version for Mavericks

10.10 (Yosemite)

• Disable a random production slave, commenting with whatever bug you need it for, (you're essentially doing a loan - but need puppet running)
• Make sure you have a (clean/updated) puppetagain environment setup you can use (See The Docs)
• Add packagemaker and xcode to the node def for testers (applies to all OSX jobs)
  • Note This will break any tester puppet runs for non OSX (but is fixable by reverting this)

# linux64 and OS X
node /t.*-\d+\.test\.releng\.scl3\.mozilla\.com/ {
    # hosts starting with t and ending in -digit.test.releng.scl3.mozilla.com
    $slave_trustlevel = 'try'
    include toplevel::slave::releng::test::gpu
    # New Lines Here
    include packages::packagemaker
    include packages::xcode
}

• login to your host and run puppet
  • CHANGE THE PUPPET ARGS APPROPRIATELY'

ssh root@talos-t-yosemite-r5-XXXX.build.mozilla.org
puppet agent --server foo --test --environment jwood

• change to cltbld

su - cltbld

• Get the necessary file:

mkdir mercurial
cd mercurial
wget -O py27_mercurial-dmg.sh \
    http://hg.mozilla.org/build/puppet/raw-file/default/modules/packages/manifests/mozilla/py27_mercurial-dmg.sh
chmod 755 ./py27_mercurial-dmg.sh

• Update the script to be the correct version

OLDHGVER=3.2.1
NEWHGVER=4.0
sed -i "s/version=$OLDHGVER/version=$NEWHGVER/" ./py27_mercurial-dmg.sh

• Get the new tar.gz

wget http://mercurial.selenic.com/release/mercurial-$NEWHGVER.tar.gz

• Run the script

./py27_mercurial-dmg.sh

• The script will output where the dmg is, copy it outside the host (storing it in a way that you know it was for 10.10)
• You now want to reimage the slave following this mana doc, and put it back in production in slavealloc

Windows

As of this doc's writing; this is done via GPO installing to <mozilla-build>/hg as a blow-away-what-was-there install. This needs to be created for both builders (c.f. bug 1056981) and testers (c.f. bug 1110304)

Package Deployment

CentOS

Note: This is an informative depiction of https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#CentOS:_Adding_New_Packages

CentOS needs the files on releng-puppet2 (the distinguished puppetmaster).

• Transfer the files from local to releng-puppet2 (your user dir)

scp ./*.rpm `whoami`@releng-puppet2.srv.releng.scl3.mozilla.com:mercurial/

• As root, copy the files over and fixup

**CHANGE THE USERNAME**
sudo su -
for i in `ls ~jwood/mercurial/mozilla-python27-mercurial-*.el6.{src,x86_64,i686}.rpm`;
  do
    cp ./$i /data/repos/yum/releng/public/CentOS/6/x86_64/$i; 
done
for i in `ls ~jwood/mercurial/mozilla-python27-mercurial-*.el6.{src,i686}.rpm`;
  do
    cp ./$i /data/repos/yum/releng/public/CentOS/6/i386/$i; 
done
pushd /data/repos/yum/releng/public/CentOS/6/i386
puppetmaster-fixperms
popd
pushd /data/repos/yum/releng/public/CentOS/6/x86_64
puppetmaster-fixperms
popd

Ubuntu

Note: This is an informative depiction of https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Ubuntu:_Adding_New_Packages

• Transfer the files from local to releng-puppet2 (your user dir)

scp ./* `whoami`@releng-puppet2.srv.releng.scl3.mozilla.com:mercurial-apt/

• As root, copy the files over and fixup

**CHANGE THE USERNAME**
sudo su -
cd /data/repos/apt/custom/mozilla-mercurial
cp -v ~jwood/mercurial-apt/amd64/mozilla-python27-mercurial_*{_amd64.changes,_amd64.deb,.debian.tar.gz,.dsc,.orig.tar.gz} ./pool/precise/
cp -v ~jwood/mercurial-apt/i386/mozilla-python27-mercurial_*{_i386.changes,_i386.deb,.debian.tar.gz,.dsc,.orig.tar.gz} ./pool/precise/
./update.sh
puppetmaster-fixperms

OSX

Note: This is an informative depiction of https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Darwin

• Transfer the files from local to releng-puppet2 (your user dir)

scp ./*.dmg `whoami`@releng-puppet2.srv.releng.scl3.mozilla.com:mercurial-DMGs/
** This instruction assumes your DMGs are located at 10.X/ under that directory

• As root, copy the files over and fixup

**CHANGE THE USERNAME**
sudo su -
cd /data/repos/DMGs
cp -v ~jwood/mercurial-DMGs/10.6/* 10.6/
cp -v ~jwood/mercurial-DMGs/10.7/* 10.7/
cp -v ~jwood/mercurial-DMGs/10.8/* 10.8/
cp -v ~jwood/mercurial-DMGs/10.10/* 10.10/
# 10.9 uses the same build as 10.8
cp -v ~jwood/mercurial-DMGs/10.8/* 10.9/
puppetmaster-fixperms

Create puppet patch

• replace the spec file for CentOS
  • The final spec file you used above for the CentOS build you should place at <path-to-local-puppet-repo>/modules/packages/manifests/mozilla/py27_mercurial.spec
• replace all files for the Debian packaging
  • The final files you use above for Ubuntu should be replace the entire contents of the directory at <path-to-local-puppet-repo>modules/packages/manifests/mozilla/mozilla-python27-mercurial-debian
• replace the dmg creation script
  • If you changed the package building script above for OSX, replace the one at <path-to-local-puppet-repo>modules/packages/manifests/mozilla/py27_mercurial-dmg.sh
• Update the mercurial versions in <path-to-local-puppet-repo>modules/packages/manifests/mozilla/py27_mercurial.pp
  • You can do this by hand too, be sure to change the versions specified

sed -i "s/3.2.1/3.2.2/" modules/packages/manifests/mozilla/py27_mercurial.pp

• Update the repoflag to cause the system package repository cache to refresh
  • Inside modules/packages/manifests/setup.pp you want to increase the repoflag var everwhere
• Post the patch to the upgrade bug, and request review. (testing first if desired, see next section)

Testing the patch

• On a puppetmaster you should apply the above created patch and run against one of each of the host types (centos builder both arch's, ubuntu testers both arch's, all vers of OSX)

Deploy the change

• Commit the reviewed patch to default branch of puppet
• Merge patch to production branch of puppet
• push patch
• Inform sheriffs of the deploy
• Allow 24 hours for AWS slaves to regen their "golden image" to ensure no breakage there.
• Patch should begin deploying to all server and "atboot" class machines within about 30 minutes of push.