MozillaWiki

ReleaseEngineering/PuppetAgain/Packages: Difference between revisions

Page Discussion

ReleaseEngineering/PuppetAgain/Packages (view source)

Revision as of 14:55, 28 February 2017

2,724 bytes added ,  28 February 2017
→‎node.js devtoolset-2-binutils
 
(60 intermediate revisions by 6 users not shown)
Line 1: Line 1:
= Principles =
= Requirements =


PuppetAgain needs to be able to repeatably produce identical machines over long time ranges (years).  That means carefully controlling which versions of which packages are available to be installed, both during machine creation and in runs of the puppet agent.  However, it's impractical and unnecessary to specify the exact version of *every* package installed on a host.
PuppetAgain needs to be able to repeatably produce identical machines over long time ranges (years).  That means carefully controlling which versions of which packages are available to be installed, both during machine creation and in runs of the puppet agent.  However, it's impractical and unnecessary to specify the exact version of *every* package installed on a host.
Line 8: Line 8:


How all of this happens differs widely among supported operating systems!
How all of this happens differs widely among supported operating systems!
= Principles =
* Do not allow the copying of a package to a repository to be the event that deploys a change -- that's un-trackable and difficult to revert
* Repositories form a strong isolation boundary: if you modify repository A, it's easy to guarantee that repository B hasn't changed.  Repositories are cheap, so use lots of them.
* A repository's update cadence is part of its definition, so define that up front.
* Repositories should be for use by puppet only.  Other consumers, such as mock environments, docker, etc., will have a different required update cadence and are unlikely to implement the proper resiliency to mirror failure, and thus should use repositories hosted elsewhere.


= CentOS =
= CentOS =


== Repositories ==
== CentOS: Repositories ==


Every puppetmaster hosts a bunch of [[ReleaseEngineering/PuppetAgain/Repositories|yum repositories]]:
Every puppetmaster hosts a bunch of [[ReleaseEngineering/PuppetAgain/Repositories|yum repositories]]:
Line 17: Line 24:
<table class="wikitable">
<table class="wikitable">
<tr><th>url</th><th>repository</th><th>arch</th><th>mirror date</th></tr>
<tr><th>url</th><th>repository</th><th>arch</th><th>mirror date</th></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/epel/6/latest/x86_64 repos/yum/mirrors/epel/6/latest/x86_64]</td><td>EPEL 6</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/epel/6/latest/x86_64}}</td><td>EPEL 6</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/epel/6/latest/i386 repos/yum/mirrors/epel/6/latest/i386]</td><td>EPEL 6</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/epel/6/latest/i386}}</td><td>EPEL 6</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/x86_64 repos/yum/mirrors/centos/6/latest/os/x86_64]</td><td>CentOS 6 Base</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/os/x86_64}}</td><td>CentOS 6.2 Base</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/i386 repos/yum/mirrors/centos/6/latest/os/i386]</td><td>CentOS 6 Base</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/os/i386}}</td><td>CentOS 6.2 Base</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/Source repos/yum/mirrors/centos/6/latest/os/Source]</td><td>CentOS 6 Base</td><td>source</td><td>2012-09-06</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/os/Source}}</td><td>CentOS 6.2 Base</td><td>source</td><td>2012-09-06</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/x86_64 repos/yum/mirrors/centos/6/latest/updates/x86_64]</td><td>CentOS 6 Updates</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/updates/x86_64}}</td><td>CentOS 6.2 Updates</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/i386 repos/yum/mirrors/centos/6/latest/updates/i386]</td><td>CentOS 6 Updates</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/updates/i386}}</td><td>CentOS 6.2 Updates</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/Source repos/yum/mirrors/centos/6/latest/updates/Source]</td><td>CentOS 6 Updates</td><td>source</td><td>2012-09-06</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/updates/Source}}</td><td>CentOS 6.2 Updates</td><td>source</td><td>2012-09-06</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/releases/Everything/i386/os repos/yum/mirrors/fedora/16/latest/releases/Everything/i386/os]</td><td>Fedora 16 Base</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/os/x86_64}}</td><td>CentOS 6.5 Base</td><td>x86_64</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/releases/Everything/x86_64/os repos/yum/mirrors/fedora/16/latest/releases/Everything/x86_64/os]</td><td>Fedora 16 Base</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/os/i386}}</td><td>CentOS 6.5 Base</td><td>i386</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/updates/i386 repos/yum/mirrors/fedora/16/latest/updates/i386]</td><td>Fedora 16 Updates</td><td>i386</td><td>2012-03-07</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/extras/x86_64}}</td><td>CentOS 6.5 Extras</td><td>x86_64</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/updates/x86_64 repos/yum/mirrors/fedora/16/latest/updates/x86_64]</td><td>Fedora 16 Updates</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/extras/i386}}</td><td>CentOS 6.5 Extras</td><td>i386</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64 repos/yum/mirrors/puppetlabs/el/6/products/x86_64]</td><td>Puppetlabs</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/updates/x86_64}}</td><td>CentOS 6.5 Updates</td><td>x86_64</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/puppetlabs/el/6/products/i386 repos/yum/mirrors/puppetlabs/el/6/products/i386]</td><td>Puppetlabs</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/updates/i386}}</td><td>CentOS 6.5 Updates</td><td>i386</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/puppetlabs/el/6/dependencies/x86_64 repos/yum/mirrors/puppetlabs/el/6/dependencies/x86_64]</td><td>Puppetlabs Deps</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/products/x86_64}}</td><td>Puppetlabs</td><td>x86_64</td><td>as necessary</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/puppetlabs/el/6/dependencies/i386 repos/yum/mirrors/puppetlabs/el/6/dependencies/i386]</td><td>Puppetlabs Deps</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/products/i386}}</td><td>Puppetlabs</td><td>i386</td><td>as necessary</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/passenger/rhel/6/latest/x86_64 repos/yum/mirrors/passenger/rhel/6/latest/x86_64]</td><td>Passenger</td><td>x86_64</td><td>2012-07-05</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/dependencies/x86_64}}</td><td>Puppetlabs Deps</td><td>x86_64</td><td>as necessary</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/proliantsupportpack/CentOS/6/i386/current/ repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/]</td><td>HP Proliant Support</td><td>i386</td><td>2012-08-21</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/dependencies/i386}}</td><td>Puppetlabs Deps</td><td>i386</td><td>as necessary</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/proliantsupportpack/CentOS/6/x86_64/current/ repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/]</td><td>HP Proliant Support</td><td>x86_64</td><td>2012-08-21</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/passenger/rhel/6/latest/x86_64}}</td><td>Passenger</td><td>x86_64</td><td>2012-07-05</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/noarch repos/yum/releng/public/CentOS/6/noarch]</td><td>Releng CentOS 6 Custom RPMs</td><td>noarch</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/}}</td><td>HP Proliant Support</td><td>i386</td><td>2012-08-21</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64 repos/yum/releng/public/CentOS/6/x86_64]</td><td>Releng CentOS 6 Custom RPMs</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/}}</td><td>HP Proliant Support</td><td>x86_64</td><td>2012-08-21</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/i386 repos/yum/releng/public/CentOS/6/i386]</td><td>Releng CentOS 6 Custom RPMs</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/CentOS/6/noarch}}</td><td>Releng CentOS 6 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>noarch</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/Fedora/16/noarch repos/yum/releng/public/Fedora/16/noarch]</td><td>Releng Fedora 16 Custom RPMs</td><td>noarch</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/CentOS/6/x86_64}}</td><td>Releng CentOS 6 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>x86_64</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/Fedora/16/x86_64 repos/yum/releng/public/Fedora/16/x86_64]</td><td>Releng Fedora 16 Custom RPMs</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/CentOS/6/i386}}</td><td>Releng CentOS 6 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>i386</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/Fedora/16/i386 repos/yum/releng/public/Fedora/16/i386]</td><td>Releng Fedora 16 Custom RPMs</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/Fedora/16/noarch}}</td><td>Releng Fedora 16 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>noarch</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/Fedora/16/x86_64}}</td><td>Releng Fedora 16 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/Fedora/16/i386}}</td><td>Releng Fedora 16 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/custom/*}}</td><td>Custom repositories</td><td>*</td><td></td></tr>
</table>
</table>


Notes:
Notes:
* Paths under ''yum/releng'' are custom packages, and are not mirrored from anywhere.
* Repos under ''repos/yum/custom'' are custom-built for some purpose, and are not mirrored from anywhere.  They may contain custom packages, or just packages culled from some mirror.  Look for references to the repos in the puppet manifests to find their purpose.
* We generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind.
* We generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind.
* The Fedora repos are only used within ''mock'' on buildslaves.  They are not used to install anything on puppet-managed hosts.
* The ''repos/yum/releng'' repos are deprecated; if you're tempted to add a package there, please create a new custom repository or update another existing custom repository instead
** The releng Fedora repos are currently empty, in fact
* The Fedora repos are unused and will be deleted soon
* Dynamic repositories are snapshots that are made on demand, where <tt>latest</tt> always points to the latest active snapshot.  They are *not* automatically updated.  The date on which they were most recently mirrored is given above.
* Dynamic repositories are snapshots that are made on demand, where <tt>latest</tt> always points to the latest active snapshot.  They are *not* automatically updated.  The date on which they were most recently mirrored is given above.
* The current CentOS version is available at http://repos/repos/yum/mirrors/centos/6/latest/centos-version.txt.
* The current CentOS version is available at http://repos/repos/yum/mirrors/centos/6/latest/centos-version.txt.
Line 55: Line 65:
=== Mirror Synchronization Commands ===
=== Mirror Synchronization Commands ===
{{note|Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space.  See {{bug|836014}}.  The process takes about 10 minutes.}}
{{note|Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space.  See {{bug|836014}}.  The process takes about 10 minutes.}}
{{note|From inside Mozilla, you'll need to use the Squid proxy: set RSYNC_PROXY to proxy.dmz.scl3.mozilla.com:3128}}


Remember that once changes land on the puppet master, they can be used for any purpose that needs packages.  That includes kickstart, which always takes the most recent version of a package it can find!  It also includes installs by puppet that specify ''version => "latest"'', and installs to fulfill requirements of packages defined by puppet.  Think twice!
Remember that once changes land on the puppet master, they can be used for any purpose that needs packages.  That includes kickstart, which always takes the most recent version of a package it can find!  It also includes installs by puppet that specify ''version => "latest"'', and installs to fulfill requirements of packages defined by puppet.  Think twice!
Line 66: Line 77:


==== CentOS 6 ====
==== CentOS 6 ====
* Base and Updates (6.2, x86_64)
rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://mirrors.usc.edu/centos/6/os/x86_64/ /data/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64
rsync -n --no-p -rv --exclude=drpms rsync://mirrors.usc.edu/centos/6/updates/x86_64/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/x86_64
This was badly botched.  Among other things, group information was omitted from the repodata.  Fixing that directly turns out to close the trees.  So we have re-mirrored the 6.2 os repos as of 2013-01-17 at `/data/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64-for-ks`.
time hardlink -v /data/repos/yum/mirrors/centos
* Base and Updates (6.2, i386)
rsync -n --exclude isolinux -aP rsync://linux.mirrors.es.net/centos/6.2/os/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/i386/
rsync -n --exclude drpms -aP rsync://linux.mirrors.es.net/centos/6.2/updates/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/os/i386/
time hardlink -v /data/repos/yum/mirrors/centos
NOTE: the i386 CentOS 6.2 repos were mirrored in July of 2012, but are in the 2012-03-07 directory.  See {{bug|773379}}.


* Base and Updates (6.3 and higher)
* Base and Updates (6.3 and higher)
CENTOS_MAJOR=6
  CENTOS_FULL=6.5
  CENTOS_FULL=6.3
  rsync -v -n -aP   --filter='-r centos-version.txt'   --exclude isos --exclude drpms --exclude centosplus --exclude xen4 --exclude fasttrack --exclude contrib --exclude cr \
DATE=2012-07-12
    --delete --delete-excluded rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_FULL/
  rsync -v -n -aP \
    --filter='-r updates/Source' --filter='-r os/Source' --filter='-r centos-version.txt'\
    --exclude isos --exclude drpms \
    --delete --delete-excluded \
    rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/
rsync -v -n -aP \
    rsync://mirror.nsc.liu.se/centos-store/$CENTOS_FULL/os/Source /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/os/Source
rsync -v -n -aP \
    rsync://mirror.nsc.liu.se/centos-store/$CENTOS_FULL/updates/Source /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/updates/Source
echo $CENTOS_FULL > /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/centos-version.txt
  time hardlink -v /data/repos/yum/mirrors/centos
  time hardlink -v /data/repos/yum/mirrors/centos


* sync these at "nearly" the same time, so that we have a good chance of having the srpm for a particular RPM
Note that this pulls along 'SCL' and 'extras' and some other stuff.  As long as it's not huge, it doesn't hurt.
* when re-mirroring, it may save a bit of time and space to use rsync's hard-link capabilities to copy the last date to the new date. This can be done with the --link-dest option:
  OLDDATE=2012-07-12
  NEWDATE=2012-09-06
  mkdir /data/repos/yum/mirrors/centos/6/$NEWDATE/
  rsync -v -aP --link-dest=/data/repos/yum/mirrors/centos/6/$OLDDATE/ /data/repos/yum/mirrors/centos/6/$OLDDATE/ /data/repos/yum/mirrors/centos/6/$NEWDATE/


==== Fedora 16 ====
==== Fedora 16 ====
Line 119: Line 100:
  rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/dependencies/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/dependencies/i386/
  rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/dependencies/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/dependencies/i386/
  time hardlink -v /data/repos/yum/mirrors/puppetlabs/
  time hardlink -v /data/repos/yum/mirrors/puppetlabs/
Puppet-2.7.18 and 2.7.19 do not work with the SSL certificate chaining (https://projects.puppetlabs.com/issues/15561), so those RPMs have been removed and `createrepo .` run to remove the evidence.


==== Passenger ====
==== Passenger ====
Line 127: Line 106:
  rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/
  rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/
  time hardlink -v /data/repos/yum/mirrors/passenger/
  time hardlink -v /data/repos/yum/mirrors/passenger/
==== node.js ====
# sync nodejs 6.10.0 packages
mkdir -p /data/repos/yum/mirrors/nodesource/el/6/x86_64 && cd $_
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-6.10.0-1nodesource.el6.x86_64.rpm
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-debuginfo-6.10.0-1nodesource.el6.x86_64.rpm
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-devel-6.10.0-1nodesource.el6.x86_64.rpm
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-docs-6.10.0-1nodesource.el6.noarch.rpm
createrepo /data/repos/yum/mirrors/nodesource/el/6/x86_64/
time hardlink -v /data/repos/yum/mirrors/nodesource/
# sync C++11 packages (required by some nodejs libs)
mkdir -p /data/repos/yum/mirrors/devtools-2/6/x86_64/RPMS && cd $_
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-gcc-4.8.2-15.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-gcc-c++-4.8.2-15.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-libstdc++-devel-4.8.2-15.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-runtime-2.1-4.el6.noarch.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/scl-utils-20120927-8.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-binutils-2.23.52.0.1-10.el6.x86_64.rpm
createrepo /data/repos/yum/mirrors/devtools-2/6/x86_64/RPMS/
time hardlink -v /data/repos/yum/mirrors/devtools-2/


==== HP ====
==== HP ====
Line 135: Line 136:
  time hardlink -v /data/repos/yum/mirrors/hp/
  time hardlink -v /data/repos/yum/mirrors/hp/


== Installing Packages with Puppet ==
== CentOS: Installing Packages with Puppet ==


CentOS Packages are simple to install:
CentOS Packages are simple to install:
Line 148: Line 149:
Generally, if it's important enough to install explicitly, it's important enough to pin a particular version.  If you also need to pin versions for requirements, be sure you model the requirements with ''requires'' in puppet, so that puppet knows to install the requirements first.
Generally, if it's important enough to install explicitly, it's important enough to pin a particular version.  If you also need to pin versions for requirements, be sure you model the requirements with ''requires'' in puppet, so that puppet knows to install the requirements first.


== Custom Packages ==
== CentOS: Building Custom Packages ==


See [[ReleaseEngineering/PuppetAgain/HowTo/Build RPMs]] for building RPMs.  Custom-built packages should be placed in the appropriate ''repos/yum/releng/public/CentOS/6/*'' repository, depending on architecture.  All dependencies should be included in that repository if they are not in the mirrored repositories.
See [[ReleaseEngineering/PuppetAgain/HowTo/Build RPMs]] for building RPMs.


Before landing the patch, update the documentation on [[ReleaseEngineering/PuppetAgain/Modules/packages]].
== CentOS: Adding New Packages ==


When the patch containing the new or updated package spec is r+'d, commit it as usual, and also add *both* the RPM (or multiple RPMs if multiple architectures are required!) and the SRPM into /data on the designated puppet master (releng-puppet1.srv.releng.scl3), so that it will be distributed to other systems.  Debuginfo RPMs are a good idea, too.
In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (''yum install http://wherever.it.is/package.rpm'') on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories.  If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems.  Upgrading ''librsync'' may be OK, but upgrading ''glibc'' or ''libopenssl'' this way might lead to a world of pain and sadness (noting that security releases often don't!).  In the event that you need to add custom-built packages, you should know from the building process what the requirements are; otherwise it's pretty much the same process.


== Updated Package Versions ==
Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch.  This provides a backout path for you later to install exactly the versions that were installed before your changes.  Only when that change is deployed, add the new packages to the releng repository and run ''createrepo'' (below).  When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.


In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (''yum install http://wherever.it.is/package.rpm'') on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories.  If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems.  Upgrading ''librsync'' may be OK, but upgrading ''glibc'' or ''libopenssl'' this way will lead to a world of pain and sadness.
Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose.  For example, if you're updating openssh, create a new {{PuppetAgain Repo|repos/yum/custom/openssh}}. If the appropriate repo already exists, use it. Try to include SRPMs as well!  When setting up a custom repo, it's helpful to include an `update.sh` script in the root of the repo that can be used to re-mirror it later.  Check other update scripts for useful techniques (for example, `repotrack` is pretty useful!)


Before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch.  This provides a backout path for you later to install exactly the versions that were installed before your changes.  Only when that change is deployed, add the new packages to the repository and run ''createrepo'' (below).  When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.
Once you've assembled a directory containing the proper packages, run
  createrepo --update .
in that directory to update the metadata.  Don't forget to run ''puppetmaster-fixperms'' afterward to make sure permissions are correct.


== Landing Custom Repository Changes ==
If you added a new repository, you'll need to refer to it from the puppet configs.  Add a clause to ''modules/packages/manifests/setup.pp'', either a regular ''packages::yumrepo'' if the repo should be avialable on every host (like openssh) or a virtual one (prefixed with ''@'') if it should only be available on some hosts.  Only installing the repo on some hosts limits the carnage if something goes wrong with the repo.


Run
Then, write or update the classes under ''modules/packages''.  If your repository is virtual, you'll need to add something like ''realize(Packages::Yumrepo['passenger'])'' to the package class to ensure the repo is in place.
  createrepo --update $repo_path
to update the metadata.  Don't forget to run ''puppetagain-fixperms'' to make sure permissions are correct.


Once this is done, the package is available and will be used if possible.
{{note|Packages used in the mock environment require both i686 and x86_64 packages to be in the x86_64 repo, just like upstream}}


Example session:
If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp:
<pre>
<pre>
# at releng-puppet2.srv.releng.scl3.mozilla.com
  # to flush the package index, increase this value by one (or
$ wget http://people.mozilla.org/~jhopkins/bug772446/supervisor-3.0-0.10.a12.el6.noarch.rpm
  # anything, really, just change it).  
$ wget http://people.mozilla.org/~jhopkins/bug772446/supervisor-3.0-0.10.a12.el6.src.rpm
- $repoflag = 5
$ chmod 644 *.rpm
+ $repoflag = 6
$ sudo chown puppetagainsync:puppetagainsync *.rpm
$ sudo mv -vi *.rpm /data/repos/yum/releng/public/CentOS/6/noarch/
`supervisor-3.0-0.10.a12.el6.noarch.rpm' -> `/data/repos/yum/releng/public/CentOS/6/noarch/supervisor-3.0-0.10.a12.el6.noarch.rpm'
`supervisor-3.0-0.10.a12.el6.src.rpm' -> `/data/repos/yum/releng/public/CentOS/6/noarch/supervisor-3.0-0.10.a12.el6.src.rpm'
$ sudo -u puppetagainsync createrepo --update /data/repos/yum/releng/public/CentOS/6/noarch
2/2 - supervisor-3.0-0.10.a12.el6.src.rpm                                     
Saving Primary metadata
Saving file lists metadata
Saving other metadata
$ sudo puppetmaster-fixperms
</pre>
</pre>
This will cause all CentOS machines to run ''yum cache clean all''.


= Ubuntu =
= Ubuntu =
== Reference Links ==
* https://wiki.debian.org/RepositoryFormat
* [https://projetos.ossystems.com.br/projects/debpartial-mirror/repository/revisions/master/entry/doc/README debpartial-mirror README]


== Repositories ==
== Ubuntu: Repositories ==
<table class="wikitable">
<table class="wikitable">
<tr><th>url</th><th>repository</th><th>arch</th><th>section</th><th>dist</th><th>mirror date</th></tr>
<tr><th>url</th><th>repository</th><th>arch</th><th>section</th><th>dist</th><th>mirror date</th></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/ubuntu repos/apt/ubuntu]</td><td>Ubuntu 12.04 LTS</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise,precise-security<br>(note: no precise-updates)</td><td>2013-02-21</td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/ubuntu}}</td><td>Ubuntu 12.04 and 14.04</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise,precise-security,trusty,trusty-security<br>(note: no precise-updates)</td><td>varies</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/xorg-edgers repos/apt/xorg-edgers]</td><td>[https://launchpad.net/~xorg-edgers/+archive/ppa xorg-edgers fresh X Crack]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td>2013-02-21</td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/xorg-edgers}}</td><td>[https://launchpad.net/~xorg-edgers/+archive/ppa xorg-edgers fresh X Crack]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td>2013-02-21</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng repos/apt/releng]</td><td>custom-built packages</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/releng}}</td><td>custom-built packages</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng-updates repos/apt/releng-updates]</td><td>[[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs|partial mirror of precise-updates]]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/releng-updates}}</td><td>[[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs|partial mirror of precise-updates]]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng-updates repos/apt/puppetlabs]</td><td>mirror of apt.puppetlabs.com</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/puppetlabs}}</td><td>mirror of apt.puppetlabs.com</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/precise-updates}}</td><td>mirror of precise-updates</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise-updates</td><td></td></tr>
</table>
</table>


Line 209: Line 205:


==== Setup ====
==== Setup ====
GNUPGHOME has the Ubuntu arch key in it:
GNUPGHOME has the Ubuntu arch key in it.  If a sync operation fails because a signature does not verify, download the key using


  GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys 437D05B5 4BD6EC30
  GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg \
  --keyserver keyserver.ubuntu.com --recv-keys $KEY_ID
 
Add `--keyserver-options http-proxy=proxy.dmz.scl3.mozilla.com:3128` at Mozilla.  Note that this keyserver's search option appears to be broken.  You can usually google for the key id, and find the relevant link on the keyserver, and then copy-paste the result into
 
GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import


with `/etc/debmirror.conf` containing only the Perl no-op "1;".
with `/etc/debmirror.conf` containing only the Perl no-op "1;".
Line 218: Line 219:


==== Ubuntu ====
==== Ubuntu ====
We mirror all Ubuntu releases to the same directory, using --nocleanup to prevent deletion of packages not touched in the current mirror operation. Note that we mirror 'universe', too.  Although it's huge, puppet relies on some packages in that section.
For example, the following mirrors precise.  The DIST can be changed to e.g., mirror a different version, or just mirror security.


  SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer
  SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer
Line 225: Line 230:
     -a $ARCH -s $SECTION -d $DIST \
     -a $ARCH -s $SECTION -d $DIST \
     -h us.archive.ubuntu.com -r /ubuntu -e rsync --progress \
     -h us.archive.ubuntu.com -r /ubuntu -e rsync --progress \
    --nocleanup \
     --dry-run \
     --dry-run \
     /data/repos/apt/ubuntu/
     /data/repos/apt/ubuntu/
Line 237: Line 243:
     -h ppa.launchpad.net -r /xorg-edgers/ppa/ubuntu --rsync-extra=none -e http --progress \
     -h ppa.launchpad.net -r /xorg-edgers/ppa/ubuntu --rsync-extra=none -e http --progress \
     --dry-run \
     --dry-run \
    --nocleanup \
     /data/repos/apt/xorg-edgers/
     /data/repos/apt/xorg-edgers/
==== nginx-development ====
SECTION=main
DIST=precise,trusty
ARCH=i386,amd64
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
    -a $ARCH -s $SECTION -d $DIST \
    -h ppa.launchpad.net -r /nginx/development/ubuntu --rsync-extra=none -e http --progress \
    --dry-run \
    --nocleanup \
    /data/repos/apt/nginx-development/


==== puppetlabs ====
==== puppetlabs ====
  wget https://apt.puppetlabs.com/pubkey.gpg
  GNUPGHOME=/etc/debmirror-gpg  gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import pubkey.gpg


   SECTION=main,dependencies
   SECTION=main,dependencies
   DIST=precise
   DIST=precise,trusty
   ARCH=i386,amd64
   ARCH=i386,amd64
   GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
   GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
Line 250: Line 272:
     /data/repos/apt/puppetlabs/
     /data/repos/apt/puppetlabs/


== Installing Packages with Puppet ==
== Ubuntu: Installing Packages with Puppet ==


Simple:
Simple:
Line 256: Line 278:
         Ubuntu: {
         Ubuntu: {
             package {
             package {
                 "mypackage"
                 "mypackage":
                     ensure => "1.2.3";
                     ensure => "1.2.3";
             }
             }
Line 263: Line 285:
As with CentOS packages, it's generally a good idea to pin the version of things that are important enough to be named in puppet.  If requirements need to be pinned too, then model the requirements relationship properly with ''requires''.
As with CentOS packages, it's generally a good idea to pin the version of things that are important enough to be named in puppet.  If requirements need to be pinned too, then model the requirements relationship properly with ''requires''.


== Custom Packages ==
== Ubuntu: Building Custom Packages ==


See [[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs]] for details on building DEBs.
See [[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs]] for details on building DEBs.


{{todo|describe a means of recording package particulars in Mercurial}}
== Ubuntu: Adding New Packages ==


== Updated Package Versions ==
{{warning|This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the custom repositories) and packages installed by puppet (which does).}}


{{todo|need a reliable method of accomplishing this without accidentally upgrading everything}}
Sometimes you may need to update only one package from upstream without syncing the whole repo.  Or you may have custom-built packages that you need to install.  In either case, the tool for the job is a new, custom, repository.


{{warning|This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the releng-updates repository) and packages installed by puppet (which does).}}
Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch.  This provides a backout path for you later to install exactly the versions that were installed before your changes.  Only when that change is deployed, add the new packages to the releng repository and run ''createrepo'' (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.


Sometimes you may need to update only one package from upstream without syncing the whole repo (what may cause unpredictable results). The releng-updates repo is set up for these cases.
Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose.  For example, if you're updating openssh, create a new {{PuppetAgain Repo|repos/apt/custom/openssh}}.  If the appropriate repo already exists, use it.
 
To build a custom repository, start by laying out your package pool.  The best plan is to divide the packages by dist.  For example:
<pre>
[root@releng-puppet2.srv.releng.scl3.mozilla.com openssl]# find pool/
pool/
pool/trusty
pool/trusty/openssl_1.0.1f-1ubuntu2.5_i386.deb
pool/trusty/openssl_1.0.1.orig.tar.gz
pool/trusty/libssl1.0.0-dbg_1.0.1f-1ubuntu2.5_i386.deb
...
pool/precise
pool/precise/libssl1.0.0_1.0.1-4ubuntu5.17_i386.deb
...
</pre>
Then edit an update.sh in the root of the repository with something along the lines of
<pre>
for arch in i386 amd64; do
  for dist in precise trusty; do
    mkdir -p dists/${dist}/all/binary-$arch
    dpkg-scanpackages --multiversion --arch $arch pool/$dist > dists/${dist}/all/binary-$arch/Packages
    bzip2 < dists/${dist}/all/binary-$arch/Packages > dists/${dist}/all/binary-$arch/Packages.bz2
  done
done
</pre>
This script will update the apt indexes.  Note that the `--multiversion` is required if the repo is to contain multiple versions of the same package, like yum repositories can.
 
Don't forget to run ''puppetmaster-fixperms'' afterward to make sure permissions are correct.
 
If you added a new repository, you'll need to refer to it from the puppet configs.  Add a clause to ''modules/packages/manifests/setup.pp'', either a regular ''packages::aptrepo'' if the repo should be avialable on every host (like openssh) or a virtual one (prefixed with ''@'') if it should only be available on some hosts.  Only installing the repo on some hosts limits the carnage if something goes wrong with the repo.
 
Then, write or update the classes under ''modules/packages''.  If your repository is virtual, you'll need to add something like ''realize(Packages::Aptrepo['xorg-edgers'])'' to the package class to ensure the repo is in place.
 
If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp:
<pre>
  # to flush the package index, increase this value by one (or
  # anything, really, just change it).
- $repoflag = 5
+ $repoflag = 6
</pre>
This will cause all Ubuntu machines to run apt-get update.
 
=== Automatically Pulling Dependencies ===
 
{{note|This is not at all clear, sorry -- there are no real experts on this topic at Mozilla, so learn what you can and update the wiki!}}


This needs to be done on an Ubuntu machine.  You need to use [http://wiki.debian.org/DebPartialMirror debpartial-mirror], apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes:
This needs to be done on an Ubuntu machine.  You need to use [http://wiki.debian.org/DebPartialMirror debpartial-mirror], apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes:
Line 289: Line 355:
done
done
</pre>
</pre>
[http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng-updates.conf releng-updates.conf]
{{PuppetAgain Repo|releng-updates.conf}}
<pre>
<pre>
[GLOBAL]
[GLOBAL]
Line 311: Line 377:
filter = name:gnome-settings-daemon
filter = name:gnome-settings-daemon
</pre>
</pre>
= Landing Custom Repository Changes =
{{note|be careful with this not-yet-ready-to-production version.}}
We use [http://mirrorer.alioth.debian.org/ reprepro] to manage packages.
ATM, the repo lives in /data/repos/apt on releng-puppet2.srv.releng.scl3.
There are 2 important files under <tt>conf</tt> directory:
* options
<pre>
verbose
outdir +b/releng
logdir +b/logs
dbdir +b/db
morguedir +b/morgue
keepunreferencedfiles
keepunusednewfiles
</pre>
* distributions
<pre>
Origin: mozilla
Label: Mozilla
Codename: precise
Version: 12.04
Architectures: amd64 i386 source
Components: main
Description: Releng repos
DebIndices: Packages Release . .bz2
</pre>
Until we have a shared location (an Ubuntu machine) for package building and repo updates (using incoming directory) you need to sync that directory to your machine.
Example import:
<pre>
cd repo
reprepro -V --basedir . include precise ~/debs/puppet/build-area/puppet_2.7.17-1mozilla1_amd64.changes
</pre>
It will copy packages to releng/, generate indices and update the database. Make sure to not get into a race condition with other people.
When you are done with the repo rsync it back, then publish:
sudo chown puppetagainsync:puppetagainsync -R .
sudo rsync -av ./ /data/repos/apt/


= Darwin =
= Darwin =


== Repositories ==
== Darwin: Repositories ==


There's no such thing as a repository for OS X packages, sadly.  DMGs are stored in [http://puppetagain.pub.build.mozilla.org/data/repos/DMGs repos/DMGs].  DMGs are generally built for a specific OS version and put in per-os-version subdirectories, although DMGs that are compatible across versions are in the root.  Each DMG is named $packagename-$version.dmg.
There's no such thing as a repository for OS X packages, sadly.  DMGs are stored in {{PuppetAgain Repo|repos/DMGs}}.  DMGs are generally built for a specific OS version and put in per-os-version subdirectories, although DMGs that are compatible across versions are in the root.  Each DMG is named $packagename-$version.dmg.


== Installing Packages with Puppet ==
== Darwin: Installing Packages with Puppet ==


Use the ''[[ReleaseEngineering/PuppetAgain/Modules/packages#Darwin|packages]]::pkgdmg'' defined type to install DMGs, giving the package name as the resource name and the package version in the ''version'' parameter.  The type will construct the correct filename from this information.  For example:
Use the ''[[ReleaseEngineering/PuppetAgain/Modules/packages#Darwin|packages]]::pkgdmg'' defined type to install DMGs, giving the package name as the resource name and the package version in the ''version'' parameter.  The type will construct the correct filename from this information.  For example:
Line 390: Line 411:
For DMGs that are *not* os-version-specific, pass <tt>os_version_specific => false</tt>.
For DMGs that are *not* os-version-specific, pass <tt>os_version_specific => false</tt>.


== Custom Packages ==
== Darwin: Custom Packages ==


DMGs that are custom built should have a shell script in [http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests modules/packages/manifests] named $package-dmg.sh which builds the DMG.  If there is a corresponding RPM (custom or stock) for the package, then the shell script can require that the source RPM be unpacked first.  See [[ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs]] for more details.
DMGs that are custom built should have a shell script in [http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests modules/packages/manifests] named $package-dmg.sh which builds the DMG.  If there is a corresponding RPM (custom or stock) for the package, then the shell script can require that the source RPM be unpacked first.  See [[ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs]] for more details.
Grenade
22

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/699857...1163967"