MozillaWiki

ReleaseEngineering/PuppetAgain/Packages: Difference between revisions

Page Discussion

ReleaseEngineering/PuppetAgain/Packages (view source)

Revision as of 14:55, 28 February 2017

2,535 bytes added ,  28 February 2017
→‎node.js devtoolset-2-binutils
 
(38 intermediate revisions by 6 users not shown)
Line 1: Line 1:
= Principles =
= Requirements =


PuppetAgain needs to be able to repeatably produce identical machines over long time ranges (years).  That means carefully controlling which versions of which packages are available to be installed, both during machine creation and in runs of the puppet agent.  However, it's impractical and unnecessary to specify the exact version of *every* package installed on a host.
PuppetAgain needs to be able to repeatably produce identical machines over long time ranges (years).  That means carefully controlling which versions of which packages are available to be installed, both during machine creation and in runs of the puppet agent.  However, it's impractical and unnecessary to specify the exact version of *every* package installed on a host.
Line 8: Line 8:


How all of this happens differs widely among supported operating systems!
How all of this happens differs widely among supported operating systems!
= Principles =
* Do not allow the copying of a package to a repository to be the event that deploys a change -- that's un-trackable and difficult to revert
* Repositories form a strong isolation boundary: if you modify repository A, it's easy to guarantee that repository B hasn't changed.  Repositories are cheap, so use lots of them.
* A repository's update cadence is part of its definition, so define that up front.
* Repositories should be for use by puppet only.  Other consumers, such as mock environments, docker, etc., will have a different required update cadence and are unlikely to implement the proper resiliency to mirror failure, and thus should use repositories hosted elsewhere.


= CentOS =
= CentOS =
Line 17: Line 24:
<table class="wikitable">
<table class="wikitable">
<tr><th>url</th><th>repository</th><th>arch</th><th>mirror date</th></tr>
<tr><th>url</th><th>repository</th><th>arch</th><th>mirror date</th></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/epel/6/latest/x86_64 repos/yum/mirrors/epel/6/latest/x86_64]</td><td>EPEL 6</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/epel/6/latest/x86_64}}</td><td>EPEL 6</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/epel/6/latest/i386 repos/yum/mirrors/epel/6/latest/i386]</td><td>EPEL 6</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/epel/6/latest/i386}}</td><td>EPEL 6</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/x86_64 repos/yum/mirrors/centos/6/latest/os/x86_64]</td><td>CentOS 6 Base</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/os/x86_64}}</td><td>CentOS 6.2 Base</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/i386 repos/yum/mirrors/centos/6/latest/os/i386]</td><td>CentOS 6 Base</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/os/i386}}</td><td>CentOS 6.2 Base</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/Source repos/yum/mirrors/centos/6/latest/os/Source]</td><td>CentOS 6 Base</td><td>source</td><td>2012-09-06</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/os/Source}}</td><td>CentOS 6.2 Base</td><td>source</td><td>2012-09-06</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/x86_64 repos/yum/mirrors/centos/6/latest/updates/x86_64]</td><td>CentOS 6 Updates</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/updates/x86_64}}</td><td>CentOS 6.2 Updates</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/i386 repos/yum/mirrors/centos/6/latest/updates/i386]</td><td>CentOS 6 Updates</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/updates/i386}}</td><td>CentOS 6.2 Updates</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/Source repos/yum/mirrors/centos/6/latest/updates/Source]</td><td>CentOS 6 Updates</td><td>source</td><td>2012-09-06</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.2/latest/updates/Source}}</td><td>CentOS 6.2 Updates</td><td>source</td><td>2012-09-06</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/releases/Everything/i386/os repos/yum/mirrors/fedora/16/latest/releases/Everything/i386/os]</td><td>Fedora 16 Base</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/os/x86_64}}</td><td>CentOS 6.5 Base</td><td>x86_64</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/releases/Everything/x86_64/os repos/yum/mirrors/fedora/16/latest/releases/Everything/x86_64/os]</td><td>Fedora 16 Base</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/os/i386}}</td><td>CentOS 6.5 Base</td><td>i386</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/updates/i386 repos/yum/mirrors/fedora/16/latest/updates/i386]</td><td>Fedora 16 Updates</td><td>i386</td><td>2012-03-07</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/extras/x86_64}}</td><td>CentOS 6.5 Extras</td><td>x86_64</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/updates/x86_64 repos/yum/mirrors/fedora/16/latest/updates/x86_64]</td><td>Fedora 16 Updates</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/extras/i386}}</td><td>CentOS 6.5 Extras</td><td>i386</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64 repos/yum/mirrors/puppetlabs/el/6/products/x86_64]</td><td>Puppetlabs</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/updates/x86_64}}</td><td>CentOS 6.5 Updates</td><td>x86_64</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/puppetlabs/el/6/products/i386 repos/yum/mirrors/puppetlabs/el/6/products/i386]</td><td>Puppetlabs</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/updates/i386}}</td><td>CentOS 6.5 Updates</td><td>i386</td><td>2013-12-??</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/puppetlabs/el/6/dependencies/x86_64 repos/yum/mirrors/puppetlabs/el/6/dependencies/x86_64]</td><td>Puppetlabs Deps</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/products/x86_64}}</td><td>Puppetlabs</td><td>x86_64</td><td>as necessary</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/puppetlabs/el/6/dependencies/i386 repos/yum/mirrors/puppetlabs/el/6/dependencies/i386]</td><td>Puppetlabs Deps</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/products/i386}}</td><td>Puppetlabs</td><td>i386</td><td>as necessary</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/passenger/rhel/6/latest/x86_64 repos/yum/mirrors/passenger/rhel/6/latest/x86_64]</td><td>Passenger</td><td>x86_64</td><td>2012-07-05</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/dependencies/x86_64}}</td><td>Puppetlabs Deps</td><td>x86_64</td><td>as necessary</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/proliantsupportpack/CentOS/6/i386/current/ repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/]</td><td>HP Proliant Support</td><td>i386</td><td>2012-08-21</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/dependencies/i386}}</td><td>Puppetlabs Deps</td><td>i386</td><td>as necessary</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/proliantsupportpack/CentOS/6/x86_64/current/ repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/]</td><td>HP Proliant Support</td><td>x86_64</td><td>2012-08-21</td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/passenger/rhel/6/latest/x86_64}}</td><td>Passenger</td><td>x86_64</td><td>2012-07-05</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/noarch repos/yum/releng/public/CentOS/6/noarch]</td><td>Releng CentOS 6 Custom RPMs</td><td>noarch</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/}}</td><td>HP Proliant Support</td><td>i386</td><td>2012-08-21</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64 repos/yum/releng/public/CentOS/6/x86_64]</td><td>Releng CentOS 6 Custom RPMs</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/}}</td><td>HP Proliant Support</td><td>x86_64</td><td>2012-08-21</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/i386 repos/yum/releng/public/CentOS/6/i386]</td><td>Releng CentOS 6 Custom RPMs</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/CentOS/6/noarch}}</td><td>Releng CentOS 6 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>noarch</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/Fedora/16/noarch repos/yum/releng/public/Fedora/16/noarch]</td><td>Releng Fedora 16 Custom RPMs</td><td>noarch</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/CentOS/6/x86_64}}</td><td>Releng CentOS 6 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>x86_64</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/Fedora/16/x86_64 repos/yum/releng/public/Fedora/16/x86_64]</td><td>Releng Fedora 16 Custom RPMs</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/CentOS/6/i386}}</td><td>Releng CentOS 6 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>i386</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/Fedora/16/i386 repos/yum/releng/public/Fedora/16/i386]</td><td>Releng Fedora 16 Custom RPMs</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/Fedora/16/noarch}}</td><td>Releng Fedora 16 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>noarch</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/Fedora/16/x86_64}}</td><td>Releng Fedora 16 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>x86_64</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/Fedora/16/i386}}</td><td>Releng Fedora 16 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>i386</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/yum/custom/*}}</td><td>Custom repositories</td><td>*</td><td></td></tr>
</table>
</table>


Notes:
Notes:
* Paths under ''yum/releng'' are custom packages, and are not mirrored from anywhere.
* Repos under ''repos/yum/custom'' are custom-built for some purpose, and are not mirrored from anywhere.  They may contain custom packages, or just packages culled from some mirror.  Look for references to the repos in the puppet manifests to find their purpose.
* We generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind.
* We generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind.
* The Fedora repos are only used within ''mock'' on buildslaves.  They are not used to install anything on puppet-managed hosts.
* The ''repos/yum/releng'' repos are deprecated; if you're tempted to add a package there, please create a new custom repository or update another existing custom repository instead
** The releng Fedora repos are currently empty, in fact
* The Fedora repos are unused and will be deleted soon
* Dynamic repositories are snapshots that are made on demand, where <tt>latest</tt> always points to the latest active snapshot.  They are *not* automatically updated.  The date on which they were most recently mirrored is given above.
* Dynamic repositories are snapshots that are made on demand, where <tt>latest</tt> always points to the latest active snapshot.  They are *not* automatically updated.  The date on which they were most recently mirrored is given above.
* The current CentOS version is available at http://repos/repos/yum/mirrors/centos/6/latest/centos-version.txt.
* The current CentOS version is available at http://repos/repos/yum/mirrors/centos/6/latest/centos-version.txt.
Line 69: Line 79:


* Base and Updates (6.3 and higher)
* Base and Updates (6.3 and higher)
CENTOS_MAJOR=6
  CENTOS_FULL=6.5
  CENTOS_FULL=6.5
DATE=2013-12-03
  rsync -v -n -aP   --filter='-r centos-version.txt'   --exclude isos --exclude drpms --exclude centosplus --exclude xen4 --exclude fasttrack --exclude contrib --exclude cr \
  rsync -v -n -aP \
    --delete --delete-excluded rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_FULL/
    --filter='-r updates/Source' --filter='-r os/Source' --filter='-r centos-version.txt'\
    --exclude isos --exclude drpms --exclude centosplus --exclude xen4 --exclude fasttrack --exclude contrib --exclude cr \
    --delete --delete-excluded \
    rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/
rsync -v -n -aP \
    rsync://archive.kernel.org/centos/$CENTOS_FULL/os/Source/SPackages/ /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/os/source/
rsync -v -n -aP \
    rsync://archive.kernel.org/centos/$CENTOS_FULL/updates/Source/SPackages/ /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/updates/source/
echo $CENTOS_FULL > /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/centos-version.txt
  time hardlink -v /data/repos/yum/mirrors/centos
  time hardlink -v /data/repos/yum/mirrors/centos


* when re-mirroring, it may save a bit of time and space to use rsync's hard-link capabilities to copy the last date to the new date. This can be done with the --link-dest option:
Note that this pulls along 'SCL' and 'extras' and some other stuff.  As long as it's not huge, it doesn't hurt.
  OLDDATE=2012-07-12
  NEWDATE=2012-09-06
  mkdir /data/repos/yum/mirrors/centos/6/$NEWDATE/
  rsync -v -aP --link-dest=/data/repos/yum/mirrors/centos/6/$OLDDATE/ /data/repos/yum/mirrors/centos/6/$OLDDATE/ /data/repos/yum/mirrors/centos/6/$NEWDATE/


==== Fedora 16 ====
==== Fedora 16 ====
Line 110: Line 106:
  rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/
  rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/
  time hardlink -v /data/repos/yum/mirrors/passenger/
  time hardlink -v /data/repos/yum/mirrors/passenger/
==== node.js ====
# sync nodejs 6.10.0 packages
mkdir -p /data/repos/yum/mirrors/nodesource/el/6/x86_64 && cd $_
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-6.10.0-1nodesource.el6.x86_64.rpm
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-debuginfo-6.10.0-1nodesource.el6.x86_64.rpm
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-devel-6.10.0-1nodesource.el6.x86_64.rpm
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-docs-6.10.0-1nodesource.el6.noarch.rpm
createrepo /data/repos/yum/mirrors/nodesource/el/6/x86_64/
time hardlink -v /data/repos/yum/mirrors/nodesource/
# sync C++11 packages (required by some nodejs libs)
mkdir -p /data/repos/yum/mirrors/devtools-2/6/x86_64/RPMS && cd $_
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-gcc-4.8.2-15.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-gcc-c++-4.8.2-15.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-libstdc++-devel-4.8.2-15.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-runtime-2.1-4.el6.noarch.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/scl-utils-20120927-8.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-binutils-2.23.52.0.1-10.el6.x86_64.rpm
createrepo /data/repos/yum/mirrors/devtools-2/6/x86_64/RPMS/
time hardlink -v /data/repos/yum/mirrors/devtools-2/


==== HP ====
==== HP ====
Line 131: Line 149:
Generally, if it's important enough to install explicitly, it's important enough to pin a particular version.  If you also need to pin versions for requirements, be sure you model the requirements with ''requires'' in puppet, so that puppet knows to install the requirements first.
Generally, if it's important enough to install explicitly, it's important enough to pin a particular version.  If you also need to pin versions for requirements, be sure you model the requirements with ''requires'' in puppet, so that puppet knows to install the requirements first.


== CentOS: Custom Packages ==
== CentOS: Building Custom Packages ==


See [[ReleaseEngineering/PuppetAgain/HowTo/Build RPMs]] for building RPMs.
See [[ReleaseEngineering/PuppetAgain/HowTo/Build RPMs]] for building RPMs.


== CentOS: Updated Versions of Upstream Packages ==
== CentOS: Adding New Packages ==


In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (''yum install http://wherever.it.is/package.rpm'') on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories.  If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems.  Upgrading ''librsync'' may be OK, but upgrading ''glibc'' or ''libopenssl'' this way will lead to a world of pain and sadness.
In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (''yum install http://wherever.it.is/package.rpm'') on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories.  If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems.  Upgrading ''librsync'' may be OK, but upgrading ''glibc'' or ''libopenssl'' this way might lead to a world of pain and sadness (noting that security releases often don't!).  In the event that you need to add custom-built packages, you should know from the building process what the requirements are; otherwise it's pretty much the same process.


Before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch.  This provides a backout path for you later to install exactly the versions that were installed before your changes.  Only when that change is deployed, add the new packages to the repository and run ''createrepo'' (below).  When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.
Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch.  This provides a backout path for you later to install exactly the versions that were installed before your changes.  Only when that change is deployed, add the new packages to the releng repository and run ''createrepo'' (below).  When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.


== CentOS: Landing Custom Repository Changes ==
Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose.  For example, if you're updating openssh, create a new {{PuppetAgain Repo|repos/yum/custom/openssh}}.  If the appropriate repo already exists, use it.  Try to include SRPMs as well!  When setting up a custom repo, it's helpful to include an `update.sh` script in the root of the repo that can be used to re-mirror it later.  Check other update scripts for useful techniques (for example, `repotrack` is pretty useful!)


Custom-built packages should be placed in the appropriate ''repos/yum/releng/public/CentOS/6/*'' repository, depending on architecture.  All dependencies should be included in that repository if they are not in the mirrored repositories.
Once you've assembled a directory containing the proper packages, run
  createrepo --update .
in that directory to update the metadata.  Don't forget to run ''puppetmaster-fixperms'' afterward to make sure permissions are correct.


When the patch containing the new or updated package spec is r+'d, commit it as usual, and also add *both* the RPM (or multiple RPMs if multiple architectures are required!) and the SRPM into /data on the designated puppet master, so that it will be distributed to other systems.  Debuginfo RPMs are a good idea, too.
If you added a new repository, you'll need to refer to it from the puppet configs.  Add a clause to ''modules/packages/manifests/setup.pp'', either a regular ''packages::yumrepo'' if the repo should be avialable on every host (like openssh) or a virtual one (prefixed with ''@'') if it should only be available on some hosts.  Only installing the repo on some hosts limits the carnage if something goes wrong with the repo.


Run
Then, write or update the classes under ''modules/packages''.  If your repository is virtual, you'll need to add something like ''realize(Packages::Yumrepo['passenger'])'' to the package class to ensure the repo is in place.
  createrepo --update $repo_path
to update the metadata.  Don't forget to run ''puppetmaster-fixperms'' to make sure permissions are correct.


Once this is done, the package is available and will be used if possible.
{{note|Packages used in the mock environment require both i686 and x86_64 packages to be in the x86_64 repo, just like upstream}}


Example session:
If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp:
<pre>
<pre>
# at releng-puppet2.srv.releng.scl3.mozilla.com
  # to flush the package index, increase this value by one (or
$ wget http://people.mozilla.org/~jhopkins/bug772446/supervisor-3.0-0.10.a12.el6.noarch.rpm
  # anything, really, just change it).  
$ wget http://people.mozilla.org/~jhopkins/bug772446/supervisor-3.0-0.10.a12.el6.src.rpm
- $repoflag = 5
$ chmod 644 *.rpm
+ $repoflag = 6
$ sudo chown puppetsync: *.rpm
$ sudo mv -vi *.rpm /data/repos/yum/releng/public/CentOS/6/noarch/
`supervisor-3.0-0.10.a12.el6.noarch.rpm' -> `/data/repos/yum/releng/public/CentOS/6/noarch/supervisor-3.0-0.10.a12.el6.noarch.rpm'
`supervisor-3.0-0.10.a12.el6.src.rpm' -> `/data/repos/yum/releng/public/CentOS/6/noarch/supervisor-3.0-0.10.a12.el6.src.rpm'
$ sudo createrepo --update /data/repos/yum/releng/public/CentOS/6/noarch
2/2 - supervisor-3.0-0.10.a12.el6.src.rpm                                     
Saving Primary metadata
Saving file lists metadata
Saving other metadata
# set permissions and ownership on repodata and package files
$ sudo puppetmaster-fixperms
</pre>
</pre>
This will cause all CentOS machines to run ''yum cache clean all''.


= Ubuntu =
= Ubuntu =
== Reference Links ==
* https://wiki.debian.org/RepositoryFormat
* [https://projetos.ossystems.com.br/projects/debpartial-mirror/repository/revisions/master/entry/doc/README debpartial-mirror README]


== Ubuntu: Repositories ==
== Ubuntu: Repositories ==
<table class="wikitable">
<table class="wikitable">
<tr><th>url</th><th>repository</th><th>arch</th><th>section</th><th>dist</th><th>mirror date</th></tr>
<tr><th>url</th><th>repository</th><th>arch</th><th>section</th><th>dist</th><th>mirror date</th></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/ubuntu repos/apt/ubuntu]</td><td>Ubuntu 12.04 LTS</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise,precise-security<br>(note: no precise-updates)</td><td>2013-02-21</td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/ubuntu}}</td><td>Ubuntu 12.04 and 14.04</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise,precise-security,trusty,trusty-security<br>(note: no precise-updates)</td><td>varies</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/xorg-edgers repos/apt/xorg-edgers]</td><td>[https://launchpad.net/~xorg-edgers/+archive/ppa xorg-edgers fresh X Crack]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td>2013-02-21</td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/xorg-edgers}}</td><td>[https://launchpad.net/~xorg-edgers/+archive/ppa xorg-edgers fresh X Crack]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td>2013-02-21</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng repos/apt/releng]</td><td>custom-built packages</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/releng}}</td><td>custom-built packages</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng-updates repos/apt/releng-updates]</td><td>[[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs|partial mirror of precise-updates]]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/releng-updates}}</td><td>[[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs|partial mirror of precise-updates]]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng-updates repos/apt/puppetlabs]</td><td>mirror of apt.puppetlabs.com</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/puppetlabs}}</td><td>mirror of apt.puppetlabs.com</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr>
<tr><td>{{PuppetAgain Repo|repos/apt/precise-updates}}</td><td>mirror of precise-updates</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise-updates</td><td></td></tr>
</table>
</table>


Line 193: Line 205:


==== Setup ====
==== Setup ====
GNUPGHOME has the Ubuntu arch key in it:
GNUPGHOME has the Ubuntu arch key in it.  If a sync operation fails because a signature does not verify, download the key using


  GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys 437D05B5 4BD6EC30
  GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg \
  --keyserver keyserver.ubuntu.com --recv-keys $KEY_ID


{{note|this doesn't work anymore - those keys have been removed..}}
Add `--keyserver-options http-proxy=proxy.dmz.scl3.mozilla.com:3128` at Mozilla.  Note that this keyserver's search option appears to be broken.  You can usually google for the key id, and find the relevant link on the keyserver, and then copy-paste the result into


Add `--keyserver-options http-proxy=proxy.dmz.scl3.mozilla.com:3128` at Mozilla.
GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import


with `/etc/debmirror.conf` containing only the Perl no-op "1;".
with `/etc/debmirror.conf` containing only the Perl no-op "1;".
Line 206: Line 219:


==== Ubuntu ====
==== Ubuntu ====
We mirror all Ubuntu releases to the same directory, using --nocleanup to prevent deletion of packages not touched in the current mirror operation. Note that we mirror 'universe', too.  Although it's huge, puppet relies on some packages in that section.
For example, the following mirrors precise.  The DIST can be changed to e.g., mirror a different version, or just mirror security.


  SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer
  SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer
Line 213: Line 230:
     -a $ARCH -s $SECTION -d $DIST \
     -a $ARCH -s $SECTION -d $DIST \
     -h us.archive.ubuntu.com -r /ubuntu -e rsync --progress \
     -h us.archive.ubuntu.com -r /ubuntu -e rsync --progress \
    --nocleanup \
     --dry-run \
     --dry-run \
     /data/repos/apt/ubuntu/
     /data/repos/apt/ubuntu/
Line 225: Line 243:
     -h ppa.launchpad.net -r /xorg-edgers/ppa/ubuntu --rsync-extra=none -e http --progress \
     -h ppa.launchpad.net -r /xorg-edgers/ppa/ubuntu --rsync-extra=none -e http --progress \
     --dry-run \
     --dry-run \
    --nocleanup \
     /data/repos/apt/xorg-edgers/
     /data/repos/apt/xorg-edgers/
==== nginx-development ====
SECTION=main
DIST=precise,trusty
ARCH=i386,amd64
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
    -a $ARCH -s $SECTION -d $DIST \
    -h ppa.launchpad.net -r /nginx/development/ubuntu --rsync-extra=none -e http --progress \
    --dry-run \
    --nocleanup \
    /data/repos/apt/nginx-development/


==== puppetlabs ====
==== puppetlabs ====
Line 232: Line 263:
   GNUPGHOME=/etc/debmirror-gpg  gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import pubkey.gpg
   GNUPGHOME=/etc/debmirror-gpg  gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import pubkey.gpg


   SECTION=main,dependencies \
   SECTION=main,dependencies
   DIST=precise \
   DIST=precise,trusty
   ARCH=i386,amd64 \
   ARCH=i386,amd64
   GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
   GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
     -a $ARCH -s $SECTION -d $DIST \
     -a $ARCH -s $SECTION -d $DIST \
Line 254: Line 285:
As with CentOS packages, it's generally a good idea to pin the version of things that are important enough to be named in puppet.  If requirements need to be pinned too, then model the requirements relationship properly with ''requires''.
As with CentOS packages, it's generally a good idea to pin the version of things that are important enough to be named in puppet.  If requirements need to be pinned too, then model the requirements relationship properly with ''requires''.


== Ubuntu: Custom Packages ==
== Ubuntu: Building Custom Packages ==


See [[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs]] for details on building DEBs.
See [[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs]] for details on building DEBs.


== Ubuntu: Updated Versions of Upstream Packages ==
== Ubuntu: Adding New Packages ==
 
{{warning|This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the custom repositories) and packages installed by puppet (which does).}}
 
Sometimes you may need to update only one package from upstream without syncing the whole repo.  Or you may have custom-built packages that you need to install.  In either case, the tool for the job is a new, custom, repository.
 
Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch.  This provides a backout path for you later to install exactly the versions that were installed before your changes.  Only when that change is deployed, add the new packages to the releng repository and run ''createrepo'' (below).  When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.
 
Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose.  For example, if you're updating openssh, create a new {{PuppetAgain Repo|repos/apt/custom/openssh}}.  If the appropriate repo already exists, use it.
 
To build a custom repository, start by laying out your package pool.  The best plan is to divide the packages by dist.  For example:
<pre>
[root@releng-puppet2.srv.releng.scl3.mozilla.com openssl]# find pool/
pool/
pool/trusty
pool/trusty/openssl_1.0.1f-1ubuntu2.5_i386.deb
pool/trusty/openssl_1.0.1.orig.tar.gz
pool/trusty/libssl1.0.0-dbg_1.0.1f-1ubuntu2.5_i386.deb
...
pool/precise
pool/precise/libssl1.0.0_1.0.1-4ubuntu5.17_i386.deb
...
</pre>
Then edit an update.sh in the root of the repository with something along the lines of
<pre>
for arch in i386 amd64; do
  for dist in precise trusty; do
    mkdir -p dists/${dist}/all/binary-$arch
    dpkg-scanpackages --multiversion --arch $arch pool/$dist > dists/${dist}/all/binary-$arch/Packages
    bzip2 < dists/${dist}/all/binary-$arch/Packages > dists/${dist}/all/binary-$arch/Packages.bz2
  done
done
</pre>
This script will update the apt indexes.  Note that the `--multiversion` is required if the repo is to contain multiple versions of the same package, like yum repositories can.


{{todo|need a reliable method of accomplishing this without accidentally upgrading everything}}
Don't forget to run ''puppetmaster-fixperms'' afterward to make sure permissions are correct.


{{warning|This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the releng-updates repository) and packages installed by puppet (which does).}}
If you added a new repository, you'll need to refer to it from the puppet configs.  Add a clause to ''modules/packages/manifests/setup.pp'', either a regular ''packages::aptrepo'' if the repo should be avialable on every host (like openssh) or a virtual one (prefixed with ''@'') if it should only be available on some hosts.  Only installing the repo on some hosts limits the carnage if something goes wrong with the repo.


Sometimes you may need to update only one package from upstream without syncing the whole repo (what may cause unpredictable results). The releng-updates repo is set up for these cases.
Then, write or update the classes under ''modules/packages''.  If your repository is virtual, you'll need to add something like ''realize(Packages::Aptrepo['xorg-edgers'])'' to the package class to ensure the repo is in place.
 
If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp:
<pre>
  # to flush the package index, increase this value by one (or
  # anything, really, just change it).  
- $repoflag = 5
+ $repoflag = 6
</pre>
This will cause all Ubuntu machines to run apt-get update.
 
=== Automatically Pulling Dependencies ===
 
{{note|This is not at all clear, sorry -- there are no real experts on this topic at Mozilla, so learn what you can and update the wiki!}}


This needs to be done on an Ubuntu machine.  You need to use [http://wiki.debian.org/DebPartialMirror debpartial-mirror], apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes:
This needs to be done on an Ubuntu machine.  You need to use [http://wiki.debian.org/DebPartialMirror debpartial-mirror], apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes:
Line 278: Line 355:
done
done
</pre>
</pre>
[http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng-updates.conf releng-updates.conf]
{{PuppetAgain Repo|releng-updates.conf}}
<pre>
<pre>
[GLOBAL]
[GLOBAL]
Line 300: Line 377:
filter = name:gnome-settings-daemon
filter = name:gnome-settings-daemon
</pre>
</pre>
= Ubuntu: Landing Custom Repository Changes =
{{note|This is currently Mozilla-only, since Mozilla is the upstream for all packages.  Be careful, as this can easily wipe out the existing releng repository!}}
{{warning|Debian repositories can only contain one version of a package.  When you land an upgrade into a repo, land the corresponding puppet change that will upgrade the package as soon as possible.  Remember to bump the repo version (below).}}
We use [http://manpages.ubuntu.com/manpages/dapper/man1/reprepro.1.html reprepro] to manage packages.  Nobody's quite sure how to pronounce it.
The repo lives in /data/repos/apt on releng-puppet2.srv.releng.scl3, so you'll need to begin by rsync'ing it to ubuntu64packager1:
  rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/releng/ /data/repos/apt/releng/
  rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/conf/ /data/repos/apt/conf/
  rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/db/ /data/repos/apt/db/
The <tt>conf/</tt> directory contains config files pointing to the releng/ repository only -- this won't (shouldn't!) mess with the Ubuntu mirrors.
Next, import your files into the repo
<pre>
cd /data/repos/apt
reprepro -V --basedir . include precise /tmp/somepackage-precise-amd64/*.changes
</pre>
if you're installing two architectures, then the second one will complain that the debian tarball's signature doesn't match.  That's OK.  Just import the debs from that architecture:
<pre>
reprepro -V --basedir . includedeb precise /tmp/somepackage-precise-i386/*.deb
</pre>
This will copy packages to releng/, generate indices and update the database. Make sure to not get into a race condition with other people on the same host.
When you are done with the repo, first check that any previous versions of the packages you're updating are pinned in puppet.  Then rsync the repo and db back to the distinguished master:
  rsync -av /data/repos/apt/releng/ releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/releng/
  rsync -av /data/repos/apt/db/ releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/db/
and run this on the master:
  puppetmaster-fixperms
You need to bump the counter in modules/packages/manifests/setup.pp:
<pre>
  # to flush the package index, increase this value by one (or
  # anything, really, just change it).
- $repoflag = 5
+ $repoflag = 6
</pre>
This will cause all Ubuntu machines to run apt-get update.


= Darwin =
= Darwin =
Line 351: Line 382:
== Darwin: Repositories ==
== Darwin: Repositories ==


There's no such thing as a repository for OS X packages, sadly.  DMGs are stored in [http://puppetagain.pub.build.mozilla.org/data/repos/DMGs repos/DMGs].  DMGs are generally built for a specific OS version and put in per-os-version subdirectories, although DMGs that are compatible across versions are in the root.  Each DMG is named $packagename-$version.dmg.
There's no such thing as a repository for OS X packages, sadly.  DMGs are stored in {{PuppetAgain Repo|repos/DMGs}}.  DMGs are generally built for a specific OS version and put in per-os-version subdirectories, although DMGs that are compatible across versions are in the root.  Each DMG is named $packagename-$version.dmg.


== Darwin: Installing Packages with Puppet ==
== Darwin: Installing Packages with Puppet ==
Grenade
22

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/946555...1163967"