22
edits
ReleaseEngineering/PuppetAgain/Packages: Difference between revisions
ReleaseEngineering/PuppetAgain/Packages (view source)
Revision as of 14:55, 28 February 2017
, 28 February 2017→node.js devtoolset-2-binutils
| (16 intermediate revisions by 4 users not shown) | |||
| Line 38: | Line 38: | ||
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/updates/x86_64}}</td><td>CentOS 6.5 Updates</td><td>x86_64</td><td>2013-12-??</td></tr> | <tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/updates/x86_64}}</td><td>CentOS 6.5 Updates</td><td>x86_64</td><td>2013-12-??</td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/updates/i386}}</td><td>CentOS 6.5 Updates</td><td>i386</td><td>2013-12-??</td></tr> | <tr><td>{{PuppetAgain Repo|repos/yum/mirrors/centos/6.5/updates/i386}}</td><td>CentOS 6.5 Updates</td><td>i386</td><td>2013-12-??</td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/products/x86_64}}</td><td>Puppetlabs</td><td>x86_64</td><td>as necessary</td></tr> | <tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/products/x86_64}}</td><td>Puppetlabs</td><td>x86_64</td><td>as necessary</td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/products/i386}}</td><td>Puppetlabs</td><td>i386</td><td>as necessary</td></tr> | <tr><td>{{PuppetAgain Repo|repos/yum/mirrors/puppetlabs/el/6/products/i386}}</td><td>Puppetlabs</td><td>i386</td><td>as necessary</td></tr> | ||
| Line 55: | Line 51: | ||
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/Fedora/16/x86_64}}</td><td>Releng Fedora 16 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>x86_64</td><td></td></tr> | <tr><td>{{PuppetAgain Repo|repos/yum/releng/public/Fedora/16/x86_64}}</td><td>Releng Fedora 16 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>x86_64</td><td></td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/yum/releng/public/Fedora/16/i386}}</td><td>Releng Fedora 16 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>i386</td><td></td></tr> | <tr><td>{{PuppetAgain Repo|repos/yum/releng/public/Fedora/16/i386}}</td><td>Releng Fedora 16 Custom RPMs (DO NOT ADD NEW PACKAGES)</td><td>i386</td><td></td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/yum/custom/*}}</td><td>Custom repositories</td><td>*</td><td></td></tr> | |||
</table> | </table> | ||
| Line 68: | Line 65: | ||
=== Mirror Synchronization Commands === | === Mirror Synchronization Commands === | ||
{{note|Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space. See {{bug|836014}}. The process takes about 10 minutes.}} | {{note|Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space. See {{bug|836014}}. The process takes about 10 minutes.}} | ||
{{note|From inside Mozilla, you'll need to use the Squid proxy: set | {{note|From inside Mozilla, you'll need to use the Squid proxy: set RSYNC_PROXY to proxy.dmz.scl3.mozilla.com:3128}} | ||
Remember that once changes land on the puppet master, they can be used for any purpose that needs packages. That includes kickstart, which always takes the most recent version of a package it can find! It also includes installs by puppet that specify ''version => "latest"'', and installs to fulfill requirements of packages defined by puppet. Think twice! | Remember that once changes land on the puppet master, they can be used for any purpose that needs packages. That includes kickstart, which always takes the most recent version of a package it can find! It also includes installs by puppet that specify ''version => "latest"'', and installs to fulfill requirements of packages defined by puppet. Think twice! | ||
| Line 109: | Line 106: | ||
rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/ | rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/ | ||
time hardlink -v /data/repos/yum/mirrors/passenger/ | time hardlink -v /data/repos/yum/mirrors/passenger/ | ||
==== node.js ==== | |||
# sync nodejs 6.10.0 packages | |||
mkdir -p /data/repos/yum/mirrors/nodesource/el/6/x86_64 && cd $_ | |||
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-6.10.0-1nodesource.el6.x86_64.rpm | |||
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-debuginfo-6.10.0-1nodesource.el6.x86_64.rpm | |||
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-devel-6.10.0-1nodesource.el6.x86_64.rpm | |||
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-docs-6.10.0-1nodesource.el6.noarch.rpm | |||
createrepo /data/repos/yum/mirrors/nodesource/el/6/x86_64/ | |||
time hardlink -v /data/repos/yum/mirrors/nodesource/ | |||
# sync C++11 packages (required by some nodejs libs) | |||
mkdir -p /data/repos/yum/mirrors/devtools-2/6/x86_64/RPMS && cd $_ | |||
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-gcc-4.8.2-15.el6.x86_64.rpm | |||
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-gcc-c++-4.8.2-15.el6.x86_64.rpm | |||
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-libstdc++-devel-4.8.2-15.el6.x86_64.rpm | |||
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-runtime-2.1-4.el6.noarch.rpm | |||
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/scl-utils-20120927-8.el6.x86_64.rpm | |||
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-binutils-2.23.52.0.1-10.el6.x86_64.rpm | |||
createrepo /data/repos/yum/mirrors/devtools-2/6/x86_64/RPMS/ | |||
time hardlink -v /data/repos/yum/mirrors/devtools-2/ | |||
==== HP ==== | ==== HP ==== | ||
| Line 134: | Line 153: | ||
See [[ReleaseEngineering/PuppetAgain/HowTo/Build RPMs]] for building RPMs. | See [[ReleaseEngineering/PuppetAgain/HowTo/Build RPMs]] for building RPMs. | ||
== CentOS: | == CentOS: Adding New Packages == | ||
In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (''yum install http://wherever.it.is/package.rpm'') on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories. If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems. Upgrading ''librsync'' may be OK, but upgrading ''glibc'' or ''libopenssl'' this way might lead to a world of pain and sadness (noting that security releases often don't!). | In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (''yum install http://wherever.it.is/package.rpm'') on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories. If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems. Upgrading ''librsync'' may be OK, but upgrading ''glibc'' or ''libopenssl'' this way might lead to a world of pain and sadness (noting that security releases often don't!). In the event that you need to add custom-built packages, you should know from the building process what the requirements are; otherwise it's pretty much the same process. | ||
Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run ''createrepo'' (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production. | Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run ''createrepo'' (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production. | ||
Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose. For example, if you're updating openssh, create a new {{PuppetAgain Repo|repos/yum/custom/openssh}}. If the appropriate repo already exists, use it. Try to include SRPMs as well! | Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose. For example, if you're updating openssh, create a new {{PuppetAgain Repo|repos/yum/custom/openssh}}. If the appropriate repo already exists, use it. Try to include SRPMs as well! When setting up a custom repo, it's helpful to include an `update.sh` script in the root of the repo that can be used to re-mirror it later. Check other update scripts for useful techniques (for example, `repotrack` is pretty useful!) | ||
Once you've assembled a directory containing the proper packages, run | Once you've assembled a directory containing the proper packages, run | ||
| Line 151: | Line 170: | ||
{{note|Packages used in the mock environment require both i686 and x86_64 packages to be in the x86_64 repo, just like upstream}} | {{note|Packages used in the mock environment require both i686 and x86_64 packages to be in the x86_64 repo, just like upstream}} | ||
If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp: | |||
<pre> | |||
# to flush the package index, increase this value by one (or | |||
# anything, really, just change it). | |||
- $repoflag = 5 | |||
+ $repoflag = 6 | |||
</pre> | |||
This will cause all CentOS machines to run ''yum cache clean all''. | |||
= Ubuntu = | = Ubuntu = | ||
| Line 160: | Line 188: | ||
<table class="wikitable"> | <table class="wikitable"> | ||
<tr><th>url</th><th>repository</th><th>arch</th><th>section</th><th>dist</th><th>mirror date</th></tr> | <tr><th>url</th><th>repository</th><th>arch</th><th>section</th><th>dist</th><th>mirror date</th></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/apt/ubuntu}}</td><td>Ubuntu 12.04 | <tr><td>{{PuppetAgain Repo|repos/apt/ubuntu}}</td><td>Ubuntu 12.04 and 14.04</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise,precise-security,trusty,trusty-security<br>(note: no precise-updates)</td><td>varies</td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/apt/xorg-edgers}}</td><td>[https://launchpad.net/~xorg-edgers/+archive/ppa xorg-edgers fresh X Crack]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td>2013-02-21</td></tr> | <tr><td>{{PuppetAgain Repo|repos/apt/xorg-edgers}}</td><td>[https://launchpad.net/~xorg-edgers/+archive/ppa xorg-edgers fresh X Crack]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td>2013-02-21</td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/apt/releng}}</td><td>custom-built packages</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr> | <tr><td>{{PuppetAgain Repo|repos/apt/releng}}</td><td>custom-built packages</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/apt/releng-updates}}</td><td>[[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs|partial mirror of precise-updates]]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr> | <tr><td>{{PuppetAgain Repo|repos/apt/releng-updates}}</td><td>[[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs|partial mirror of precise-updates]]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/apt/puppetlabs}}</td><td>mirror of apt.puppetlabs.com</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr> | <tr><td>{{PuppetAgain Repo|repos/apt/puppetlabs}}</td><td>mirror of apt.puppetlabs.com</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/apt/precise-updates}}</td><td>mirror of precise-updates</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise-updates</td><td></td></tr> | |||
</table> | </table> | ||
| Line 178: | Line 207: | ||
GNUPGHOME has the Ubuntu arch key in it. If a sync operation fails because a signature does not verify, download the key using | GNUPGHOME has the Ubuntu arch key in it. If a sync operation fails because a signature does not verify, download the key using | ||
GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys $KEY_ID | GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg \ | ||
--keyserver keyserver.ubuntu.com --recv-keys $KEY_ID | |||
Add `--keyserver-options http-proxy=proxy.dmz.scl3.mozilla.com:3128` at Mozilla. Note that this keyserver's search option appears to be broken. You can usually google for the key id, and find the relevant link on the keyserver, and then copy-paste the result into | Add `--keyserver-options http-proxy=proxy.dmz.scl3.mozilla.com:3128` at Mozilla. Note that this keyserver's search option appears to be broken. You can usually google for the key id, and find the relevant link on the keyserver, and then copy-paste the result into | ||
| Line 215: | Line 245: | ||
--nocleanup \ | --nocleanup \ | ||
/data/repos/apt/xorg-edgers/ | /data/repos/apt/xorg-edgers/ | ||
==== nginx-development ==== | |||
SECTION=main | |||
DIST=precise,trusty | |||
ARCH=i386,amd64 | |||
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \ | |||
-a $ARCH -s $SECTION -d $DIST \ | |||
-h ppa.launchpad.net -r /nginx/development/ubuntu --rsync-extra=none -e http --progress \ | |||
--dry-run \ | |||
--nocleanup \ | |||
/data/repos/apt/nginx-development/ | |||
==== puppetlabs ==== | ==== puppetlabs ==== | ||
| Line 247: | Line 289: | ||
See [[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs]] for details on building DEBs. | See [[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs]] for details on building DEBs. | ||
== Ubuntu: | == Ubuntu: Adding New Packages == | ||
{{warning|This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the | {{warning|This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the custom repositories) and packages installed by puppet (which does).}} | ||
Sometimes you may need to update only one package from upstream without syncing the whole repo | Sometimes you may need to update only one package from upstream without syncing the whole repo. Or you may have custom-built packages that you need to install. In either case, the tool for the job is a new, custom, repository. | ||
Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run ''createrepo'' (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production. | |||
Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose. For example, if you're updating openssh, create a new {{PuppetAgain Repo|repos/apt/custom/openssh}}. If the appropriate repo already exists, use it. | |||
To build a custom repository, start by laying out your package pool. The best plan is to divide the packages by dist. For example: | |||
<pre> | |||
[root@releng-puppet2.srv.releng.scl3.mozilla.com openssl]# find pool/ | |||
pool/ | |||
pool/trusty | |||
pool/trusty/openssl_1.0.1f-1ubuntu2.5_i386.deb | |||
pool/trusty/openssl_1.0.1.orig.tar.gz | |||
pool/trusty/libssl1.0.0-dbg_1.0.1f-1ubuntu2.5_i386.deb | |||
... | |||
pool/precise | |||
pool/precise/libssl1.0.0_1.0.1-4ubuntu5.17_i386.deb | |||
... | |||
</pre> | |||
Then edit an update.sh in the root of the repository with something along the lines of | |||
<pre> | |||
for arch in i386 amd64; do | |||
for dist in precise trusty; do | |||
mkdir -p dists/${dist}/all/binary-$arch | |||
dpkg-scanpackages --multiversion --arch $arch pool/$dist > dists/${dist}/all/binary-$arch/Packages | |||
bzip2 < dists/${dist}/all/binary-$arch/Packages > dists/${dist}/all/binary-$arch/Packages.bz2 | |||
done | |||
done | |||
</pre> | |||
This script will update the apt indexes. Note that the `--multiversion` is required if the repo is to contain multiple versions of the same package, like yum repositories can. | |||
Don't forget to run ''puppetmaster-fixperms'' afterward to make sure permissions are correct. | |||
( | If you added a new repository, you'll need to refer to it from the puppet configs. Add a clause to ''modules/packages/manifests/setup.pp'', either a regular ''packages::aptrepo'' if the repo should be avialable on every host (like openssh) or a virtual one (prefixed with ''@'') if it should only be available on some hosts. Only installing the repo on some hosts limits the carnage if something goes wrong with the repo. | ||
Then, write or update the classes under ''modules/packages''. If your repository is virtual, you'll need to add something like ''realize(Packages::Aptrepo['xorg-edgers'])'' to the package class to ensure the repo is in place. | |||
If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp: | |||
<pre> | <pre> | ||
# to flush the package index, increase this value by one (or | |||
# anything, really, just change it). | |||
- $repoflag = 5 | |||
+ $repoflag = 6 | |||
</pre> | </pre> | ||
This will cause all Ubuntu machines to run apt-get update. | |||
=== Automatically Pulling Dependencies === | === Automatically Pulling Dependencies === | ||
{{note|This is not at all clear, sorry -- there are no real experts on this topic at Mozilla, so learn what you can and update the wiki!}} | |||
This needs to be done on an Ubuntu machine. You need to use [http://wiki.debian.org/DebPartialMirror debpartial-mirror], apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes: | This needs to be done on an Ubuntu machine. You need to use [http://wiki.debian.org/DebPartialMirror debpartial-mirror], apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes: | ||
| Line 313: | Line 377: | ||
filter = name:gnome-settings-daemon | filter = name:gnome-settings-daemon | ||
</pre> | </pre> | ||
= Darwin = | = Darwin = | ||