canmove, Confirmed users
1,394
edits
ReleaseEngineering/PuppetAgain/Packages: Difference between revisions
ReleaseEngineering/PuppetAgain/Packages (view source)
Revision as of 19:34, 25 July 2014
, 25 July 2014no edit summary
No edit summary |
|||
| Line 68: | Line 68: | ||
=== Mirror Synchronization Commands === | === Mirror Synchronization Commands === | ||
{{note|Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space. See {{bug|836014}}. The process takes about 10 minutes.}} | {{note|Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space. See {{bug|836014}}. The process takes about 10 minutes.}} | ||
{{note|From inside Mozilla, you'll need to use the Squid proxy: set | {{note|From inside Mozilla, you'll need to use the Squid proxy: set RSYNC_PROXY to proxy.dmz.scl3.mozilla.com:3128}} | ||
Remember that once changes land on the puppet master, they can be used for any purpose that needs packages. That includes kickstart, which always takes the most recent version of a package it can find! It also includes installs by puppet that specify ''version => "latest"'', and installs to fulfill requirements of packages defined by puppet. Think twice! | Remember that once changes land on the puppet master, they can be used for any purpose that needs packages. That includes kickstart, which always takes the most recent version of a package it can find! It also includes installs by puppet that specify ''version => "latest"'', and installs to fulfill requirements of packages defined by puppet. Think twice! | ||
| Line 134: | Line 134: | ||
See [[ReleaseEngineering/PuppetAgain/HowTo/Build RPMs]] for building RPMs. | See [[ReleaseEngineering/PuppetAgain/HowTo/Build RPMs]] for building RPMs. | ||
== CentOS: | == CentOS: Adding New Packages == | ||
In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (''yum install http://wherever.it.is/package.rpm'') on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories. If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems. Upgrading ''librsync'' may be OK, but upgrading ''glibc'' or ''libopenssl'' this way might lead to a world of pain and sadness (noting that security releases often don't!). | In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (''yum install http://wherever.it.is/package.rpm'') on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories. If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems. Upgrading ''librsync'' may be OK, but upgrading ''glibc'' or ''libopenssl'' this way might lead to a world of pain and sadness (noting that security releases often don't!). In the event that you need to add custom-built packages, you should know from the building process what the requirements are; otherwise it's pretty much the same process. | ||
Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run ''createrepo'' (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production. | Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run ''createrepo'' (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production. | ||
| Line 151: | Line 151: | ||
{{note|Packages used in the mock environment require both i686 and x86_64 packages to be in the x86_64 repo, just like upstream}} | {{note|Packages used in the mock environment require both i686 and x86_64 packages to be in the x86_64 repo, just like upstream}} | ||
If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp: | |||
<pre> | |||
# to flush the package index, increase this value by one (or | |||
# anything, really, just change it). | |||
- $repoflag = 5 | |||
+ $repoflag = 6 | |||
</pre> | |||
This will cause all CentOS machines to run ''yum cache clean all''. | |||
= Ubuntu = | = Ubuntu = | ||
| Line 160: | Line 169: | ||
<table class="wikitable"> | <table class="wikitable"> | ||
<tr><th>url</th><th>repository</th><th>arch</th><th>section</th><th>dist</th><th>mirror date</th></tr> | <tr><th>url</th><th>repository</th><th>arch</th><th>section</th><th>dist</th><th>mirror date</th></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/apt/ubuntu}}</td><td>Ubuntu 12.04 | <tr><td>{{PuppetAgain Repo|repos/apt/ubuntu}}</td><td>Ubuntu 12.04 and 14.04</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise,precise-security,trusty,trusty-security<br>(note: no precise-updates)</td><td>varies</td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/apt/xorg-edgers}}</td><td>[https://launchpad.net/~xorg-edgers/+archive/ppa xorg-edgers fresh X Crack]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td>2013-02-21</td></tr> | <tr><td>{{PuppetAgain Repo|repos/apt/xorg-edgers}}</td><td>[https://launchpad.net/~xorg-edgers/+archive/ppa xorg-edgers fresh X Crack]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td>2013-02-21</td></tr> | ||
<tr><td>{{PuppetAgain Repo|repos/apt/releng}}</td><td>custom-built packages</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr> | <tr><td>{{PuppetAgain Repo|repos/apt/releng}}</td><td>custom-built packages</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td></td></tr> | ||
| Line 247: | Line 256: | ||
See [[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs]] for details on building DEBs. | See [[ReleaseEngineering/PuppetAgain/HowTo/Build DEBs]] for details on building DEBs. | ||
== Ubuntu: | == Ubuntu: Adding New Packages == | ||
{{warning|This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the | {{warning|This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the custom repositories) and packages installed by puppet (which does).}} | ||
Sometimes you may need to update only one package from upstream without syncing the whole repo | Sometimes you may need to update only one package from upstream without syncing the whole repo. Or you may have custom-built packages that you need to install. In either case, the tool for the job is a new, custom, repository. | ||
Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run ''createrepo'' (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production. | |||
( | Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose. For example, if you're updating openssh, create a new {{PuppetAgain Repo|repos/apt/custom/openssh}}. If the appropriate repo already exists, use it. | ||
To build a custom repository, you'll need an Ubuntu system with apt-utils installed. A package builder host will do the trick, but the host's details aren't important. Set up a directory containing the proper packages in the normal apt pool/* format, run | |||
<pre> | <pre> | ||
rm -rf | rm -rf dists | ||
cd releng-updates | cd releng-updates | ||
for arch in i386 amd64; do | for arch in i386 amd64; do | ||
| Line 276: | Line 278: | ||
done | done | ||
</pre> | </pre> | ||
in that directory to update the apt indexes -- in this case for both precise (12.04) and trusty (14.04). Don't forget to run ''puppetmaster-fixperms'' afterward to make sure permissions are correct. | |||
If you added a new repository, you'll need to refer to it from the puppet configs. Add a clause to ''modules/packages/manifests/setup.pp'', either a regular ''packages::aptrepo'' if the repo should be avialable on every host (like openssh) or a virtual one (prefixed with ''@'') if it should only be available on some hosts. Only installing the repo on some hosts limits the carnage if something goes wrong with the repo. | |||
Then, write or update the classes under ''modules/packages''. If your repository is virtual, you'll need to add something like ''realize(Packages::Aptrepo['xorg-edgers'])'' to the package class to ensure the repo is in place. | |||
If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp: | |||
<pre> | |||
# to flush the package index, increase this value by one (or | |||
# anything, really, just change it). | |||
- $repoflag = 5 | |||
+ $repoflag = 6 | |||
</pre> | |||
This will cause all Ubuntu machines to run apt-get update. | |||
=== Automatically Pulling Dependencies === | === Automatically Pulling Dependencies === | ||
{{note|This is not at all clear, sorry -- there are no real experts on this topic at Mozilla, so learn what you can and update the wiki!}} | |||
This needs to be done on an Ubuntu machine. You need to use [http://wiki.debian.org/DebPartialMirror debpartial-mirror], apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes: | This needs to be done on an Ubuntu machine. You need to use [http://wiki.debian.org/DebPartialMirror debpartial-mirror], apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes: | ||
| Line 313: | Line 331: | ||
filter = name:gnome-settings-daemon | filter = name:gnome-settings-daemon | ||
</pre> | </pre> | ||
= Darwin = | = Darwin = | ||