ReleaseEngineering/PuppetAgain/Repositories: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Replaced content with "The [http://puppetagain.pub.build.mozilla.org/data/repos repos] data tree contains package repositories. These are a whole complic...")
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
= Mirrored Repositories =
The [http://puppetagain.pub.build.mozilla.org/data/repos repos] [[ReleaseEngineering/PuppetAgain/Data|data]] tree contains package repositories.  These are a whole complicated ball of wax.  See [[ReleaseEngineering/PuppetAgain/Packages]] for details.
== Yum Repositories ==
<table>
<tr><th>url</th><th>repository</th><th>arch</th><th>mirror date</th></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/epel/6/latest/x86_64 repos/yum/mirrors/epel/6/latest/x86_64]</td><td>EPEL 6</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/epel/6/latest/i386 repos/yum/mirrors/epel/6/latest/i386]</td><td>EPEL 6</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/x86_64 repos/yum/mirrors/centos/6/latest/os/x86_64]</td><td>CentOS 6 Base</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/i386 repos/yum/mirrors/centos/6/latest/os/i386]</td><td>CentOS 6 Base</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/Source repos/yum/mirrors/centos/6/latest/os/Source]</td><td>CentOS 6 Base</td><td>source</td><td>2012-09-06</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/x86_64 repos/yum/mirrors/centos/6/latest/updates/x86_64]</td><td>CentOS 6 Updates</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/i386 repos/yum/mirrors/centos/6/latest/updates/i386]</td><td>CentOS 6 Updates</td><td>i386</td><td>2012-03-07*</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/Source repos/yum/mirrors/centos/6/latest/updates/Source]</td><td>CentOS 6 Updates</td><td>source</td><td>2012-09-06</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/releases/Everything/i386/os repos/yum/mirrors/fedora/16/latest/releases/Everything/i386/os]</td><td>Fedora 16 Base</td><td>i386</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/releases/Everything/x86_64/os repos/yum/mirrors/fedora/16/latest/releases/Everything/x86_64/os]</td><td>Fedora 16 Base</td><td>x86_64</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/updates/i386 repos/yum/mirrors/fedora/16/latest/updates/i386]</td><td>Fedora 16 Updates</td><td>i386</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/updates/x86_64 repos/yum/mirrors/fedora/16/latest/updates/x86_64]</td><td>Fedora 16 Updates</td><td>x86_64</td><td>2012-03-07</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64 repos/yum/mirrors/puppetlabs/el/6/products/x86_64]</td><td>Puppetlabs</td><td>x86_64</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/passenger/rhel/6/latest/x86_64 repos/yum/mirrors/passenger/rhel/6/latest/x86_64]</td><td>Passenger</td><td>x86_64</td><td>2012-07-05</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/proliantsupportpack/CentOS/6/i386/current/ repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/]</td><td>HP Proliant Support</td><td>i386</td><td>2012-08-21</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/proliantsupportpack/CentOS/6/x86_64/current/ repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/]</td><td>HP Proliant Support</td><td>x86_64</td><td>2012-08-21</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/noarch repos/yum/releng/public/CentOS/6/noarch]</td><td>Releng CentOS 6 Custom RPMs</td><td>noarch</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64 repos/yum/releng/public/CentOS/6/x86_64]</td><td>Releng CentOS 6 Custom RPMs</td><td>x86_64</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/i386 repos/yum/releng/public/CentOS/6/i386]</td><td>Releng CentOS 6 Custom RPMs</td><td>i386</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/Fedora/16/noarch repos/yum/releng/public/Fedora/16/noarch]</td><td>Releng Fedora 16 Custom RPMs</td><td>noarch</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/Fedora/16/x86_64 repos/yum/releng/public/Fedora/16/x86_64]</td><td>Releng Fedora 16 Custom RPMs</td><td>x86_64</td><td></td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/Fedora/16/i386 repos/yum/releng/public/Fedora/16/i386]</td><td>Releng Fedora 16 Custom RPMs</td><td>i386</td><td></td></tr>
</table>
 
== Apt Repositories ==
 
<table>
<tr><th>url</th><th>repository</th><th>arch</th><th>section</th><th>dist</th><th>mirror date</th></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/ubuntu repos/apt/ubuntu]</td><td>Ubuntu 12.04 LTS</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise,precise-security<br>(note: no precise-updates)</td><td>2013-02-21</td></tr>
<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/apt/xorg-edgers repos/apt/xorg-edgers]</td><td>[https://launchpad.net/~xorg-edgers/+archive/ppa xorg-edgers fresh X Crack]</td><td>i386,amd64</td><td>main,restricted,universe</td><td>precise</td><td>2013-02-21</td></tr>
</table>
 
== Sync Commands ==
 
{{note|Note that we generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind}}
{{note|Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space.  See {{bug|836014}}.  The process takes about 10 minutes.}}
 
=== EPEL ===
rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/x86_64/ /data/repos/yum/mirrors/epel/6/2012-03-07/x86_64/
rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/i386/ /data/repos/yum/mirrors/epel/6/2012-03-07/i386/
time hardlink -v /data/repos/yum/mirrors/epel
 
Note that, as for CentOS below, the i386 mirrors were done much later than 2012-03-07 (July 2012).
 
=== CentOS 6 ===
 
* Base and Updates (6.2, x86_64)
 
rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://mirrors.usc.edu/centos/6/os/x86_64/ /data/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64
rsync -n --no-p -rv --exclude=drpms rsync://mirrors.usc.edu/centos/6/updates/x86_64/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/x86_64
This was badly botched.  Among other things, group information was omitted from the repodata.  Fixing that directly turns out to close the trees.  So we have re-mirrored the 6.2 os repos as of 2013-01-17 at `/data/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64-for-ks`.
time hardlink -v /data/repos/yum/mirrors/centos
 
* Base and Updates (6.2, i386)
 
rsync -n --exclude isolinux -aP rsync://linux.mirrors.es.net/centos/6.2/os/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/i386/
rsync -n --exclude drpms -aP rsync://linux.mirrors.es.net/centos/6.2/updates/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/os/i386/
time hardlink -v /data/repos/yum/mirrors/centos
 
NOTE: the i386 CentOS 6.2 repos were mirrored in July of 2012, but are in the 2012-03-07 directory.  See {{bug|773379}}.
 
* Base and Updates (6.3 and higher)
CENTOS_MAJOR=6
CENTOS_FULL=6.3
DATE=2012-07-12
rsync -v -n -aP \
    --filter='-r updates/Source' --filter='-r os/Source' --filter='-r centos-version.txt'\
    --exclude isos --exclude drpms \
    --delete --delete-excluded \
    rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/
rsync -v -n -aP \
    rsync://mirror.nsc.liu.se/centos-store/$CENTOS_FULL/os/Source /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/os/Source
rsync -v -n -aP \
    rsync://mirror.nsc.liu.se/centos-store/$CENTOS_FULL/updates/Source /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/updates/Source
echo $CENTOS_FULL > /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/centos-version.txt
time hardlink -v /data/repos/yum/mirrors/centos
 
* sync these at "nearly" the same time, so that we have a good chance of having the srpm for a particular RPM
* when re-mirroring, it may save a bit of time and space to use rsync's hard-link capabilities to copy the last date to the new date.  This can be done with the --link-dest option:
  OLDDATE=2012-07-12
  NEWDATE=2012-09-06
  mkdir /data/repos/yum/mirrors/centos/6/$NEWDATE/
  rsync -v -aP --link-dest=/data/repos/yum/mirrors/centos/6/$OLDDATE/ /data/repos/yum/mirrors/centos/6/$OLDDATE/ /data/repos/yum/mirrors/centos/6/$NEWDATE/
 
=== Fedora 16 ===
rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://mirrors.usc.edu/fedora/linux/releases/16/Everything/i386/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/i386/os/
rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://fedora.osuosl.org/fedora/linux/releases/16/Everything/x86_64/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/x86_64/os/
rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/i386/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/i386/
rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/x86_64/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/x86_64/
time hardlink -v /data/repos/yum/mirrors/fedora/
 
=== PuppetLabs ===
 
rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64/
rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/products/i386/
time hardlink -v /data/repos/yum/mirrors/puppetlabs/
 
Puppet-2.7.18 and 2.7.19 do not work with the SSL certificate chaining (https://projects.puppetlabs.com/issues/15561), so those RPMs have been removed and `createrepo .` run to remove the evidence.
 
=== Passenger ===
 
rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/x86_64/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/x86_64/
rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/
time hardlink -v /data/repos/yum/mirrors/passenger/
 
=== hp ===
not everything is mirrored, so this just gets the good stuff
 
rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/x86_64/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/
rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/i386/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/
time hardlink -v /data/repos/yum/mirrors/hp/
 
=== Ubuntu ===
{{warning|The debmirror command includes the equivalent of rsync's <tt>--delete</tt>.  If you try to mirror only one section, dist, arch, etc., debmirror will '''delete''' the section/dist/arch you omit.  Don't do that.  See {{bug|844172}}.}}
GNUPGHOME has the Ubuntu arch key in it:
 
GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys 437D05B5
 
To mirror:
 
SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer
DIST=precise,precise-security
ARCH=i386,amd64
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
    -a $ARCH -s $SECTION -d $DIST \
    -h us.archive.ubuntu.com -r /ubuntu -e rsync --progress \
    --dry-run \
    /data/repos/apt/ubuntu/
 
with `/etc/debmirror.conf` containing only the Perl no-op "1;".
 
Note that `hardlink` isn't used here - debmirror's name-based deduplication squeezes out just about all the space possible.�
 
==== xorg-edgers ====
 
SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer
DIST=precise
ARCH=i386,amd64
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
    -a $ARCH -s $SECTION -d $DIST \
    -h ppa.launchpad.net -r /xorg-edgers/ppa/ubuntu --rsync-extra=none -e http --progress \
    --dry-run \
    /data/repos/apt/xorg-edgers/
 
= Notes =
* Dynamic repositories are snapshots that are made on demand, where <tt>latest</tt> always points to the latest active snapshot.  They are *not* automatically updated.  The date on which they were most recently mirrored is given above.
* The current CentOS version is available at http://repos/repos/yum/mirrors/centos/6/latest/centos-version.txt.
* The CentOS base repos each have an images/ subdirectory, so they can be used with the 'url' command in a kickstart script
 
= Mac OS X DMGs =
DMGs are stored in [http://puppetagain.pub.build.mozilla.org/data/repos/DMGs repos/DMGs].  This is a flat directory, unlike the heavily nested RPM directories.  Each DMG is named $packagename-$version.dmg.
 
DMGs that are downloaded as-is should be documented briefly (at least with the source) in the manifest files that install them (e.g., [http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests/puppet.pp puppet.pp]).
 
DMGs that are custom built should have a shell script in [http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests modules/packages/manifests] named $package-dmg.sh which builds the DMG.  If there is a corresponding RPM (custom or stock) for the package, then the shell script can require that the source RPM be unpacked first.  See [[ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs]] for more details.

Latest revision as of 21:43, 2 April 2013

The repos data tree contains package repositories. These are a whole complicated ball of wax. See ReleaseEngineering/PuppetAgain/Packages for details.

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/PuppetAgain/Repositories&oldid=643455"