ReleaseEngineering/PuppetAgain/Repositories: Difference between revisions

Latest revision as of 21:43, 2 April 2013

The repos data tree contains package repositories. These are a whole complicated ball of wax. See ReleaseEngineering/PuppetAgain/Packages for details.

@@ Line 1: / Line 1: @@
-= Yum Repositories =
+The [http://puppetagain.pub.build.mozilla.org/data/repos repos] [[ReleaseEngineering/PuppetAgain/Data|data]] tree contains package repositories.  These are a whole complicated ball of wax.  See [[ReleaseEngineering/PuppetAgain/Packages]] for details.
-<table>
-<tr><th>url</th><th>repository</th><th>arch</th><th>mirror date</th></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/epel/6/latest/x86_64 repos/yum/mirrors/epel/6/latest/x86_64]</td><td>EPEL 6</td><td>x86_64</td><td>2012-03-07</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/epel/6/latest/i386 repos/yum/mirrors/epel/6/latest/i386]</td><td>EPEL 6</td><td>i386</td><td>2012-03-07*</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/x86_64 repos/yum/mirrors/centos/6/latest/os/x86_64]</td><td>CentOS 6 Base</td><td>x86_64</td><td>2012-03-07</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/i386 repos/yum/mirrors/centos/6/latest/os/i386]</td><td>CentOS 6 Base</td><td>i386</td><td>2012-03-07*</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/x86_64 repos/yum/mirrors/centos/6/latest/updates/x86_64]</td><td>CentOS 6 Updates</td><td>x86_64</td><td>2012-03-07</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/updates/i386 repos/yum/mirrors/centos/6/latest/updates/i386]</td><td>CentOS 6 Updates</td><td>i386</td><td>2012-03-07*</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/releases/Everything/i386/os repos/yum/mirrors/fedora/16/latest/releases/Everything/i386/os]</td><td>Fedora 16 Base</td><td>i386</td><td></td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/releases/Everything/x86_64/os repos/yum/mirrors/fedora/16/latest/releases/Everything/x86_64/os]</td><td>Fedora 16 Base</td><td>x86_64</td><td></td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/updates/i386 repos/yum/mirrors/fedora/16/latest/updates/i386]</td><td>Fedora 16 Updates</td><td>i386</td><td>2012-03-07</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/fedora/16/latest/updates/x86_64 repos/yum/mirrors/fedora/16/latest/updates/x86_64]</td><td>Fedora 16 Updates</td><td>x86_64</td><td>2012-03-07</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64 repos/yum/mirrors/puppetlabs/el/6/products/x86_64]</td><td>Puppetlabs</td><td>x86_64</td><td></td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/passenger/rhel/6/latest/x86_64 repos/yum/mirrors/passenger/rhel/6/latest/x86_64]</td><td>Passenger</td><td>x86_64</td><td>2012-07-05</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/proliantsupportpack/CentOS/6/i386/current/ repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/]</td><td>HP Proliant Support</td><td>i386</td><td>2012-08-21</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/proliantsupportpack/CentOS/6/x86_64/current/ repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/]</td><td>HP Proliant Support</td><td>x86_64</td><td>2012-08-21</td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/noarch repos/yum/releng/public/CentOS/6/noarch]</td><td>Releng CentOS 6 Custom RPMs</td><td>noarch</td><td></td></tr>
-<tr><td>[http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64 repos/yum/releng/public/CentOS/6/x86_64]</td><td>Releng CentOS 6 Custom RPMs</td><td>x86_64</td><td></td></tr>
-</table>
-== Sync Commands ==
-=== EPEL ===
- rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/x86_64/ /data/repos/yum/mirrors/epel/6/2012-03-07/x86_64/
- rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/i386/ /data/repos/yum/mirrors/epel/6/2012-03-07/i386/
-Note that, as for CentOS below, the i386 mirrors were done much later than 2012-03-07 (July 2012).
-=== CentOS 6 ===
-* Base and Updates (6.2, x86_64)
- rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://mirrors.usc.edu/centos/6/os/x86_64/ /data/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64
- rsync -n --no-p -rv --exclude=drpms rsync://mirrors.usc.edu/centos/6/updates/x86_64/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/x86_64
-This was badly botched.  Among other things, group information was omitted from the repodata.
-* Base and Updates (6.2, i386)
- rsync -n --exclude isolinux -aP rsync://linux.mirrors.es.net/centos/6.2/os/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/i386/
-  rsync -n --exclude drpms -aP rsync://linux.mirrors.es.net/centos/6.2/updates/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/os/i386/
-NOTE: the i386 CentOS 6.2 repos were mirrored in July of 2012, but are in the 2012-03-07 directory.  See {{bug|773379}}.
-* Base and Updates (6.3 and higher)
- CENTOS_MAJOR=6
- CENTOS_FULL=6.3
- DATE=2012-07-12
- rsync -v -n -aP \
-    --filter='-r updates/Source' --filter='-r os/Source' --filter='-r centos-version.txt'\
-    --exclude isos --exclude images --exclude drpms \
-    --delete --delete-excluded \
-    rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/
- rsync -v -n -aP \
-    rsync://archive.kernel.org/centos-vault/$CENTOS_FULL/os/Source /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/os/Source
- rsync -v -n -aP \
-    rsync://archive.kernel.org/centos-vault/$CENTOS_FULL/updates/Source /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/updates/Source
- echo $CENTOS_FULL > /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/centos-version.txt
-* sync these at "nearly" the same time, so that we have a good chance of having the srpm for a particular RPM
-* when re-mirroring, it may save a bit of time and space to use rsync's hard-link capabilities to copy the last date to the new date.  This can be done with the --link-dest option, but you'll more likely want to do it on every host (this will save a lot of space!), like so:
-  OLDDATE=2012-07-12
-  NEWDATE=2012-09-06
-  mkdir /data/repos/yum/mirrors/centos/6/$NEWDATE/
-  rsync -v -aP --link-dest=/data/repos/yum/mirrors/centos/6/$OLDDATE/ /data/repos/yum/mirrors/centos/6/$OLDDATE/ /data/repos/yum/mirrors/centos/6/$NEWDATE/
-=== Fedora 16 ===
- rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://mirrors.usc.edu/fedora/linux/releases/16/Everything/i386/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/i386/os/
- rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://fedora.osuosl.org/fedora/linux/releases/16/Everything/x86_64/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/x86_64/os/
- rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/i386/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/i386/
- rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/x86_64/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/x86_64/
-=== PupppetLabs ===
- rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64/
- rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/products/i386/
-Puppet-2.7.18 and 2.7.19 do not work with the SSL certificate chaining (https://projects.puppetlabs.com/issues/15561), so those RPMs have been removed and `createrepo .` run to remove the evidence.
-=== Passenger ===
- rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/x86_64/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/x86_64/
- rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/
-=== hp ===
-not everything is mirrored, so this just gets the good stuff
- rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/x86_64/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/
- rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/i386/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/
-= Notes =
-* Dynamic repositories are snapshots that are made on demand, where <tt>latest</tt> always points to the latest active snapshot.  They are *not* automatically updated.  The date on which they were most recently mirrored is given above.
-* The current CentOS version is available at http://repos/repos/yum/mirrors/centos/6/latest/centos-version.txt.
-* The CentOS base repos each have an images/ subdirectory, so they can be used with the 'url' command in a kickstart script
-= Mac OS X DMGs =
-DMGs are stored in [http://puppetagain.pub.build.mozilla.org/data/repos/DMGs repos/DMGs].  This is a flat directory, unlike the heavily nested RPM directories.  Each DMG is named $packagename-$version.dmg.
-DMGs that are downloaded as-is should be documented briefly (at least with the source) in the manifest files that install them (e.g., [http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests/puppet.pp puppet.pp]).
-DMGs that are custom built should have a shell script in [http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests modules/packages/manifests] named $package-dmg.sh which builds the DMG.  If there is a corresponding RPM (custom or stock) for the package, then the shell script can require that the source RPM be unpacked first.  See [[ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs]] for more details.