ReleaseEngineering/PuppetAgain/Repositories
Jump to navigation
Jump to search
Warning: The debmirror command includes the equivalent of rsync's --delete. If you try to mirror only one section, dist, arch, etc., debmirror will delete the section/dist/arch you omit. Don't do that. See bug 844172.
Mirrored Repositories
Yum Repositories
Apt Repositories
| url | repository | arch | section | dist | mirror date |
|---|---|---|---|---|---|
| repos/apt/ubuntu | Ubuntu 12.04 LTS | i386,amd64 | main,restricted,universe | precise,precise-security (note: no precise-updates) | 2013-02-21 |
| repos/apt/xorg-edgers | xorg-edgers fresh X Crack | i386,amd64 | main,restricted,universe | precise | 2013-02-21 |
| repos/apt/releng | custom-built packages | i386,amd64 | main,restricted,universe | precise | |
| repos/apt/releng-updates | partial mirror of precise-updates | i386,amd64 | main,restricted,universe | precise |
Sync Commands
Note: Note that we generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind
Note: Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space. See bug 836014. The process takes about 10 minutes.
EPEL
rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/x86_64/ /data/repos/yum/mirrors/epel/6/2012-03-07/x86_64/ rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/i386/ /data/repos/yum/mirrors/epel/6/2012-03-07/i386/ time hardlink -v /data/repos/yum/mirrors/epel
Note that, as for CentOS below, the i386 mirrors were done much later than 2012-03-07 (July 2012).
CentOS 6
- Base and Updates (6.2, x86_64)
rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://mirrors.usc.edu/centos/6/os/x86_64/ /data/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64 rsync -n --no-p -rv --exclude=drpms rsync://mirrors.usc.edu/centos/6/updates/x86_64/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/x86_64
This was badly botched. Among other things, group information was omitted from the repodata. Fixing that directly turns out to close the trees. So we have re-mirrored the 6.2 os repos as of 2013-01-17 at `/data/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64-for-ks`.
time hardlink -v /data/repos/yum/mirrors/centos
- Base and Updates (6.2, i386)
rsync -n --exclude isolinux -aP rsync://linux.mirrors.es.net/centos/6.2/os/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/i386/ rsync -n --exclude drpms -aP rsync://linux.mirrors.es.net/centos/6.2/updates/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/os/i386/ time hardlink -v /data/repos/yum/mirrors/centos
NOTE: the i386 CentOS 6.2 repos were mirrored in July of 2012, but are in the 2012-03-07 directory. See bug 773379.
- Base and Updates (6.3 and higher)
CENTOS_MAJOR=6 CENTOS_FULL=6.3 DATE=2012-07-12 rsync -v -n -aP \ --filter='-r updates/Source' --filter='-r os/Source' --filter='-r centos-version.txt'\ --exclude isos --exclude drpms \ --delete --delete-excluded \ rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/ rsync -v -n -aP \ rsync://mirror.nsc.liu.se/centos-store/$CENTOS_FULL/os/Source /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/os/Source rsync -v -n -aP \ rsync://mirror.nsc.liu.se/centos-store/$CENTOS_FULL/updates/Source /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/updates/Source echo $CENTOS_FULL > /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/centos-version.txt time hardlink -v /data/repos/yum/mirrors/centos
- sync these at "nearly" the same time, so that we have a good chance of having the srpm for a particular RPM
- when re-mirroring, it may save a bit of time and space to use rsync's hard-link capabilities to copy the last date to the new date. This can be done with the --link-dest option:
OLDDATE=2012-07-12 NEWDATE=2012-09-06 mkdir /data/repos/yum/mirrors/centos/6/$NEWDATE/ rsync -v -aP --link-dest=/data/repos/yum/mirrors/centos/6/$OLDDATE/ /data/repos/yum/mirrors/centos/6/$OLDDATE/ /data/repos/yum/mirrors/centos/6/$NEWDATE/
Fedora 16
rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://mirrors.usc.edu/fedora/linux/releases/16/Everything/i386/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/i386/os/ rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://fedora.osuosl.org/fedora/linux/releases/16/Everything/x86_64/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/x86_64/os/ rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/i386/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/i386/ rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/x86_64/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/x86_64/ time hardlink -v /data/repos/yum/mirrors/fedora/
PuppetLabs
rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64/ rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/products/i386/ time hardlink -v /data/repos/yum/mirrors/puppetlabs/
Puppet-2.7.18 and 2.7.19 do not work with the SSL certificate chaining (https://projects.puppetlabs.com/issues/15561), so those RPMs have been removed and `createrepo .` run to remove the evidence.
Passenger
rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/x86_64/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/x86_64/ rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/ time hardlink -v /data/repos/yum/mirrors/passenger/
hp
not everything is mirrored, so this just gets the good stuff
rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/x86_64/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/ rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/i386/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/ time hardlink -v /data/repos/yum/mirrors/hp/
Ubuntu
Warning: The debmirror command includes the equivalent of rsync's --delete. If you try to mirror only one section, dist, arch, etc., debmirror will delete the section/dist/arch you omit. Don't do that. See bug 844172.GNUPGHOME has the Ubuntu arch key in it:
GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys 437D05B5
To mirror:
SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer DIST=precise,precise-security ARCH=i386,amd64 GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \ -a $ARCH -s $SECTION -d $DIST \ -h us.archive.ubuntu.com -r /ubuntu -e rsync --progress \ --dry-run \ /data/repos/apt/ubuntu/
with `/etc/debmirror.conf` containing only the Perl no-op "1;".
Note that `hardlink` isn't used here - debmirror's name-based deduplication squeezes out just about all the space possible.�
xorg-edgers
SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer DIST=precise ARCH=i386,amd64 GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \ -a $ARCH -s $SECTION -d $DIST \ -h ppa.launchpad.net -r /xorg-edgers/ppa/ubuntu --rsync-extra=none -e http --progress \ --dry-run \ /data/repos/apt/xorg-edgers/
Notes
- Dynamic repositories are snapshots that are made on demand, where latest always points to the latest active snapshot. They are *not* automatically updated. The date on which they were most recently mirrored is given above.
- The current CentOS version is available at http://repos/repos/yum/mirrors/centos/6/latest/centos-version.txt.
- The CentOS base repos each have an images/ subdirectory, so they can be used with the 'url' command in a kickstart script
Mac OS X DMGs
DMGs are stored in repos/DMGs. This is a flat directory, unlike the heavily nested RPM directories. Each DMG is named $packagename-$version.dmg.
DMGs that are downloaded as-is should be documented briefly (at least with the source) in the manifest files that install them (e.g., puppet.pp).
DMGs that are custom built should have a shell script in modules/packages/manifests named $package-dmg.sh which builds the DMG. If there is a corresponding RPM (custom or stock) for the package, then the shell script can require that the source RPM be unpacked first. See ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs for more details.