canmove, Confirmed users
1,394
edits
ReleaseEngineering/PuppetAgain/Repositories: Difference between revisions
Jump to navigation
Jump to search
ReleaseEngineering/PuppetAgain/Repositories (view source)
Revision as of 23:30, 29 January 2013
, 29 January 2013no edit summary
No edit summary |
|||
| Line 38: | Line 38: | ||
{{note|Note that we generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind}} | {{note|Note that we generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind}} | ||
{{note|Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space. See {{bug|836014}}. The process takes about 10 minutes.}} | |||
=== EPEL === | === EPEL === | ||
rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/x86_64/ /data/repos/yum/mirrors/epel/6/2012-03-07/x86_64/ | rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/x86_64/ /data/repos/yum/mirrors/epel/6/2012-03-07/x86_64/ | ||
rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/i386/ /data/repos/yum/mirrors/epel/6/2012-03-07/i386/ | rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/i386/ /data/repos/yum/mirrors/epel/6/2012-03-07/i386/ | ||
time hardlink -v /data/repos/yum/mirrors/epel | |||
Note that, as for CentOS below, the i386 mirrors were done much later than 2012-03-07 (July 2012). | Note that, as for CentOS below, the i386 mirrors were done much later than 2012-03-07 (July 2012). | ||
| Line 52: | Line 54: | ||
rsync -n --no-p -rv --exclude=drpms rsync://mirrors.usc.edu/centos/6/updates/x86_64/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/x86_64 | rsync -n --no-p -rv --exclude=drpms rsync://mirrors.usc.edu/centos/6/updates/x86_64/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/x86_64 | ||
This was badly botched. Among other things, group information was omitted from the repodata. Fixing that directly turns out to close the trees. So we have re-mirrored the 6.2 os repos as of 2013-01-17 at `/data/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64-for-ks`. | This was badly botched. Among other things, group information was omitted from the repodata. Fixing that directly turns out to close the trees. So we have re-mirrored the 6.2 os repos as of 2013-01-17 at `/data/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64-for-ks`. | ||
time hardlink -v /data/repos/yum/mirrors/centos | |||
* Base and Updates (6.2, i386) | * Base and Updates (6.2, i386) | ||
| Line 57: | Line 60: | ||
rsync -n --exclude isolinux -aP rsync://linux.mirrors.es.net/centos/6.2/os/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/i386/ | rsync -n --exclude isolinux -aP rsync://linux.mirrors.es.net/centos/6.2/os/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/updates/i386/ | ||
rsync -n --exclude drpms -aP rsync://linux.mirrors.es.net/centos/6.2/updates/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/os/i386/ | rsync -n --exclude drpms -aP rsync://linux.mirrors.es.net/centos/6.2/updates/i386/ /data/repos/yum/mirrors/centos/6/2012-03-07/os/i386/ | ||
time hardlink -v /data/repos/yum/mirrors/centos | |||
NOTE: the i386 CentOS 6.2 repos were mirrored in July of 2012, but are in the 2012-03-07 directory. See {{bug|773379}}. | NOTE: the i386 CentOS 6.2 repos were mirrored in July of 2012, but are in the 2012-03-07 directory. See {{bug|773379}}. | ||
| Line 74: | Line 78: | ||
rsync://mirror.nsc.liu.se/centos-store/$CENTOS_FULL/updates/Source /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/updates/Source | rsync://mirror.nsc.liu.se/centos-store/$CENTOS_FULL/updates/Source /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/updates/Source | ||
echo $CENTOS_FULL > /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/centos-version.txt | echo $CENTOS_FULL > /data/repos/yum/mirrors/centos/$CENTOS_MAJOR/$DATE/centos-version.txt | ||
time hardlink -v /data/repos/yum/mirrors/centos | |||
* sync these at "nearly" the same time, so that we have a good chance of having the srpm for a particular RPM | * sync these at "nearly" the same time, so that we have a good chance of having the srpm for a particular RPM | ||
* when re-mirroring, it may save a bit of time and space to use rsync's hard-link capabilities to copy the last date to the new date. This can be done with the --link-dest option | * when re-mirroring, it may save a bit of time and space to use rsync's hard-link capabilities to copy the last date to the new date. This can be done with the --link-dest option: | ||
OLDDATE=2012-07-12 | OLDDATE=2012-07-12 | ||
NEWDATE=2012-09-06 | NEWDATE=2012-09-06 | ||
| Line 86: | Line 92: | ||
rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/i386/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/i386/ | rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/i386/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/i386/ | ||
rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/x86_64/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/x86_64/ | rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/x86_64/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/x86_64/ | ||
time hardlink -v /data/repos/yum/mirrors/fedora/ | |||
=== PupppetLabs === | === PupppetLabs === | ||
| Line 91: | Line 98: | ||
rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64/ | rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64/ | ||
rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/products/i386/ | rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/products/i386/ | ||
time hardlink -v /data/repos/yum/mirrors/puppetlabs/ | |||
Puppet-2.7.18 and 2.7.19 do not work with the SSL certificate chaining (https://projects.puppetlabs.com/issues/15561), so those RPMs have been removed and `createrepo .` run to remove the evidence. | Puppet-2.7.18 and 2.7.19 do not work with the SSL certificate chaining (https://projects.puppetlabs.com/issues/15561), so those RPMs have been removed and `createrepo .` run to remove the evidence. | ||
| Line 98: | Line 106: | ||
rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/x86_64/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/x86_64/ | rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/x86_64/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/x86_64/ | ||
rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/ | rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/ | ||
time hardlink -v /data/repos/yum/mirrors/passenger/ | |||
=== hp === | === hp === | ||
| Line 104: | Line 113: | ||
rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/x86_64/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/ | rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/x86_64/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/ | ||
rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/i386/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/ | rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/i386/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/ | ||
time hardlink -v /data/repos/yum/mirrors/hp/ | |||
=== Ubuntu === | === Ubuntu === | ||
| Line 122: | Line 132: | ||
with `/etc/debmirror.conf` containing only the Perl no-op "1;". | with `/etc/debmirror.conf` containing only the Perl no-op "1;". | ||
Note that `hardlink` isn't used here - debmirror's name-based deduplication squeezes out just about all the space possible.� | |||
= Notes = | = Notes = | ||