ReferencePlatforms/Puppet Server

Warning: This page documents (mostly) the old release engineering puppet deployment. See ReleaseEngineering/PuppetAgain for documentation of the current deployment.

Puppet: Usage | Server Setup | Client Setup | Links | Troubleshooting

CentOS 5.x

This page describes how to recreate a Puppet server such as staging-puppet.build.mozilla.org or production-puppet.build.mozilla.org. These machines are based on the CentOS 5.0 Reference Platform and were originally cloned from Version 19 - but any version should do. The following changes were made to it after cloning:

Install the EPEL (Extra Packages for Enterprise Linux) Package

su -c 'rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm'

Install Mercurial

yum install mercurial

Install puppet

Start with dependencies:

yum install ruby facter ruby-shadow augeas-libs ruby-augeas rubygem-mongrel

Until ReleaseEngineering/PuppetAgain, we still use puppet version 0.24.8. These rpms are out-of-date now (as is CentOS 5.x). I found them at http://linux.web.psi.ch/dist/scientific/5/beta. They will likely also be available on one of the existing puppet servers, probably in /root. Catlee also has them on his home dir http://people.mozilla.org/~catlee/centos55-puppet-master/manifests/files/

cd /root
wget http://linux.web.psi.ch/dist/scientific/5/beta/puppet-0.24.8-1.el5.1.noarch.rpm
wget http://linux.web.psi.ch/dist/scientific/5/beta/puppet-server-0.24.8-1.el5.1.noarch.rpm
md5sum puppet-0.24.8-1.el5.1.noarch.rpm 
 0dfaf87a4b6abc753c2727cd1efdf390  puppet-0.24.8-1.el5.1.noarch.rpm
md5sum puppet-server-0.24.8-1.el5.1.noarch.rpm 
 8dc2cf5623c14fdb02575ad953b52e51  puppet-server-0.24.8-1.el5.1.noarch.rpm
 
rpm -Uvh puppet-0.24.8-1.el5.1.noarch.rpm puppet-server-0.24.8-1.el5.1.noarch.rpm

chkconfig --level 235 puppetmaster on

Clone manifests

As root,

cd /etc/puppet
rm -rf manifests
hg clone http://hg.mozilla.org/build/puppet-manifests manifests

Copy packages from an existing puppet server

mkdir /N
cd /N
rsync -navuzb scl-production-puppet.build.scl1.mozilla.com:/N/production .
rsync -navuzb scl-production-puppet.build.scl1.mozilla.com:/N/staging .

NFS mount (deprecated)

NFS turned out to be ineffective between datacenters, so each master is now manually synchronized using the steps outlined in #Copy_packages_from_an_existing_puppet_server. These steps are provided for posterity.

Mount the puppet-files directory by adding an entry to fstab:

10.2.71.136:/export/buildlogs/puppet-files /N   nfs	defaults	0 0

And then run,

mkdir /N
mount -a

to mount it.

Start the puppetmaster daemon

cd /etc/puppet
# Choose the appropriate fileserver config for the new server (or create a new one)
ln -s manifests/scl-production-fileserver.pp fileserver.pp

cd /etc/puppet/manifests
# Choose the appropriate config for the new server (or create a new one)
ln -s scl-production.pp site.pp

# Uncomment the line that reads PUPPETMASTER_PORTS=( 18140 18141 18142 18143 )
vi /etc/sysconfig/puppetmaster

service puppetmaster start

Setup Apache server

yum install httpd
yum install mod_ssl
cd /etc/httpd/conf.d
scp scl-production-puppet.build.scl1.mozilla.com:/etc/httpd/conf.d/puppetmaster.conf .
scp scl-production-puppet.build.scl1.mozilla.com:/etc/httpd/conf.d/ssl.conf .

Generate certs for apache with

puppetca --generate `hostname --fqdn`

Update /etc/httpd/conf.d/puppetmaster.conf to point to

/var/lib/puppet/ssl/certs/$fqdn.pem

Change the SSLCertificateFile and SSLCertificateKeyFile locations to the correct disk locations for your new server.

vi puppetmaster.conf

Edit httpd.conf for so the DocumentRoot and Director point to /N to allow it to serve up the packages.

service httpd start

Install nrpe and add Nagios libs

 yum install nrpe nagios-plugins-nrpe

 [root@staging-puppet nagios]# rpm -qa | grep nrpe 
 nrpe-2.12-16.el5
 nagios-plugins-nrpe-2.12-16.el5

Copy nrpe config from an existing production server

 cp /etc/nagios
 scp cl-production-puppet.build.scl1.mozilla.com:/etc/nagios/nrpe.cfg .

 cp puppet-manifests/modules/nagios/files/check_ganglia /usr/lib64/nagios/plugins/check_ganglia

 chkconfig --level 235 nrpe on

 service nrpe start

Open a bug with relops to monitor the server via Nagios. See bug https://bugzilla.mozilla.org/show_bug.cgi?id=754963 as an example.

Install and configure ganglia

• Installed ganglia, ganglia-gmond, ganglia-gmond-python from rpmfind.net (DAG el5 x86_64 versions)
• Copied gmond.conf from an scl3 master
• Restarted gmond
• chkconfig --level 2345 gmond on

Update root crontab with puppet specific scripts

 */1 * * * * source /etc/profile && bash /etc/puppet/manifests/accept-hostname-keys.sh /etc/puppet/manifests/site.pp 2>/dev/null
 */10 * * * * chown root:root -R /etc/puppet/manfiests &> /dev/null

Verify the new server

Attempt to connect to the new server using a slave.

See https://wiki.mozilla.org/ReleaseEngineering/Puppet/Usage#Testing_a_slave