Note: You only need to bootstrap the first master in an organization. Subsequent masters should be installed like any other server - just turn them on. Well, it's not that easy, but puppetize them and they will tell you what to do.

This is a tricky thing to implement, and doesn't occur too often, so it's quite possible you'll run into problems along the way. We've done our best to make this work with the following instructions, but any improvements are welcome.

First Master

If this is the first master, you're probably adding a new organization. Create a fresh `manifests/$org-config.pp` and `manifests/$org-nodes.pp` and check them into your repository. The config should include, at a minimum:

• $org
• $puppet_again_repo
• $puppet_notif_email
• $puppet_servers and $puppet_server
• $data_servers and $data_server
• $distinguished_puppet_master
• $extra_user_ssh_keys with your username and key in it
• $admin_users with your username in it

If you will be using the $apt_repo_server variable, consider also setting $puppetmaster_cert_extra_names.

1. Build a CentOS system with only a Base install
2. Install git
3. clone https://github.com/mozilla/build-puppet or your clone of it in /tmp (don't do this in /root, as non-root users will need access during the process)
   • if you want to follow a particular NamedBranch in that repo (e.g., production), check out that branch now, and verify later that /etc/puppet/production is on the appropriate branch. If it's not, file a bug against the masterizing process to fix it.
4. in the clone, run ./setup/masterize.sh
5. Masterize will stop a few times along the way to tell you what to do, and how to pick up again where it left off.
   • Relax, don't worry, have a homebrew!
   • For help with the SSL portions, see ReleaseEngineering/PuppetAgain/Certificate Chaining

Subsequent Masters

• Add a node definition for the new master and install it. Do not add the new master to the org config yet.
• Kickstart the master like you would any other host.
  • in AWS for moco, note that you will need to manually set up the /data partition; see bug 1140018
  • Note that the master will start complaining via cronspam. You can kill cron to make that stop temporarily.
• Begin to manually synchronize /data from an existing master to the new master (it will take a while!)
• Puppet will eventually fail with complaints about synchronizing git and secrets with the distinguished master. The trick here is to temporarily add the new master's fqdn to ~puppetsync/.ssh/authorized_keys (you may need to repeat this if puppet runs on the DM and reverts your changes) and run puppet again.
• Puppet will next fail with a message about the master CA cert and some instructions. Follow the instructions.
• Run puppet again, solving errors as they occur. Note that httpd will start successfully only on the final run.

The master should now be ready. Test it by running the agent against the new master:

 puppet agent --test --server=$(facter fqdn)

once you're satisfied, add the master to the list of masters in the org config, and add its SSH key to modules/ssh/templates/known_hosts.erb.

In moco, you'll also want to

• add nagios monitoring
• update build-cloud-tools instance_data/* to reference new masters
• add the new hosts to the puppetagain-apt A record
• request new flows if necessary
• add the new master to the appropriate `puppet` CNAMES (puppet.somevlan.xyz1.mozilla.com)