ReleaseEngineering/How To/Adjust SSH keys on a slave: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Update for b2g, no more preprod)
 
(29 intermediate revisions by 7 users not shown)
Line 1: Line 1:
{{Release Engineering How To|Adjust SSH keys on a slave}}
{{Release Engineering How To|Adjust SSH keys on a slave}}
There are three sets of keys that are important: staging, production and try. Aside from a strange permissions problem on linux (.ssh is root:root owned), the process is roughly consistent on all three platforms. 
 
NOTE: THIS PAGE ONLY APPLIES FOR WINDOWS ([https://bugzil.la/792836 Bug 792836 - Manage slave secrets with puppet])
 
There are three sets of keys that are important: staging, production and try.


In general, copy SSH keys from a similarly-configured slave.  You will need to use <tt>-oBatchMode=no</tt> in your ssh invocation to avoid host-key failures.  Note that only the private keys (<tt>*_dsa</tt>) are required, not the public keys (<tt>*_dsa.pub</tt>) <small>(however, if you have the "<tt>.pub</tt>", it must match with the private or the key will silently fail</small>).  Also note that the staging and production keys have the same filename.  The current production ffxbld_dsa has md5 beginning with '166b900'; staging's begins with '86bcf286'.
In general, copy SSH keys from a similarly-configured slave.  You will need to use <tt>-oBatchMode=no</tt> in your ssh invocation to avoid host-key failures.  Note that only the private keys (<tt>*_dsa</tt>) are required, not the public keys (<tt>*_dsa.pub</tt>) <small>(however, if you have the "<tt>.pub</tt>", it must match with the private or the key will silently fail</small>).  Also note that the staging and production keys have the same filename.  The current production ffxbld_dsa has md5 beginning with '166b900'; staging's begins with '86bcf286'.


<strike>= Staging =
= Staging =
To test that you have the staging keys and they are set up properly, try:
Windows steps:
# *nix
rmdir /S /Q .ssh
ssh -i ~/.ssh/ffxbld_dsa ffxbld@staging-stage.build.mozilla.org hostname
mkdir .ssh
# Windows
cd .ssh
  set HOME=C:\Documents and Settings\cltbld
C:\mozilla-build\msys\bin\scp cltbld@linux-ix-slave03:~/.ssh/* .
D:\mozilla-build\msys\bin\ssh.exe -i .ssh\ffxbld_dsa ffxbld@staging-stage.build.mozilla.org hostname
set HOME=C:\Users\cltbld
</strike>
 
= Preproduction =
To test that we're good:
Preproduction keys are not the same as staging keys - see [[ReleaseEngineering/Preproduction/Stage]].
<pre>
ssh -i ~/.ssh/ffxbld_dsa   ffxbld@dev-stage01.srv.releng.scl3.mozilla.com exit
ssh -i ~/.ssh/xrbld_dsa    xrbld@dev-stage01.srv.releng.scl3.mozilla.com exit
ssh -i ~/.ssh/tbirdbld_dsa tbirdbld@dev-stage01.srv.releng.scl3.mozilla.com exit
ssh -i ~/.ssh/b2gbld_dsa    b2gbld@dev-stage01.srv.releng.scl3.mozilla.com exit
ssh -i ~/.ssh/trybld_dsa    trybld@dev-stage01.srv.releng.scl3.mozilla.com exit
</pre>
 
These should be set up but aren't, but then we don't have good staging for mozharness.
<pre>
ssh -i ~/.ssh/b2gtry_dsa    b2gtry@dev-stage01.srv.releng.scl3.mozilla.com exit
ssh -i ~/.ssh/b2gbld_dsa    ffxbld@dev-stage01.srv.releng.scl3.mozilla.com exit
</pre>


= Production =
= Production =
NOTE: '''Make sure that the host you try to grab keys from is on the same data-center'''.
Steps for Windows:
Steps for Windows:
<pre>
<pre>
rmdir /S /Q .ssh
rm -rf .ssh
C:\mozilla-build\msys\bin\scp.exe cltbld@staging-master.build.mozilla.org:~/w32-prod-keys.zip .
"C:\mozilla-build\msys\bin\scp" -o 'StrictHostKeyChecking no' -o 'BatchMode=no' -r  cltbld@b-linux64-ix-0001.build.mozilla.org:~/.ssh .ssh
C:\mozilla-build\7zip\7z.exe x w32-prod-keys.zip -o".ssh"
del w32-prod-keys.zip
rmdir /S /Q E:\builds\moz2_slave
</pre>
</pre>
if scp does not working properly coping files from windows to windows hosts, use sftp instead:
<pre>
rm -rf .ssh
sftp b-2008-ix-0083.winbuild.releng.scl3.mozilla.com:.ssh/* .ssh/
</pre>


To test that a production master slave is set up properly, you must be able to run the following commands:
To test that a production master slave is set up properly, you must be able to run the following commands:
set HOME=C:\Documents and Settings\cltbld
<pre>
C:\mozilla-build\msys\bin\ssh -i ~/.ssh/auspush      ffxbld@aus3-staging.mozilla.org hostname
ssh -i ~/.ssh/ffxbld_dsa    ffxbld@symbolpush.mozilla.org exit
C:\mozilla-build\msys\bin\ssh -i ~/.ssh/ffxbld_dsa    ffxbld@symbols1.dmz.phx1.mozilla.com hostname
ssh -i ~/.ssh/tbirdbld_dsa  tbirdbld@symbolpush.mozilla.org exit
C:\mozilla-build\msys\bin\ssh -i ~/.ssh/ffxbld_dsa    ffxbld@stage.mozilla.org hostname
ssh -i ~/.ssh/ffxbld_dsa    ffxbld@stage.mozilla.org exit
C:\mozilla-build\msys\bin\ssh -i ~/.ssh/ffxbld_dsa    ffxbld@stage-old.mozilla.org hostname
ssh -i ~/.ssh/tbirdbld_dsa  tbirdbld@stage.mozilla.org exit
C:\mozilla-build\msys\bin\ssh -i ~/.ssh/xrbld_dsa    xrbld@stage.mozilla.org hostname
ssh -i ~/.ssh/xrbld_dsa    xrbld@stage.mozilla.org exit
C:\mozilla-build\msys\bin\ssh -i ~/.ssh/auspush      tbirdbld@aus3-staging.mozilla.org hostname
ssh -i ~/.ssh/b2gbld_dsa    b2gbld@stage.mozilla.org exit
C:\mozilla-build\msys\bin\ssh -i ~/.ssh/tbirdbld_dsa  tbirdbld@symbols1.dmz.phx1.mozilla.com hostname
ssh -i ~/.ssh/ffxbld_dsa    b2gbld@stage.mozilla.org exit
C:\mozilla-build\msys\bin\ssh -i ~/.ssh/tbirdbld_dsa  tbirdbld@stage.mozilla.org hostname
ssh -i ~/.ssh/b2gbld_dsa    b2gbld@pvtbuilds.pvt.build.mozilla.org exit
ssh -i ~/.ssh/b2gbld_dsa    b2gbld@pvtbuilds2.dmz.scl3.mozilla.com exit
</pre>


= Try =
= Try =
'''Try builders use different keys!'''
'''Try builders use different keys!'''


You must wipe any ssh keys that are not trybld from a newly imaged slave, and copy in the trybld keys from another try builder (staging trybld keys are on the staging slaves)
You must wipe any ssh keys that are not trybld from a newly imaged slave, and copy in the trybld keys from another try builder (staging trybld keys are on the staging slaves).


To test that a try slave is set up properly, you must be able to run the following commands:
To test that a try slave is set up properly, you must be able to run the following commands '''without needing to answer any questions''':
  ssh -i ~/.ssh/trybld_dsa trybld@stage.mozilla.org hostname
  ssh -i ~/.ssh/trybld_dsa trybld@stage.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds.pvt.build.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds2.dmz.scl3.mozilla.com hostname


Steps for Windows:
Steps for Windows (from SSH):
<pre>
rm -rf .ssh
mkdir .ssh
scp cltbld@b-linux64-hp-0001.build.mozilla.org:~/.ssh/* .ssh
# You will have to answer 'yes' and enter the cltbld password
ssh -i ~/.ssh/trybld_dsa trybld@stage.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds.pvt.build.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds2.dmz.scl3.mozilla.com hostname
rm -rf /c/builds/moz2_slave
</pre>
 
For seamicro instances, use sftp instead
<pre>
<pre>
rmdir /S /Q .ssh
rm -rf .ssh
D:\mozilla-build\msys\bin\scp.exe cltbld@staging-master.build.mozilla.org:~/w32-try-keys.zip .
mkdir .ssh
D:\mozilla-build\7zip\7z.exe x w32-try-keys.zip
sftp b-2008-ix-0178.wintry.releng.scl3.mozilla.com:.ssh/* .ssh/
del w32-try-keys.zip
# You will have to answer 'yes' and enter the cltbld password
REM Testing - it should not ask you for authentication
ssh -i ~/.ssh/trybld_dsa trybld@stage.mozilla.org hostname
set HOME=C:\Documents and Settings\cltbld
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds.pvt.build.mozilla.org hostname
D:\mozilla-build\msys\bin\ssh.exe -i .ssh/trybld_dsa trybld@stage.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds2.dmz.scl3.mozilla.com hostname
D:\mozilla-build\msys\bin\ssh.exe -i .ssh/trybld_dsa trybld@relengweb1.dmz.scl3.mozilla.com hostname
rm -rf /c/builds/moz2_slave
rmdir /S /Q E:\builds\moz2_slave
</pre>
</pre>

Latest revision as of 23:03, 12 June 2014


NOTE: THIS PAGE ONLY APPLIES FOR WINDOWS (Bug 792836 - Manage slave secrets with puppet)

There are three sets of keys that are important: staging, production and try.

In general, copy SSH keys from a similarly-configured slave. You will need to use -oBatchMode=no in your ssh invocation to avoid host-key failures. Note that only the private keys (*_dsa) are required, not the public keys (*_dsa.pub) (however, if you have the ".pub", it must match with the private or the key will silently fail). Also note that the staging and production keys have the same filename. The current production ffxbld_dsa has md5 beginning with '166b900'; staging's begins with '86bcf286'.

Staging

Windows steps:

rmdir /S /Q .ssh
mkdir .ssh
cd .ssh
C:\mozilla-build\msys\bin\scp cltbld@linux-ix-slave03:~/.ssh/* .
set HOME=C:\Users\cltbld

To test that we're good:

ssh -i ~/.ssh/ffxbld_dsa    ffxbld@dev-stage01.srv.releng.scl3.mozilla.com exit
ssh -i ~/.ssh/xrbld_dsa     xrbld@dev-stage01.srv.releng.scl3.mozilla.com exit
ssh -i ~/.ssh/tbirdbld_dsa  tbirdbld@dev-stage01.srv.releng.scl3.mozilla.com exit
ssh -i ~/.ssh/b2gbld_dsa    b2gbld@dev-stage01.srv.releng.scl3.mozilla.com exit
ssh -i ~/.ssh/trybld_dsa    trybld@dev-stage01.srv.releng.scl3.mozilla.com exit

These should be set up but aren't, but then we don't have good staging for mozharness.

ssh -i ~/.ssh/b2gtry_dsa    b2gtry@dev-stage01.srv.releng.scl3.mozilla.com exit
ssh -i ~/.ssh/b2gbld_dsa    ffxbld@dev-stage01.srv.releng.scl3.mozilla.com exit

Production

NOTE: Make sure that the host you try to grab keys from is on the same data-center.

Steps for Windows:

rm -rf .ssh
"C:\mozilla-build\msys\bin\scp" -o 'StrictHostKeyChecking no' -o 'BatchMode=no' -r  cltbld@b-linux64-ix-0001.build.mozilla.org:~/.ssh .ssh

if scp does not working properly coping files from windows to windows hosts, use sftp instead:

rm -rf .ssh
sftp b-2008-ix-0083.winbuild.releng.scl3.mozilla.com:.ssh/* .ssh/


To test that a production master slave is set up properly, you must be able to run the following commands:

ssh -i ~/.ssh/ffxbld_dsa    ffxbld@symbolpush.mozilla.org exit
ssh -i ~/.ssh/tbirdbld_dsa  tbirdbld@symbolpush.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa    ffxbld@stage.mozilla.org exit
ssh -i ~/.ssh/tbirdbld_dsa  tbirdbld@stage.mozilla.org exit
ssh -i ~/.ssh/xrbld_dsa     xrbld@stage.mozilla.org exit
ssh -i ~/.ssh/b2gbld_dsa    b2gbld@stage.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa    b2gbld@stage.mozilla.org exit
ssh -i ~/.ssh/b2gbld_dsa    b2gbld@pvtbuilds.pvt.build.mozilla.org exit
ssh -i ~/.ssh/b2gbld_dsa    b2gbld@pvtbuilds2.dmz.scl3.mozilla.com exit

Try

Try builders use different keys!

You must wipe any ssh keys that are not trybld from a newly imaged slave, and copy in the trybld keys from another try builder (staging trybld keys are on the staging slaves).

To test that a try slave is set up properly, you must be able to run the following commands without needing to answer any questions:

ssh -i ~/.ssh/trybld_dsa trybld@stage.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds.pvt.build.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds2.dmz.scl3.mozilla.com hostname

Steps for Windows (from SSH):

rm -rf .ssh
mkdir .ssh
scp cltbld@b-linux64-hp-0001.build.mozilla.org:~/.ssh/* .ssh
# You will have to answer 'yes' and enter the cltbld password
ssh -i ~/.ssh/trybld_dsa trybld@stage.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds.pvt.build.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds2.dmz.scl3.mozilla.com hostname
rm -rf /c/builds/moz2_slave

For seamicro instances, use sftp instead

rm -rf .ssh
mkdir .ssh
sftp b-2008-ix-0178.wintry.releng.scl3.mozilla.com:.ssh/* .ssh/
# You will have to answer 'yes' and enter the cltbld password
ssh -i ~/.ssh/trybld_dsa trybld@stage.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds.pvt.build.mozilla.org hostname
ssh -i ~/.ssh/b2gtry_dsa b2gtry@pvtbuilds2.dmz.scl3.mozilla.com hostname
rm -rf /c/builds/moz2_slave