ReferencePlatforms/Win64

From MozillaWiki
Jump to: navigation, search

Version 1

Configuration details

File Name: en_windows_server_2008_r2_standard_enterprise_datacenter_web_vl_build_x64_dvd_x15-59754.iso
Date Published (UTC): 8/14/2009 9:55:36 AM Last Updated (UTC): 9/28/2009 8:45:48 PM
SHA1: AD855EA913AAEC3F1D0E1833C1AEF7A0DE326B0A
ISO/CRC: F111FF61

Machine setup

Administrator password

  • Search for secpol.msc
  • Security Settings -> Account Policies -> Password policy
  • Change "Password must meet complexity requirements" to Disabled
  • Go and change the Admin's current password

Virtual Memory

  • Sign is as Administrator
  • Control Panel -> System and Security -> System
  • Advanced system settings
  • Advanced tab -> Settings... under Performance
  • Advanced tab -> Change... under Virtual Memory
  • Un-check "Automatically manage paging file size for all drives"
  • Custom size: 4048-4048
  • Restart the machine

There are some advantages to choosing a fixed size:

  • free space doesn't fluctuate on c:\
  • the page file doesn't get fragmented on the disk
  • OOM conditions are more repeatable


Reference:

Hard-drive adjustments

  • Start -> All Programs -> Administrative tools -> Computer Management
  • Storage -> Disk Management
  • Select the C drive
    • "Shrink volume..."
    • Select a value that will make the remaining free space to match the table below

* Create the D drive with 20480 MB, label it "Tools" and do a quick NTFS format (no indexation)

NOTE: Creating the E drive will be done through a script. TODO: armenzg: digipengi we have to block indexation to happen in such drive

These are the disk distributions you should have at the end:

C drive 61440 MB 60GB NTFS
E drive (Builds) ~173GB NTFS
  • Manually right click on disks C, select properties and go to the general tab for the dialog. At the bottom of the dialog box, verify that "Compress this drive" and "Indexing service" are both unchecked (turned off).

Hard-drive improvements:

  • Start dos-cmd-shell and run the following commands. It does not matter what directory is currect directory, these changes are system wide.
 fsutil behavior set disable8dot3 1
 fsutil behavior set disablelastaccess 1
  • Reboot

We followed tips outlined http://www.windowsdevcenter.com/pub/a/windows/2005/02/08/NTFS_Hacks.html and http://technet.microsoft.com/en-us/library/cc785435.aspx.

Clean Manager

  • Open "Server Manager", right click and choose "Add features", then select "Desktop Experience" to be installed
  • You will have to restart the server for this to apply

Daemon tools

3cad7a0a3d5c110a6e70f3d64a32a44f *DTLite4402-0131.exe
  • There is a newer copy in dev-stage since that previous version is not available anymore:
http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/DTLite4454-0315.exe
  • Install it only with "Startup shortcuts"

Create cltbld user

  • Go to "Server Manager"
  • Configuration -> Local Users and Groups -> Users
  • Create the "New User"
  • Modify the properties of it
    • "User cannot change password", "password never expires"
    • Add "Remote Desktop Users" to "Member Of"

Basic OS Setup

  • Verify the name of the NetBios computer name and the DNS suffix are correct
    • Control Panel -> System and security -> System
    • "Change settings"
    • Under "Computer name" tab click on "Change..."
    • Add "win64-ix-ref" as the "Computer name"
    • Click on "More..." and add "build.mozilla.org" as the Primary DNS suffix
  • Check for updates
    • Install any updates that were available before 2011/06/28
  • Check for updates a second time after the first batch of updates are installed
    • Windows Server 2008 R2 Service Pack 1 x64 Edition (KB976932)
  • Disable automatic updates
    • Control Panel -> System and Security -> Windows Update
    • Change settings -> "Never Check for Updates" and click "OK
  • Keep Windows Firewall ON and add exceptions once you install the SSH and VNC servers
  • Turn off Screensaver
    • Control Panel -> Appearance and Personalization -> Change screen saver (under Personalization)
    • Select "None", click "OK"
  • Energy settings
    • Control Panel -> Hardware -> Power Options
    • "High Performance"
  • High performance
    • sysdm.cpl
    • Advanced -> Performance.Settings...
      • On "Advanced" set to adjust best performance for "Programs" (a.k.a foreground)
  • Networking
    • Turned off all MS networking, leaving only TCP/IP v4 and v6

Auto-login

  • Allow cltbld to restart the system
    • Start menu -> Run -> gpedit.msc (as administrator)
    • Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> User Rights Assignment
    • Double click 'Shut down the system', add cltbld to the list.
  • Download the following file, change the password and import into the registry (as administrator)
 wget -Ologon.reg --no-check-certificate https://bug645024.bugzilla.mozilla.org/attachment.cgi?id=544857

Build tools, compilers and SDKs

MozillaBuild 1.5.1

Visual Studio 2010

Even though the default compiler is VS2008's but might as well get this deployed for the future.

  • Download and mount the ISO with Daemon tools lite
5b7e2c716510c87ddf64d26ab9a072f2 *en_visual_studio_2010_professional_x86_dvd_509727.iso
  • Do a custom install:
    • C:\Tools\msvs10
    • Only select "Visual C++"

Visual Studio 2008

  • To be able to install VS2008 on Win2008 you first have to install ".NET Framework 3.5.1" (See this article for more information).
    • Server Manager -> right click on "Features" and select "Add Feature"
    • Check ".NET Framework 3.5.1 Feature"
    • "Add Required Role Services"
    • Select "Next" a couple of times and then "Install"
  • Mount the VS2008 DVD with Daemon tools Lite
  • Do a custom installation into C:\Tools\msvs9
    • Select "Visual C++" (including x64 compilers)
    • Select "Tools for Redistributing Applications"
 ba1ba1a6ddd0c93441153af9e93a7b22 *en_visual_studio_2008_professional_x86_x64wow_dvd_X14-26326.iso

Visual Studio 2008 SP 1 (Installer)

As documented in Windows Build Prerequisites 

49490945de6a32fafadb861f9f308a84 VS90sp1-KB945140-ENU.exe

DirectX SDK

This is required to build ANGLE. 

 a7fa610b1791d873162b9e33007a6777 *DXSDK_Jun10.exe
  • Install it to "C:\Tools\sdks\dx10"
  • Choose the following components:
    • Instal DirectX Runtime
    • DirectX Utilities
    • DirectX Headers and Libs
    • DirectX Redistributable Files

Windows 7 SDK

  • Like in https://wiki.mozilla.org/ReferencePlatforms/Win32#Version_20_-_20090930
  • Download the web installer or use the DVD ISO.
  • run the Windows SDK Configuration Tool to select 7.0 as the right SDK
  • Mount the DVD with Daemon Tools Lite and install into C:\Tools\sdks\v7.0
    • Windows Headers and Libraries
    • Visual C++ compilers (support for x64 and IA64)
    • Windows Development Tools
    • Application Verifier
    • Debugging Tools for Windows 
1081c87025a5a507a55426ac59222b7a *GRMSDKX_EN_DVD.iso

Windows updates

After Visual Studio and the SDKs have been installed let's check for updates again. These got installed: 

KB2160841 KB2416472 KB2524375

Disable JIT debugger

This is something similar to what was done in bug 533185 for win32 slaves.

To disable the JIT debugger I read this and this and did the following: 

In the Registry Editor window, locate and delete the follow registry keys:

 HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger
 HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\DbgManagedDebugger
 HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger
 HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\DbgManagedDebugger

Disable Windows Error Reporting

When one of the tests run by "make -k check" crashes it reports a crash dialog saying "program.exe stoppped working" which can be disabled by doing the following:

  • gpedit.msc
  • Computer configuration -> Administrative Templates
  • Windows Components -> Windows Error Reporting
  • Set "Prevent display of the user interface for critical errors" to Enabled

Add yasm.exe

To make the webm build to work for Windows 64-bit I had to: 

 wget http://www.tortall.net/projects/yasm/releases/vsyasm-1.1.0-win64.zip
 unzip vsyasm-1.1.0-win64.zip
 mv vsyasm.exe /c/mozilla-build/msys/bin/yasm.exe
 rm vsyasm.* readme.txt

NOTE: Win32 machines install yasm.exe under /c/mozilla-build/yasm and add a profile-extrapaths.sh under /c/mozilla-build/msys/etc/profile.d according to this.

Automation setup

Keys

  • Get production keys unto C:/Users/cltbld/.ssh
  • Tested that I can run this:
#!/bin/sh
set -ex
ssh -i ~/.ssh/ffxbld_dsa ffxbld@aus2-staging.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa ffxbld@dm-symbolpush01.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage-old.mozilla.org exit
ssh -i ~/.ssh/xrbld_dsa xrbld@stage.mozilla.org exit

Python 2.6 and twisted 9

  • Append ';C:\mozilla-build\python;C:\mozilla-build\msys\bin;C:\mozilla-build\vim\vim72' to system path.
  • Install this registry key:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\Pythoncore\2.6\InstallPath]
@="C:\\mozilla-build\\python"

NOTE: The registry key is different for win32 machines; notice that it says Wow6432Node

Twisted patches

 wget -Ob128.patch --no-check-certificate http://hg.mozilla.org/build/twisted/raw-rev/0b441f0f68b4
 wget -Osubproc.patch --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=451277
 cd /c/mozilla-build/python/lib/site-packages
 cd twisted
 patch -p0 < ~/subproc.patch
 patch -p2 < ~/b128.patch

Startup scripts

 cd /c/Users/cltbld/AppData/Roaming/Microsoft/Windows/Start\ Menu/Programs/Startup
 wget -Obuildbot.bat http://hg.mozilla.org/build/puppet-manifests/raw-file/c6e02dee3589/modules/buildslave/files/buildbot-win64.bat

 # As administrator
 cd /c/mozilla-build
 wget -Ostart-buildbot.bat http://hg.mozilla.org/build/puppet-manifests/raw-file/c6e02dee3589/modules/buildslave/files/start-buildbot-win64.bat

Create directories

  • Create logging and buildbot directories
mkdir /c/tmp

Other tools

Installing NRPE Daemon

To learn more about Nagios/NRPE and NSClient++ please read this blog post with some of my extra debugging and testing.

NSClient++ is an NRPE implementation on Windows.

  • Enable ICMP
runas /user:Administrator "netsh firewall set icmpsetting 8"
  • Download NSClient++ 0.3.8 for 64-bit machines
    • I am going to use the installer as it adds the firewall exceptions for me.
 05/27/10 23:11:48 0735d6ef6953e315b3ce9e02c1ccdb11 NSClient%2B%2B-0.3.8-x64.msi
  • Start the installer and install all the defaults
    • "Do not start the service" and finish the installation
  • Rename C:\Program Files\NSClient++\NSC.ini as NSC.original.ini
  • Download releng's NSC.ini and place it on C:\Program Files\NSClient++
 cd /c/Program\ Files/NSClient++
 mv NSC.ini NSC.original.ini
 wget -O NSC.ini http://hg.mozilla.org/build/opsi-package-sources/raw-file/2d51f32ff0f2/nagios/CLIENT_DATA/nsc.ini

To check that everything went well do the following (from this documentation):

  • "C:\Program Files\NSClient++\nsclient++.exe" /test
  • Type these checks and look at the results
    • CheckDriveSize ShowAll MinWarnFree=10% MinCritFree=5% Drive=c:\
    • CheckCPU warn=80 crit=90 time=20m time=10s time=4

SSH

Install the KTS SSH server and copy the key from another machine.

  • Download the installer from here
5ea25269b460ac3b99a5495a6602210a *kts118.exe
  • Install it with the default values
  • Copy the kts.ini into "C:\Program files (x86)\KTS\kts.ini" from a Win32 machine

Add firewall exception for SSH

  • Control Panel\System and Security\Windows Firewall\Allowed Programs
    • "Allow another program..."
      • "C:\Program Files (x86)\KTS\daemon.exe" and check "Home" and "Public"

VNC (UltraVNC)

  • Follow the "X64 1.0.8.2 Full" link in here.
620329f947b7085e9361f9631e45ecf8 *UltraVN1.0.8.2_x64_Setup.exe
  • Install it
    • Select the defaults up until the "Additional Tasks" screen
      • "Register UltraVNC as a Service"
      • "Start or restart UltraVNC"
      • "UltraVNC Server driver install" - explanation
    • Install the UVNC BVBA Display adapters prompt and select "Always trust"
    • After finishing the installation go to services and stop 'uvnc_service'.
    • Browse to C:\Program Files\UltraVNC and run 'winvnc.exe' as administrator (right click => 'run an administrator').
      • Under "Incoming connections" section:
        • Check "Ports" & Main: 5900
        • Uncheck "Enable JavaViewer (Http Connect)"
      • Under "File Transfer" section
        • Uncheck "Enable"
        • Uncheck "User Impersonation"
      • Add cltbld's credentials under "Authorization"
    • Restart the computer

Improve graphics performance

  • Connect through VNC (it should connect to cltbld)
  • sysdm.cpl
    • Advanced -> Performance.Settings...
      • On "Visual effects" set "Adjust for best performance"

Clean up

  • Now that we are ready to clone let's do a proper clean up to have the C and D drives as clean as possible
    Start -> All Programs -> Accessories -> System Tools -> Disk Cleanup. Do a default clean of C:
  • Let's also remove restore point to free up space on C drive

Buildbot 0.8.4

  • Install Buildbot and newer runslave.py (they are dependent on each other)
# ssh Administrator@slave
c:\mozilla-build\wget\wget.exe -O update-buildbot.bat https://bugzilla.mozilla.org/attachment.cgi?id=627242
update-buildbot.bat

Python 2.7.3

# ssh Administrator@slave
c:\mozilla-build\wget\wget.exe http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/python/python-2.7.3.msi
msiexec /i python-2.7.3.msi /quiet /norestart  /log python27_install.log TARGETDIR=c:\mozilla-build\python27 ALLUSERS=1

# Install pywin32
c:\mozilla-build\msys\bin\bash --login -i
/c/mozilla-build/wget/wget.exe http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/python/install_pywin32_python27.sh
./install_pywin32_python27.sh

Newer NSC.ini

# ssh Administrator@slave
cd "C:\Program Files\NSClient++"
c:\mozilla-build\wget\wget.exe -O NSC.ini http://hg.mozilla.org/build/opsi-package-sources/raw-file/3857d6f521f2/nagios/CLIENT_DATA/nsc.ini

Binscope

ssh Administrator@slave
c:\mozilla-build\wget\wget.exe http://people.mozilla.com/~asasaki/BinScopeSetup.msi
msiexec /i BinScopeSetup.msi /quiet /norestart /log binscope_install.log ALLUSERS=1

Fix auto-login after new password deployment

This is what you need to run to fix a machine: 

  REG ADD "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v AutoAdminLogon /t REG_SZ /d 1
  REG ADD "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v DefaultPassword /t REG_SZ /d YouWish

MaRu fixed the issue by updating the LTICleanup.wsf The lines modified were (password has been taken out for security reasons) 

oShell.RegWrite "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\AutoAdminLogon", "1", "REG_SZ"
oShell.RegWrite "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\DefaultUserName", "cltbld", "REG_SZ"
oShell.RegWrite "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\DefaultDomainName", "", "REG_SZ"
oShell.RegWrite "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\DefaultPassword", "*********", "REG_SZ"
oShell.RegDelete "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\AutoLogonCount", &H00000000, "REG_DWORD"
On Error Goto 0

Re-deploy twisted patches

  • It seems we have regressed this with the latest buildbot update
  • bug 770506 - We should verify this and file a bug for IT to fix it
    • TODO meanwhile get work around for setting new slaves up

Deploy Windows SDK for Windows 8

This can be deployed like this: 

C:\mozilla-build\wget\wget.exe https://github.com/armenzg/bearded-ironman/raw/master/win2008_64bit/install_windows_standalone_sdk_8.0.bat
install_windows_standalone_sdk_8.0.bat

File:Beispiel.jpg== Deploy the latest known_host == This is needed until bug 849942 fixed. 

ssh as cltbld to host
cd .ssh
copy known_hosts known_hosts.bak
C:\mozilla-build\msys\bin\scp.exe cltbld@buildbot-master19.build.mozilla.org:~/known_hosts .
set HOME=C:\Documents and Settings\cltbld
C:\mozilla-build\msys\bin\ssh -i ~/.ssh/ffxbld_dsa ffxbld@symbolpush.mozilla.org hostname

which should show symbols1.dmz.phx1.mozilla.com

I also updated this zip with the ssh keys etc referenced in this doc https://wiki.mozilla.org/ReleaseEngineering/How_To/Adjust_SSH_keys_on_a_slave#Production when I landed the changes for the new symbol server.

Deploy _dumbwin32proc.py

Until bug 853609 is fixed, log in as Administrator and run this: 

C:\mozilla-build\wget\wget.exe -q -Odeploy_dumbwin32proc.bat https://bugzilla.mozilla.org/attachment.cgi?id=725099
deploy_dumbwin32proc.bat

Post-reimaging steps

Once one of these steps is already in the imaging process then you can move before the "Post-re-imaging steps" section.

Update to the latest buildbot.bat

We recently added steps to buildbot.bat to cleanup Windows TEMP dirs on reboot. (As user cltbld). 

cd "C:\Users\cltbld\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup"
mv buildbot.bat "C:\Users\cltbld\buildbot.bat.precleanup"
"C:\mozilla-build\wget\wget" -O buildbot.bat https://hg.mozilla.org/build/puppet-manifests/raw-file/f24f27f2696a/modules/buildslave/files/buildbot-win64.bat

Deploy the correct keys

Not even *all* production keys come on re-imaged machines. You have to remove the keys and re-deploy them with the instructions given. You have instructions over here (As user cltbld).

Deploy google api key

This should go in E:\builds\gapi.data. Get the key from secrets.csv on one of the puppet masters. (As user cltbld).

Or this: 

"C:\mozilla-build\msys\bin\scp" -o 'StrictHostKeyChecking no' -o 'BatchMode=no' -r  cltbld@bld-linux64-ix-028:/builds/gapi.data .

Update the cltbld password

Remember: The Windows password and VNC passwords are different. Ask someone if you don't know which is which. (As user Administrator.) Windows password:

  • Control Panel -> User Accounts -> Change Your Windows Password -> Change your password
    • or via ssh to Administrator account (enter passwd when prompted):
   net user cltbld *
  • Change "DefaultPassword" to the correct cltbld password
    • via gui: Open regedit -> Browse to "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon"
    • via command prompt (ssh won't work):
   reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v DefaultPassword /d newpasswd /t REG_SZ /f

VNC Password:

  • using ssh: use "sed", but note the "-i" option is not supported:
   cd "\Program Files\UltraVNC"
   copy ultravnc.ini ultravnc.bak
   sed -e s/default_key/new_key/ ultravnc.bak > ultravnc.ini
  • via GUI:
    • Right click on VNC icon in system tray -> Admin Properties -> Enter new password -> "OK" -> "Yes"
      • if rdp has been used you need to reboot to see VNC (ultra vnc), or launch ultra-vnc manually
      • with VNC login, you'll see "ctrl-alt-del to login" screen if autologon is not set up correctly.


Reboot.

  • via ssh: shutdown /r /d p:1:2 /t 1

Deploy MAPI Headers

Until bug 891956 is closed, you will need to copy the MAPI headers onto w64-ix-slaveXX build slaves using the following procedure: 

The files you'll need are in http://people.mozilla.org/~jhopkins/bug869974/

1) Unzip the files
2) Create an ~/.adminpass file containing the Administrator password for the script to use.
3) Set up a Python virtualenv with paramiko, requests installed
4) With that virtualenv active, run the script (adjust hostname and files directory):
$ python copy-mapi-headers-relops.py w64-ix-slave22 files_to_upload/*

Kpym config changes

From bug 909445, the following should be run as administrator to update the ssh config: 

cd "c:\program files (x86)\kts"
c:\mozilla-build\wget\wget -O kts.ini "https://bugzilla.mozilla.org/attachment.cgi?id=795563"
cd scripts
c:\mozilla-build\wget\wget -O allusers.bat "https://bugzilla.mozilla.org/attachment.cgi?id=795564"

Enable LocalDumps

From bug 897768, run the following from the command-line: 

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps" -v DumpType -t REG_DWORD -d 1 
reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps"
Retrieved from "https://wiki.mozilla.org/index.php?title=ReferencePlatforms/Win64&oldid=747550"