ReleaseEngineering/OPSI

From MozillaWiki
Jump to: navigation, search

Definitions

  • OPSI Server - Serves opsi packages, instructs clients on which packages to install, remove, etc. This is a Linux machine (debian currently).
  • OPSI Client - Runs on win32 machines. Runs right before the login prompt at boot and audits the system to make sure the packages are in the proper state. This is when new installations happen.
  • OPSI Package - Contains to-be-installed files as well as installation instructions. Sources for these package are to be kept in version control. Packages themselves are kept on the OPSI Server.
  • OPSI Config Editor - This is a Java app that is used to manage what is installed on which machines. It lets you schedule installations of packages, and see what is currently installed. (Other things too, but not all packages support them.) You can download it here.
    • If you are on OS X and can't start OPSI.jar, launch 'Java Preferences' and change Java 6 64-bit to be the top of the list of runtimes for applications
Before Windows logs in the OPSI installation happens

Servers

We run two OPSI servers:

  • staging-opsi.build.mozilla.org
  • production-opsi.build.mozilla.org

Install OPSI on a client

Enable File and Print Sharing

  • Enable File & Print Sharing
    • Control Panel -> Network Connections -> Local Area Connection -> Properties
    • Enable "Client for Microsoft Networks" and "File and Print Sharing", click Close.
  • Open firewall (only to opsi server)
    • Control Panel -> Windows Firewall -> Exceptions
    • Enable "File and Print Sharing", click "Edit"
    • For each entry:
      • Click "Change Scope"
      • Select "Custom List"
      • Enter the IP address of the OPSI server. (For staging: 10.26.48.39, for production: 10.26.48.38).

OPSI Prep and Install

We need to flip a few options in Windows to enable certain packages to install without prompting.

  • Do not prompt for unsigned drivers (needed for at least the windows mobile sdk)
    • Control Panel -> System -> Hardware -> Driver Signing, choose "Ignore".
  • Security Settings
    • Start Menu -> Run -> \\$ipOfOpsiServer (staging: \\10.26.48.39, production: 10.26.48.38)
    • Use 'pcpatch' and the root password when prompted.
    • Right click 'opt_pcbin' and select 'Mount as Network Drive' and use Z: (this is not a permanent mount, so the letter really doesn't matter)
    • Browse to z:\install\preloginloader
    • Double click 'opsi-prep.vbs'
    • Double click 'service-setup.cmd', hit a key when prompted
    • When prompted for a username and password use 'root' and the opsi server's root pw.

Once OPSI is finished installing it will forcibly reboot the system. In a few minutes it should come back up and the client should be listed in the OPSI GUI.

Autoit

We require Autoit to deploy many packages. To install it, do the following:

  • Checkout the installer:
cvs -d:ext:cltbld@cvs.mozilla.org:/mofo co ref-platforms/win32/autoit-v3-setup.exe
  • (as Administrator), run the installer, accept the defaults

Using/writing OPSI in a nutshell

Installing a package on a client

  • Open up the OPSI Config Editor
  • Select a client or clients, click on "Product Configuration"
  • Change the action to "setup" for anything you want to install. (NOTE the column for "action required")

Opsi action setup.png

  • The package(s) will be installed at the next reboot.

Writing packages

There are different things you can do depending of what type wInst target you are writing for:

Creating a package

Parts of the package

An OPSI package is built from a directory with a specific layout and set of files. These are kept in version control and should be reviewed before check in. The important files in the package are as follows:

  • OPSI/control - This contains the metadata for a package: name, version number, path to install script, etc.
  • CLIENT_DATA/packagename.ins - This is the meat of a package. All of the installation work and setup should be done here. It is written in a language called wInst which is fairly well documented.

Less commonly used files:

  • OPSI/preinst - Executes before the installation script (shell)
  • OPSI/postinst - Executes after the installation script (shell)

Any installers or files a package depends on should be placed in the CLIENT_DATA/ directory, too. That entire directory is downloaded to the client before the installation script executes. Files in it can be referenced from within the installation script.

Bootstrapping a new package

To get the basic files/directories needed to start a new package you should do the following: 

hg clone http://hg.mozilla.org/build/opsi-package-sources
cd opsi-package-sources
rsync -av newproduct-template/ myproduct/

Next:

  • Edit OPSI/control with the product details.
  • Put package files in CLIENT_DATA/.
  • Create an installation script.

Installation script basics

This document doesn't cover wInst syntax in detail, please refer to the wInst manual for that.

Installing an MSI

This is by far the easiest type of package to install. Almost all MSIs have a silent mode which you can invoke. For example, here is the installation script for Visual Studio 2005: 

[Initial]
Message=installing Visual Studio 2005 ...
StayOnTop=false
[Aktionen]
  Winbatch_product_silent_install
[Winbatch_product_silent_install]
msiexec.exe /i %SCRIPTPATH%\vs_setup.msi TRANSFORM=%SCRIPTPATH%\Setup\vs.ini /passive /l* c:\visualstudioinstall.log

Installing a package without a silent install

A lot of packages don't offer silent install options. To work around this, we depend on a tool called Autoit. Autoit is a tool which automates mouse clicks and keyboard strokes. Here's how to create a package that uses Autoit:

  • Use the Autoit "Script Writer" to record the necessary clicks and strokes to install your package. Try to keep unnecessary mouse movement to a minimum while recording to avoid bloating the script.
  • Test the script to make sure it works.
  • Put the script and the package in the CLIENT_DATA/ directory.
  • Create the OPSI installation script and call Autoit from it. For example, here's the installation script for MozillaBuild 1.3:
Initial]
Message=MozillaBuild 1.3 Installation
StayOnTop=false

[Aktionen]
Winbatch_install_mozillabuild

[Winbatch_install_mozillabuild]
autoit3.exe %SCRIPTPATH%\mozillabuild.au3 %SCRIPTPATH%\MozillaBuildSetup-1.3.exe

Updating registry keys

The wInst language provides a nice built-in way to update registry keys as part of an installation. Rather than (or in addition to) calling out to an installer you can write registry keys in the installer. For example, here's how to update the system PATH to append "d:\mozilla-build\python25\scripts": 

[Initial]
Message=Installing new registry keys
StayOnTop=false

[Aktionen]
Registry_AddPath

[Registry_AddPath]
openkey [HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment]
supp "Path"; "d:\mozilla-build\python25\scripts"


Testing the package on staging

On staging you want to test out the package first to make sure it installs correctly before it goes into production. To create a .opsi package and inform OPSI about it, do the following: (For staging): 

# as root on the opsi server
cd ~cltbld/opsi-package-sources
su -c './regenerate-package dir-with-your-package'
# to confirm that what you want installed on the slave is correct
# look in:
/opt/pcbin/install/dir-with-your-package

If there are any issues and you need to regenerate the package you can simply drop in the new code and run the above again.

Installing the package file on production

Please note that you should NOT install packages onto the production OPSI server unless they have been properly reviewed and checked in. That means your package should be landed in opsi-package-sources (hg), and you should land any binaries in opsi-binaries as cltbld@staging-opsi (CVS).

(For production): 

# as cltbld on the opsi server
# update binaries
cd ~cltbld/opsi-binaries
cvs up -Pd

# update the package source clone
cd ~cltbld/opsi-package-sources
hg id && hg in -v
hg pull && hg up
./sync-binaries

# Recreate and install the package
su -c './regenerate-package package-name'

Your package should now show up in the OPSI Config Editor. Make sure to set staging slaves to install/update the package as well as production ones.

Troubleshooting

Check which master

Check the hostkey

  • Check that the hostkeys on the master and the slave are the same
    • in master: grep talos-r3-xp-066 /etc/opsi/pckeys
    • in slave: cat C:\Program Files\opsi.org\preloginloader\cfg\locked.cfg
    • The slave's hostkey is generated with a service on boot.

schtasks /create /tn opsikey /tr d:\\mozilla-build\\python25\\python.exe d:\\tools\\buildfarm\\opsi\\regenerate-hostkey.py /sc ONSTART /ru SYSTEM

Fix the hostkey

I've seen before XP slaves not running the regenerate-hostkey.py service and require you to run it manually: 

C:\>cat "C:\Program Files\opsi.org\preloginloader\cfg\locked.cfg"
[shareinfo]                       
pckey = 80dc2294bff215b3fab57d8e7ff61ccd
c:\>del "C:\Program Files\opsi.org\preloginloader\cfg\locked.cfg"
C:\>C:\mozilla-build\python25\python.exe C:\tools\buildfarm\opsi\regenerate-hostkey.py
C:\>cat "C:\Program Files\opsi.org\preloginloader\cfg\locked.cfg"               
[shareinfo]
pckey = db4bb6ed05f5359cf5ef23870fea3f93

NOTE: you must delete the key file, or the script will not regenerate it!
NOTE: the script logs to c:\tmp\key-generator.log

After rebooting you should be good to go

Understanding the key generation

talos-r3-xp-067.uib.local:db4bb6ed05f5359cf5ef23870fea3f93

[shareinfo]
pckey = db4bb6ed05f5359cf5ef23870fea3f93

If your slave was to match any of these values then it means that you have the key for one of the ref machines and it has to be fixed: 

>>> md5('win2k3-ref-img').hexdigest()
'9b4cefa1c3a1f9159cf51a6d28155be8'
>>> md5('win32-ix-ref').hexdigest()
'c1cdfb2077205f086680ada04310f8c2'
>>> md5('talos-r3-xp-ref').hexdigest()
'80dc2294bff215b3fab57d8e7ff61ccd'

Look at this code to understand the uniqueness of the value: 

>>> from md5 import md5
>>> import sys
>>> from socket import gethostname
>>> gethostname()
'talos-r3-xp-067'
>>> md5(gethostname()).hexdigest()
'db4bb6ed05f5359cf5ef23870fea3f93'
>>> md5('talos-r3-xp-ref').hexdigest()
'80dc2294bff215b3fab57d8e7ff61ccd'

Check the opsiconfd log

  • select the slave from the OPSI.jar menu
  • select the tab that says "Log files"
  • select the sub-tab named "opsiconfd"
  • go to the bottom to see the latest messages

uib.local VS .build.mozilla.org

bhearsum: it indicates what domain the OPSI server thinks it's in.
          it's long standing that it's different between production and
          staging, but it shouldn't be an issue

Re-installing OPSI

This is not needed anymore. Check with bhearsum for details.

NOTE: This instructions were build up from this page.

  • change the opsiServiceURL to point to staging (explained above)
  • In c:\program files\opsi.org\preloginloader\uninst run deinstall.cmd
    • the slave will reboot
  • disconnect the mapped drive for production opsi
  • install OPSI from staging-opsi
    • Start Menu -> Run -> \\10.26.48.39
    • Use 'pcpatch' and the OPSI password when prompted.
    • Right click 'opt_pcbin' and select 'Mount as Network Drive'
    • Browse to z:\install\preloginloader
    • Double click 'opsi-prep.vbs' (you won't see anything happen)
    • Double click 'service-setup.cmd', hit a key when prompted
    • When prompted for a username and password use 'root' and the appropriate password (the current one unlike for "pcpatch")
    • Press any key (I assume it should have rebooted after this but it didn't - armenzg)

[really? it didn't happen to me - armenzg] OPSI forces reboot on completion of install and then you can manage it through the opsi client.

Mit Netzlaufwerken verbinden, bitte noch etwas warten

It translates to: Connecting to network drives, please wait a bit

It takes several minutes until it times out.

This is a DNS problem which was spotted on bug 620517.

To fix it you have to add build.mozilla.org to the main DNS server as explained in here.

Access violation (zugriffsverletzung bei adresse)

It is a blocking message in a "wInst-Message" pop up saying: "zugriffsverletzung bei adresse <memory address> in Modul 'winst32.exe'. Lesen von Adresse 0000000"

See bug 680457

error: HTTP/1.1 401 Unauthorized

This error appears before logging in. I believe you can only stop "the installation" and be able to access the machine.

error: HTTP/1.1 401 Unauthorized
bhearsum: This usually means that the slave is talking to the wrong opsi server,
or that the hostkey on the slave doesn't match the one the server.

I assume we have to check the following:

Wrong hostkey

Failed to process rpc: opsiHostKey authentication failed for host 'talos-r3-xp-001.build.mozilla.org': wrong key (opsiconfd|623)

How to update the opt_pcbin drive mapping password - "pcpatch"

NOTE: this has never been tested.

In bug 519989, we did some preliminary investigation into how we might go about updating the password used for mounting the OPSI shared package drive in case we happen to expose it:

  1. Arrange for a long downtime. Depending about how comfortable you are about burning builds in progress, you'll likely need between 3-6 hours.
  2. Shutdown, possibly gracefully, any affected buildbot masters.
  3. Push out an OPSI product that changes the password but leaves the pcpatch user password as it is (for now)
  4. Reboot the clients, and during their OPSI check-in the change is made. The change is made after the mount is authenticated, so they *should* be fine this time around.
  5. After all the clients have the new package (as verified through the OPSI dashboard), change the pcpatch user password on the server. Future reboots will work with the new password.
  6. Reboot the clients again to make sure they can connect.

Further background can be found in bug 519989.

Documentation/Links

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/OPSI&oldid=482641"