ReleaseEngineering/OPSI

From MozillaWiki
Jump to navigation Jump to search

Definitions

  • OPSI Server - Serves opsi packages, instructs clients on which packages to install, remove, etc. This is a Linux machine (debian currently).
  • OPSI Client - Runs on win32 machines. Runs right before the login prompt at boot and audits the system to make sure the packages are in the proper state. This is when new installations happen.
  • OPSI Package - Contains to-be-installed files as well as installation instructions. Sources for these package are to be kept in version control. Packages themselves are kept on the OPSI Server.
  • OPSI Config Editor - This is a Java app that is used to manage what is installed on which machines. It lets you schedule installations of packages, and see what is currently installed. (Other things too, but not all packages support them.) You can download it here.
    • If you are on OS X and can't start OPSI.jar, launch 'Java Preferences' and change Java 6 64-bit to be the top of the list of runtimes for applications

Servers

We run two OPSI servers:

  • staging-opsi.build.mozilla.org
  • production-opsi.build.mozilla.org

Install OPSI on a client

Enable File and Print Sharing

  • Enable File & Print Sharing
    • Control Panel -> Network Connections -> Local Area Connection -> Properties
    • Enable "Client for Microsoft Networks" and "File and Print Sharing", click Close.
  • Open firewall (only to opsi server)
    • Control Panel -> Windows Firewall -> Exceptions
    • Enable "File and Print Sharing", click "Edit"
    • For each entry:
      • Click "Change Scope"
      • Select "Custom List"
      • Enter the IP address of the OPSI server. (For staging: 10.2.71.216, for production: 10.2.71.64).

OPSI Prep and Install

We need to flip a few options in Windows to enable certain packages to install without prompting.

  • Do not prompt for unsigned drivers (needed for at least the windows mobile sdk)
    • Control Panel -> System -> Hardware -> Driver Signing, choose "Ignore".
  • Security Settings
    • Start Menu -> Run -> \\$ipOfOpsiServer (staging: \\10.2.71.216, production: 10.2.71.64)
    • Use 'pcpatch' and the root password when prompted.
    • Right click 'opt_pcbin' and select 'Mount as Network Drive'
    • Browse to z:\install\preloginloader
    • Double click 'opsi-prep.vbs'
    • Double click 'service-setup.cmd', hit a key when prompted
    • When prompted for a username and password use 'root' and the appropriate password.

Once OPSI is finished installing it will forcibly reboot the system. In a few minutes it should come back up and the client should be listed in the OPSI GUI.

Autoit

We require Autoit to deploy many packages. To install it, do the following:

  • Checkout the installer:
cvs -d:ext:cltbld@cvs.mozilla.org:/mofo co ref-platforms/win32/autoit-v3-setup.exe
  • (as Administrator), run the installer, accept the defaults

Installing a package on a client

  • Open up the OPSI Config Editor
  • Select a client or clients, click on "Product Configuration"
  • Change the action to "setup" for anything you want to install. (NOTE the column for "action required")

Opsi action setup.png

  • The package(s) will be installed at the next reboot.

Creating a package

Parts of the package

An OPSI package is built from a directory with a specific layout and set of files. These are kept in version control and should be reviewed before check in. The important files in the package are as follows:

  • OPSI/control - This contains the metadata for a package: name, version number, path to install script, etc.
  • CLIENT_DATA/packagename.ins - This is the meat of a package. All of the installation work and setup should be done here. It is written in a language called wInst which is fairly well documented.

Less commonly used files:

  • OPSI/preinst - Executes before the installation script (shell)
  • OPSI/postinst - Executes after the installation script (shell)

Any installers or files a package depends on should be placed in the CLIENT_DATA/ directory, too. That entire directory is downloaded to the client before the installation script executes. Files in it can be referenced from within the installation script.

Bootstrapping a new package

To get the basic files/directories needed to start a new package you should do the following: 

hg clone http://hg.mozilla.org/build/opsi-package-sources
cd opsi-package-sources
rsync -av newproduct-template/ myproduct/

Next:

  • Edit OPSI/control with the product details.
  • Put package files in CLIENT_DATA/.
  • Create an installation script.

Installation script basics

This document doesn't cover wInst syntax in detail, please refer to the wInst manual for that.

Installing an MSI

This is by far the easiest type of package to install. Almost all MSIs have a silent mode which you can invoke. For example, here is the installation script for Visual Studio 2005: 

[Initial]
Message=installing Visual Studio 2005 ...
StayOnTop=false
[Aktionen]
  Winbatch_product_silent_install
[Winbatch_product_silent_install]
msiexec.exe /i %SCRIPTPATH%\vs_setup.msi TRANSFORM=%SCRIPTPATH%\Setup\vs.ini /passive /l* c:\visualstudioinstall.log

Installing a package without a silent install

A lot of packages don't offer silent install options. To work around this, we depend on a tool called Autoit. Autoit is a tool which automates mouse clicks and keyboard strokes. Here's how to create a package that uses Autoit:

  • Use the Autoit "Script Writer" to record the necessary clicks and strokes to install your package. Try to keep unnecessary mouse movement to a minimum while recording to avoid bloating the script.
  • Test the script to make sure it works.
  • Put the script and the package in the CLIENT_DATA/ directory.
  • Create the OPSI installation script and call Autoit from it. For example, here's the installation script for MozillaBuild 1.3:
Initial]
Message=MozillaBuild 1.3 Installation
StayOnTop=false

[Aktionen]
Winbatch_install_mozillabuild

[Winbatch_install_mozillabuild]
autoit3.exe %SCRIPTPATH%\mozillabuild.au3 %SCRIPTPATH%\MozillaBuildSetup-1.3.exe

Updating registry keys

The wInst language provides a nice built-in way to update registry keys as part of an installation. Rather than (or in addition to) calling out to an installer you can write registry keys in the installer. For example, here's how to update the system PATH to append "d:\mozilla-build\python25\scripts": 

[Initial]
Message=Installing new registry keys
StayOnTop=false

[Aktionen]
Registry_AddPath

[Registry_AddPath]
openkey [HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment]
supp "Path"; "d:\mozilla-build\python25\scripts"


Testing the package on staging

On staging you want to test out the package first to make sure it installs correctly before it goes into production. To create a .opsi package and inform OPSI about it, do the following: (For staging): 

# as root on the opsi server
cd ~cltbld/opsi-package-sources
su -c './regenerate-package dir-with-your-package'
# to confirm that what you want installed on the slave is correct
# look in:
/opt/pcbin/install/dir-with-your-package

If there are any issues and you need to regenerate the package you can simply drop in the new code and run the above again.

Installing the package file on production

Please note that you should NOT install packages onto the production OPSI server unless they have been properly reviewed and checked in. That means your package should be landed in opsi-package-sources (hg), and you should land any binaries in opsi-binaries as cltbld@staging-opsi (CVS).

(For production): 

# as cltbld on the opsi server
# update binaries
cd ~cltbld/opsi-binaries
cvs up -Pd

# update the package source clone
cd ~cltbld/opsi-package-sources
hg id && hg in -v
hg pull && hg up
./sync-binaries

# Recreate and install the package
su -c './regenerate-package package-name'

Your package should now show up in the OPSI Config Editor. Make sure to set staging slaves to install/update the package as well as production ones.

Troubleshooting

Check which master

  • Open regedit and browse to HKEY_LOCAL_MACHINE\SOFTWARE\opsi.org\pcptch
  • "opsiServiceURL" key should point to XXX for production or "https://10.2.71.216:4447" for staging.

Check the hostkey

  • Check that the hostkeys on the master and the slave are the same
    • in master: /etc/opsi/pckeys
    • in slave: C:\Program Files\opsi.org\preloginloader\cfg\locked.cfg
  • the hostkey is generated with a service on boot Win32#OPSI_Hostkey_Generator
    • ssh into staging-opsi and check /home/cltbld/opsi-package-sources/hostkey-generator/CLIENT_DATA/hostkey.ins to see what it does
      • TODO where to check the service on the windows machine

Check the opsiconfd log

  • select the slave from the OPSI.jar menu
  • select the tab that says "Log files"
  • select the sub-tab named "opsiconfd"
  • go to the bottom to see the latest messages

uib.local VS .build.mozilla.org

bhearsum: it indicates what domain the OPSI server thinks it's in.
          it's long standing that it's different between production and
          staging, but it shouldn't be an issue

Re-installing OPSI

This is probably not need anymore. Check with bhearsum.

NOTE: This instructions were build up from this page.

  • change the opsiServiceURL to point to staging (explained above)
  • In c:\program files\opsi.org\preloginloader\uninst run deinstall.cmd
    • the slave will reboot
  • disconnect the mapped drive for production opsi
  • install OPSI from staging-opsi
    • Start Menu -> Run -> \\10.2.71.216
    • Use 'pcpatch' and the root password when prompted. (Use the geTxxxx password until bug 519989 is solved)
    • Right click 'opt_pcbin' and select 'Mount as Network Drive'
    • Browse to z:\install\preloginloader
    • Double click 'opsi-prep.vbs' (you won't see anything happen)
    • Double click 'service-setup.cmd', hit a key when prompted
    • When prompted for a username and password use 'root' and the appropriate password (the current one unlike for "pcpatch")
    • Press any key (I assume it should have rebooted after this but it didn't - armenzg)

[really? it didn't happen to me - armenzg] OPSI forces reboot on completion of install and then you can manage it through the opsi client.

Mit Netzlaufwerken verbinden, bitte noch etwas warten

It translates to: Connecting to network drives, please wait a bit

TODO does it take few minutes?

error: HTTP/1.1 401 Unauthorized

This error appears before logging in. I believe you can only stop "the installation" and be able to access the machine.

error: HTTP/1.1 401 Unauthorized
bhearsum: This usually means that the slave is talking to the wrong opsi server,
or that the hostkey on the slave doesn't match the one the server.

I assume we have to check the following:

Wrong hostkey

Failed to process rpc: opsiHostKey authentication failed for host 'talos-r3-xp-001.build.mozilla.org': wrong key (opsiconfd|623)

How to update the opt_pcbin drive mapping password - "pcpatch"

NOTE: this has never been tested.

In bug 519989, we did some preliminary investigation into how we might go about updating the password used for mounting the OPSI shared package drive in case we happen to expose it:

  1. Arrange for a long downtime. Depending about how comfortable you are about burning builds in progress, you'll likely need between 3-6 hours.
  2. Shutdown, possibly gracefully, any affected buildbot masters.
  3. Push out an OPSI product that changes the password but leaves the pcpatch user password as it is (for now)
  4. Reboot the clients, and during their OPSI check-in the change is made. The change is made after the mount is authenticated, so they *should* be fine this time around.
  5. After all the clients have the new package (as verified through the OPSI dashboard), change the pcpatch user password on the server. Future reboots will work with the new password.
  6. Reboot the clients again to make sure they can connect.

Further background can be found in bug 519989.

Documentation/Links

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/OPSI&oldid=274324"