ReleaseEngineering/Infrastructure/Signing

NOTE: This is a draft. This info has to be verified by bhearsum or someone that knows more about these systems.

Introduction

Signing servers use redis to share the valid tokens amongst the different servers.

We have 4 Mac signing servers (mac-signing[1-4]) and then 3 Linux signing servers (signing{4,5,6})

How it works

Build machines, before they upload their files to FTP or S3, need their build files signed. The per-check-in and nightly builds (not the debug builds) submit the files that need signing to the signing servers, however, they initially need to request a valid token.

Download token

Every build has a step called "download token". In this step the build machine will contact one of the signing servers to download a token into disk (I think).

Signing

The downloaded token will be used to talk to the signing server to sign the files (the -t flag specifies the location of the token):

python /builds/slave/m-in-l64-000000000000000000000/tools/release/signing/signtool.py --cachedir /builds/slave/m-in-l64-000000000000000000000/signing_cache -t /builds/slave/m-in-l64-000000000000000000000/token -n /builds/slave/m-in-l64-000000000000000000000/nonce -c /builds/slave/m-in-l64-000000000000000000000/tools/release/signing/host.cert -H signing4.srv.releng.scl3.mozilla.com:9110 -H signing5.srv.releng.scl3.mozilla.com:9110 -H signing6.srv.releng.scl3.mozilla.com:9110 -f gpg '../../dist//firefox-29.0a1.en-US.linux-x86_64.checksums'

The signing server will recognize the token as valid since it had issued earlier.

NOTE: I think the upload step will rotate through the list of signing servers until it finds the one that will accept the token on-disk.

Known failures

Invalid Token

• bug 957627 - "400 Invalid token"

2014-01-08 09:25:10,475 - 60573158673390f6896d181698326c76d9a83f22: processing ../../dist//firefox-29.0a1.en-US.linux-x86_64.checksums on https://signing4.srv.releng.scl3.mozilla.com:9110
2014-01-08 09:25:11,115 - 60573158673390f6896d181698326c76d9a83f22: uploading for signing
2014-01-08 09:25:11,849 - 60573158673390f6896d181698326c76d9a83f22: error uploading file for signing: 400 Invalid token

Redis has probably been restarted recently. The issues are transient.

Trouble shooting

Sign into the signing servers and look at the logs.

TODO: Add some docs about this.