ReleaseEngineering/Infrastructure/Signing

NOTE: This is a draft. This info has to be verified by bhearsum or someone that knows more about these systems.

Introduction

Signing servers use redis to share the valid tokens amongst the different servers.

We have 4 Mac signing servers (mac-signing[1-4]) and 3 Linux signing servers (signing{4,5,6})

How it works

Build machines, before they upload their files to FTP or S3, need their build files signed. The per-check-in and nightly builds (not the debug builds) submit the files that need signing to the signing servers, however, they initially need to request a valid token.

Download token

Every build has a step called "download token". In this step the buildbot master will contact one of the signing servers, receive back a signing token, and download it to the build machine.

Signing

The downloaded token will be used to talk to the signing server to sign the files (the -t flag specifies the location of the token):

python /builds/slave/m-in-l64-000000000000000000000/tools/release/signing/signtool.py --cachedir /builds/slave/m-in-l64-000000000000000000000/signing_cache -t /builds/slave/m-in-l64-000000000000000000000/token -n /builds/slave/m-in-l64-000000000000000000000/nonce -c /builds/slave/m-in-l64-000000000000000000000/tools/release/signing/host.cert -H signing4.srv.releng.scl3.mozilla.com:9110 -H signing5.srv.releng.scl3.mozilla.com:9110 -H signing6.srv.releng.scl3.mozilla.com:9110 -f gpg '../../dist//firefox-29.0a1.en-US.linux-x86_64.checksums'

The signing server will recognize the token as valid since it had issued earlier.

NOTE: I think the upload step will rotate through the list of signing servers until it finds the one that will accept the token on-disk.

Known failures

Invalid Token *could* be due to several Redis restart

bug 957627 - Invalid token *could* be due to a redis restart. The issues, in this case, are transient.

2014-01-08 09:25:10,475 - 60573158673390f6896d181698326c76d9a83f22: processing ../../dist//firefox-29.0a1.en-US.linux-x86_64.checksums on https://signing4.srv.releng.scl3.mozilla.com:9110
2014-01-08 09:25:11,115 - 60573158673390f6896d181698326c76d9a83f22: uploading for signing
2014-01-08 09:25:11,849 - 60573158673390f6896d181698326c76d9a83f22: error uploading file for signing: 400 Invalid token

Trouble shooting

Sign into the signing servers and look at the logs.

TODO: Add some docs about this.