ReleaseEngineering/Applications/Tooltool

Original Post

One fairly common problem faced by teams at Mozilla is getting their tools plugged into the Release Engineering continuous testing infrastructure. In the past, the workflow has been for the team to file a bug that required a Release Engineer to figure out how to build, package and deploy a new set of tools. This is not the most scalable approach. During our recent Release Engineering workweek, we brainstormed on how to improve scaling and agility by empowering developers to help build and deploy new tools for our CI machines.

Tooltool basics

Tooltool is a client side program written in Python that uses a file manifest in concert with HTTP servers to retrieve sets of files and perform integrity checks on them based on hashcode verification. The manifests are JSON files which list details of individual files (see below). Each file is represented in the JSON by a dictionary with the keys “filename”, “digest”, “size” and “algorithm”.

A set of files to be fetched by tooltool is specified via a tooltool manifest, usually a file with tt extension.

The following is an example of a valid tooltool manifest:

 [
 {
 "size": 139308,
 "digest": "b2a463249bb3a9e7f2a3604697b000d2393db4f37b623fc099beb8456fbfdb332567013a3131ad138d8633cb19c50a8b77df3990d67500af896cada8b6f698b4",
 "algorithm": "sha512",
 "filename": "file2.pdf"
 },
 {
 "size": 3017536,
 "digest": "630d01a329c70aedb66ae7118d12ff7dc6fe06223d1c27b793e1bacc0ca84dd469ec1a6050184f8d9c35a0636546b0e2e5be08d9b51285e53eb1c9f959fef59d",
 "algorithm": "sha512",
 "filename": "file1.pdf"
 },
 {
 "size": 3420686,
 "digest": "931eb84f798dc9add1a10c7bbd4cc85fe08efda26cac473411638d1f856865524a517209d4c7184d838ee542c8ebc9909dc64ef60f8653a681270ce23524e8e4",
 "algorithm": "sha512",
 "filename": "file3.pdf"
 }
 ]

The simplest usecase for tooltool is to provide a manifest and the url of a tooltool server, and run a "fetch" command to download the files mentioned in the manifest:

 python  tooltool.py fetch --url http://tooltool.pub.build.mozilla.org/temp-sm-stuff -m my-manifest.tt

If a manifest name is not provided, tooltool will default to manifest.tt.

Tooltool will build an url for each of the mentioned files concatenating the server url and the file digest, and try to download.

In this example, the urls will be:

 http://tooltool.pub.build.mozilla.org/temp-sm-stuff/58941214a8334331e52114aab851fc3d8d5da5dd14983f933da8735c24b0ddcac134e8f13692553199c4d9a14a4b3188b62878a30b9d696edda1204666b60837
 http://tooltool.pub.build.mozilla.org/temp-sm-stuff/b2a463249bb3a9e7f2a3604697b000d2393db4f37b623fc099beb8456fbfdb332567013a3131ad138d8633cb19c50a8b77df3990d67500af896cada8b6f698b4
 http://tooltool.pub.build.mozilla.org/temp-sm-stuff/630d01a329c70aedb66ae7118d12ff7dc6fe06223d1c27b793e1bacc0ca84dd469ec1a6050184f8d9c35a0636546b0e2e5be08d9b51285e53eb1c9f959fef59d
 http://tooltool.pub.build.mozilla.org/temp-sm-stuff/931eb84f798dc9add1a10c7bbd4cc85fe08efda26cac473411638d1f856865524a517209d4c7184d838ee542c8ebc9909dc64ef60f8653a681270ce23524e8e4

After downloading the files, their digest is verified according to the algorithm specified in the manifest, and they are finally renamed according to the filename specified in the manifest.

The tooltool servers are simply apache server folders with a flat structure (all files are stored at root level with no subfolders).

There are global options and command arguments. All terminal interactions after the option parser finishes is done through the Python logging API. The default is to print logging.INFO and higher messages. Currently, the following global options exist:

   -q/--quiet tells Tooltool to print only logging.ERROR and higher messages
   -v/--verbose specifies to print logging.INFO and higher
   -m/--manifest <file> instructs Tooltool to reference a manifest file located at the specified path
   -d/--algorithm <algorithm> instructs Tooltool to use the specified algorithm
   -o/--overwrite tells Tooltool to overwrite a local file if the filename matches the manifest but the hash value is different to the manifest
   --url specifies the base url to be used for remote operations

Listing and Validating

The two most basic commands list a manifest and validate the local files against the manifest. The list command lists out all of the files in the manifest as well as whether they are present and/or valid. The return code from listing is zero unless there was an error in listing the files. Absent or invalid files will still result in an exit code of zero if there was no error in the listing process. The validate command is used to check if all the files in the manifest are present and valid. The exit code for validating is zero if all files in the manifest are present and their hash matches the manifest. It is non-zero if any file is missing locally or the file does not have the same hash as the manifest.

Adding

Files are added to manifests with the add command. This command looks for a manifest using either the default name of manifest.tt or by the value specified by the -m/–manifest command line option. If the manifest does not exist already on the file system, a new one will be created to store the first file given. An error message will be displayed if a file is added a second time, regardless of whether or not the local contents are the same as what is in the manifest. There is currently no logic to remove a file from a manfiest or overwrite a manifest entry.

Creating a Tooltool manifest, adding a file to it, trying to re-add file and listing contents. Note that even though ‘a’ didn’t change, tooltool didn’t allow it to be added a second time or overwrite the manifest’s entry

Future direction and contributions

Tooltool is a generic lookaside cache. My intention is to keep it as general as possible by not including logic to deal with payloads. We currently include a bootstrap script in the b2g manifests that understands how to take the rest of the payload and set up a working toolchain. Using a bootstrap script means that Tooltool can be tool agnostic while still allowing complex operations on the fetched tools. A standardized bootstrap script name and interface that is called by Tooltool might make sense. I’d also like to finish the server-side component that has an interface to upload files with and a method for storing files in a less obtuse method than just mirroring the API paths. A command for removing files from a manifest would be helpful, as would the ability to update a manifest with the contents of the directory as they exist right now. Another really cool feature would be the ability to configure a system wide cache directory where files are downloaded to. Once the server component is working, I’d like to add a way for servers to automatically sync their file stores when they are asked for a file that exists on another server but not locally.

Tooltool is GPLv2 and the source is available on github. The best way to contribute code is to send a pull request on github. Things are more likely to be merged if they pass the whole test suite (make check), have tests, improve Tooltool and don’t make Tooltool overly specific.

Upload to tooltool

See ReleaseEngineering:Buildduty:Other Duties#How to upload to Tooltool

Need to download files from it?

If you want to download a file from tooltool and you're not part of ReleaseEngineering then add your name to bug 1019011.

That should grant you access to https://secure.pub.build.mozilla.org/tooltool/pvt/build