ReleaseEngineering/Applications/Tooltool: Difference between revisions

Jump to navigation Jump to search

ReleaseEngineering/Applications/Tooltool (view source)

Revision as of 22:13, 6 April 2015

11,703 bytes removed ,  6 April 2015
remove old cruft
(Add docs for uploading using the new relengapi based tooltool)
(remove old cruft)
Line 99: Line 99:
   TOOLTOOL_HOME: the folder containing tooltool.py
   TOOLTOOL_HOME: the folder containing tooltool.py
   TOOLTOOL_CACHE: the local folder being used as tooltool cache (which will be cleaned up)
   TOOLTOOL_CACHE: the local folder being used as tooltool cache (which will be cleaned up)
== Using multiple servers with tooltool ==
The original implementation of tooltool supports one single download server: whenever using tooltool to fetch artifacts, a single server was specified and tooltool tried to download the file from that location exclusively.
The need to support multiple server is justified by at least two reasons:
* improve tooltool resiliency in case one server is not available, so that requests are made to a backup server
* allow the setup of tooltool repositories with different levels of visibility
The following example illustrates how to use tooltool with multiple servers:
  python  tooltool.py fetch --url http://server1.example.com --url http://server2.example.com
In this case, tooltool will try to fetch the desired artifacts from server1 and, in case the server is down or does not serve the desired resource, will try with server2. An arbitrary number of servers is supported by the tooltool client. Of course, in order to use this feature the servers will need to be setup.
See also [https://bugzilla.mozilla.org/show_bug.cgi?id=768123 https://bugzilla.mozilla.org/show_bug.cgi?id=768123].


= How Tooltool is invoked in build/test jobs =
= How Tooltool is invoked in build/test jobs =
Line 123: Line 106:
* TooltoolMixin class for mozharness calls: http://hg.mozilla.org/build/mozharness/file/b2fdf31664ed/mozharness/mozilla/tooltool.py (despite being an Homonym of tooltool itself, this is a python module defining TooltoolMixin class to be used in mozharness calls).
* TooltoolMixin class for mozharness calls: http://hg.mozilla.org/build/mozharness/file/b2fdf31664ed/mozharness/mozilla/tooltool.py (despite being an Homonym of tooltool itself, this is a python module defining TooltoolMixin class to be used in mozharness calls).


= How to upload to tooltool =
= How To Upload To Tooltool  =


Tooltool uploads have been traditionally managed by simply scp'ing files to the folder served by the apache instance working as tooltool download server. A new upload mechanism has been implemented to allow better tracking of what is uploaded to tooltool (see [https://bugzilla.mozilla.org/show_bug.cgi?id=772190 "Bug 772190 - tooltool upload mechanism"]).
{{note|Make sure you're using the latest tooltool.py from https://github.com/mozilla/build-tooltool}}
 
The new upload procedure allow users to upload artifacts using tooltool itself to a dedicated upload server, contextually providing some metadata about the upload. The artifacts uploaded to the upload servers will then be distributed to the relevant tooltool download servers (and their mirrors, if any) by a separate sync script which is cron'd on the upload server itself.
 
== Preliminary notes: ==
 
* The server used for uploads (AKA tooltool upload server) is tooltool-uploads.pub.build.mozilla.org (no VPN required)
* Currently, artifacts uploaded to the tooltool upload server can be made visible to only one tooltool download server:
:* Private tooltool download server http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512
::* If you don't have VPN access: https://secure.pub.build.mozilla.org/tooltool/pvt/build/sha512
::* all uploads to /tooltool/uploads/user/pvt on the upload server (more details below) will be visible here
 
== Pre-requisites to run an upload ==
 
* You need to be added to the "vpn_tooltooleditors" LDAP group (see below)
* You need ssh properly configured to access the tooltool upload server; if the key you normally use to access mozilla servers is ~/.ssh/id_rsa, no further setup is needed. If it's a different one, you will need to add the following entry to your ~/.ssh/config file:
 
    Host tooltool-uploads.pub.build.mozilla.org
      IdentityFile /path/to/my_private_key
      User my_user
 
Note that "my_user" above is your LDAP username without the "@mozilla.com" part.
 
* You need rsync installed and available in your PATH (it is used by tooltool to transfer files)
* You need tooltool! https://github.com/mozilla/build-tooltool/blob/master/tooltool.py ([https://github.com/mozilla/build-tooltool/raw/master/tooltool.py raw download])
 
=== How to enable a user to run tooltool uploads by adding him/her to the vpn_tooltooleditors LDAP group ===
First, request addition to "vpn_tooltooleditors" group, via "Infrastruture & Operations::Infrastructure:LDAP".
Once that's done, it's an hour or two for the change to propagate from LDAP to Puppet to the upload servers.
They should request an ACK on the bug from someone in release engineering for that.
 
== How to upload files to tooltool ==
 
* Create a local folder (name it as you like) and put in it all the files you want to upload
* Run the tooltool distribute command (see example below)
 
That's it!
 
Example:
  # Folder /Users/$user/upload contains file1.tar.gz, file2.tar.gz and file3.tar.gz
  tooltool.py distribute --folder /absolute/path/to/where/files/to/upload/are --message "Bug 123456 - artifacts needed for this and that" \
    --user $user --host tooltool-uploads.pub.build.mozilla.org --path "/tooltool/uploads/$user/pvt"
 
Note that "$user" above is your posix UID, i.e. your LDAP username without the "@mozilla.com" part (or different domain).
 
The parameters in the previous command are:
 
  --folder: the folder containing the files you want to upload
  --message: any comment you may want to write about this upload
  --user: the user to access the tooltool upload server
  --host: the upload server host or IP
  --path: the target folder on the upload server
 
Use /tooltool/uploads/<your_username>/pvt to upload to the private download server
 
=== Troubleshooting ===
 
You may incur in rsync errors like:
 
    rsync: connection unexpectedly closed (36 bytes received so far) [sender]
 
In this case, it may help to add the following entries to your ~/.ssh/config file:
 
    Host tooltool-uploads.pub.build.mozilla.org
      ServerAliveInterval 5
      ServerAliveCountMax 120
 
== What now? ==
 
You should receive in a few minutes (5 at most, since the sync script processing uploaded files currently runs every 5 minutes) an email confirming that your upload has been correctly processed on the upload server, and the files should appear in the relevant download server with their correct digest (and in all mirrors, if any, setup for the same visibility level).
 
The default email address used for these notifications is user@mozilla.com, where user is the user you use to access the upload server. If you want a different email address to be used for tooltool upload notifications, that will need to be added to the sync script configuration by somebody in releng.
 
All metadata about the upload is stored in the upload server.
 
= How To Upload To Tooltool (new, not deployed yet) =
 
{{note|This won't work until {{Bug|1148545}} is closed; at that point, make sure you're using the latest tooltool.py from https://github.com/mozilla/build-tooltool}}


First, plan out your upload.  Are all of the files you need to upload public?  If not, tooltool also offers storage of "internal" files, which are not made publicly available but can be downloaded by those with proper permissions.  Internal files might be under a non-redistribution license (e.g., Google SDKs, Microsoft DLLs), but '''most not include secrets''' such as passwords or private keys.
First, plan out your upload.  Are all of the files you need to upload public?  If not, tooltool also offers storage of "internal" files, which are not made publicly available but can be downloaded by those with proper permissions.  Internal files might be under a non-redistribution license (e.g., Google SDKs, Microsoft DLLs), but '''most not include secrets''' such as passwords or private keys.
Line 244: Line 149:


* Add the filename, filesize, and sha512 digest to the bug you are working on.  These can be added to the tooltool manifests later.
* Add the filename, filesize, and sha512 digest to the bug you are working on.  These can be added to the tooltool manifests later.
= The tooltool sync script =
For the curious, here is a description of what happens on the upload server once an upload is completed.
As we saw so far, tooltool uploaders copy hashed files and manifests to their own upload folders - they have one for each "distribution type", typically a pub one (to distribute files publicly) and a pvt one (for files to be available to mozilla employees only).
The tooltool sync script serves the purpose of collecting all uploads from individual uploaders and aggregating them per distribution type (public, private), storing some metadata about the uploads themselves. It is supposed to run periodically via crontab to periodically pick new uploads and distribute their content.
== Where is the sync script code? ==
In the main tooltool repo: [https://github.com/mozilla/build-tooltool/blob/master/sync.py https://github.com/mozilla/build-tooltool/blob/master/sync.py]
== Where is the sync script running? ==
The sync scripts is located in /data/releng/src/tooltool/tooltool on relengwebadm.private.scl3.mozilla.com, where configuration (see below) and logs also live.
== How do I deploy a new version of the sync script? ==
Just grab the desired version from the github repo and copy it to the location where sync.py lives (see previous paragraph)
== The sync script configuration ==
The sync script needs a configuration file in json format named config.json, let's see an example here:
  {
    "upload_root": "/mnt/netapp/tooltool_uploads",
    "target_folders": {"pvt": "/mnt/netapp/relengweb/tooltool/pvt/build/sha512",
                      "pub": "/mnt/netapp/relengweb/tooltool/pub/build/sha512"},
    "smtp_server": "localhost",
    "smtp_port": 25,
    "user_email_mapping": {"dmitchell": "dustin@mozilla.com"},
    "default_domain": "mozilla.com",
    "smtp_from": "no-reply@mozilla.org"
  }
* '''upload_root''' is the base directory in which all personal uploaders directories are located
* '''target_folders''' for each supported "distribution type" (in this case "pvt" and "pub"), the corresponding local destination folder (where all the files from different uploaders will be collected) is specified here
* '''smtp*''' will be used for notification emails to users after uploads
* '''default_domain''' is used to determine the email address of a user which is not explicitly mentioned in the "user_email_mapping" part: the address will be simply username@defaultdomain
In the case of two uploaders, Alice and Bob; this is how the (tooltool) root folder will look like:
  root
    alice
          pub
          pvt
    bob
          pub
            a-package.TOOLTOOL-PACKAGE             
            a_package.tt
            a_package.txt
          pvt
Each user has two upload folders, one for each supported distribution type. The sync script supports an arbitrary number of distribution types, as far as they are defined in the configuration file in the matching section
In its next execution, the sync script will detect the manifest, check that all referenced hash files are present, verify their integrity, and it will proceed copying the hashed files to the destination provided in the config file. Some upload metadata will be stored in text file.
== The sync script in action ==
[[File:Sync script in action.png|frameless]]
In more detail, the sync script will perform the following actions:
=== Step 1: Copy of hash files to the appropriate destination ===
All files mentioned in package_test1.tt manifest, in Bob's pub folder, will be hash checked and copied to the destination specified in the conf file (in this case, "/tooltool/servers/pub"). This will make the files available for tooltool downloads.
Please note that in the following circumstances the copy will not occur:
* if at least one of the files mentioned in the manifest are missing (for example in case of a partial upload)
* if at least one of the hashed files' hashes are different from their filename (hash check failure)
If one of these circumstances occur, the sync script will just log that it is impossible to process the given manifest and send a notification message to the uploader.
=== Step 2: Store locally a copy of the processed manifest ===
After processing a manifest, the sync script with store locally a copy of the processed manifest, prepending the user, the distribution type, and the processing timestamp to its name. The same will be done with the txt file containing comments (if any).
In this example we will have a manifest stored locally (and a corresponding notes file) as
  bob.pvt.2013_10_16-10.20.29.a-package.tt
  bob.pvt.2013_10_16-10.20.29.a-package.txt
=== Step 3: Rename the processed manifests in the uploader folder as well ===
The content of Bob's public folder after sync script processed its manifest will be the following:
  /tooltool/uploads/bob/pvt
      2013_10_16-10.20.29.a-package.tt
      2013_10_16-10.20.29.a-package.txt
The processed package has been deleted, and only the manifest with the corresponding note is present, with the processing timestamp; this means that the manifest has been successfully processed by the sync script.
=== Step 4: Notify the uploader of success/failure ===
An email will be sent to the uploader, notifying whether the sync process was completed succesfully or failed for some reasons.
If the sync process fails, the corresponding files will obviously not be available for tooltool downloads.
= Need to download files from tooltool private server? =
If you want to download a file from tooltool private server and you're not part of ReleaseEngineering then add your name to {{bug|1019011}}.
That should grant you access to https://secure.pub.build.mozilla.org/tooltool/pvt/build
Djmitche
canmove, Confirmed users
1,394

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1066607"

Navigation menu