MozillaWiki - User contributions [en]

Template:Version/Gecko/central/current

2018-10-22T17:22:12Z

Garbas: bump version

Template:Version/Gecko/release/next

2018-10-22T17:21:55Z

Garbas: bump version

ReleaseEngineering/Applications/RelengAPI

2018-07-04T20:12:08Z

Garbas: adding obsolete warning

{{Obsolete RelEng Page|Future development of RelEng applications was moved to https://docs.mozilla-releng.net}}

As [[ReleaseEngineering/Applications]] attests, releng has a lot of apps, and many of them have APIs.
But these APIs are all bespoke implementations, and are not tied to a single endpoint, authentication scheme, etc.
RelengAPI aims to be the glue that binds all of these apps together, makes it easy to add new ones, and makes it easy for others to build tools that consume releng's APIs.

= Goals =
* Simple self-service usage for consumers
* Simple, rapid implementation of new apps
* Operations-friendly
See [https://github.com/mozilla/build-relengapi#goals README.md] for more

= Links =

* Code: https://github.com/mozilla/build-relengapi
* Deployment: https://api.pub.build.mozilla.org (production) or https://api-pub-build.allizom.org (staging)
* Documentation: https://api.pub.build.mozilla.org/docs
* [[ReleaseEngineering/How_To/Update_RelengAPI|How To: Update RelengAPI]]

= Getting Involved =

* IRC: #releng, look for dustin
* Patches: releng and relops can push directly for now; GitHub pull requests are welcome from others
* Bugs/RFE's: Use [https://github.com/mozilla/build-relengapi/issues GitHub issues] for minor issues; for major work that must be tracked within Mozilla, file bugs in "Release Engineering: Other", cc :dustin

ReleaseEngineering/Applications/Clobberer

2018-07-04T20:06:11Z

Garbas: improved formatting of the warning

{{Obsolete RelEng Page|With migration to Taskcluster Clobberer is not anymore needed and will be turned off in September, 2018.}}

= Application Description =
The clobberer allows developers to "clobber" particular build directories (objdirs) on particular slaves. This is sometimes necessary if a change is committed that is not compatible with previous objdirs, or occasionally when a compile failure results in a bogus objdir.

The application has an outward-facing web interface for developers, and also provides an internal web service which buildslaves contact to determine what build directories they should clobber.

As with all other RelengAPI integrated applications, clobberer is completely API based. For information about its endpoints see:

https://api.pub.build.mozilla.org/clobberer/

* UI
* production
* https://api.pub.build.mozilla.org/clobberer/ - https
* staging
* https://api-pub-build.allizom.org/clobberer

= Requirements =
Python

= Resources =
* two MySQL databases (production and staging)
* runs on the [https://mana.mozilla.org/wiki/display/websites/Releng+web+cluster releng web cluster]

= Security =
Clobberer uses the LDAP username verified by RelengAPI's permissions system.

= Testing/Updating =

Clobberer is a Flask blueprint, and part of the RelengAPI project. For information about Testing/Updating see the project readme: https://github.com/mozilla/build-relengapi-clobberer

= More =
See [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=24805981 the websites page]

ReleaseEngineering/Applications/Clobberer

2018-07-04T20:04:00Z

Garbas: adding deprecation warning

= DEPRECATION WARNING !!! =

With migration to Taskcluster Clobberer is not anymore needed and will be turned off in September, 2018.

= Application Description =
The clobberer allows developers to "clobber" particular build directories (objdirs) on particular slaves. This is sometimes necessary if a change is committed that is not compatible with previous objdirs, or occasionally when a compile failure results in a bogus objdir.

The application has an outward-facing web interface for developers, and also provides an internal web service which buildslaves contact to determine what build directories they should clobber.

As with all other RelengAPI integrated applications, clobberer is completely API based. For information about its endpoints see:

https://api.pub.build.mozilla.org/clobberer/

* UI
* production
* https://api.pub.build.mozilla.org/clobberer/ - https
* staging
* https://api-pub-build.allizom.org/clobberer

= Requirements =
Python

= Resources =
* two MySQL databases (production and staging)
* runs on the [https://mana.mozilla.org/wiki/display/websites/Releng+web+cluster releng web cluster]

= Security =
Clobberer uses the LDAP username verified by RelengAPI's permissions system.

= Testing/Updating =

Clobberer is a Flask blueprint, and part of the RelengAPI project. For information about Testing/Updating see the project readme: https://github.com/mozilla/build-relengapi-clobberer

= More =
See [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=24805981 the websites page]

ReleaseEngineering/Applications/Tooltool

2018-06-30T02:14:10Z

Garbas: new tooltool url

= Tooltool basics =

Some of the jobs in the releng infrastructure make use of generic binary artifacts, which are stored in dedicated artifacts repositories.

Tooltool is the client program, written in Python, which is used to retrieve these artifacts from the servers, and to perform integrity checks on them based on hashcode verification. Hence, the servers storing these artifacts are named "tooltool servers".

A set of files to be fetched by tooltool is specified via a tooltool manifest, usually a file in JSON format with tt extension.

The following is an example of a valid tooltool manifest:

[
{
"size": 139308,
"digest": "b2a463249bb3a9e7f2a3604697b000d2393db4f37b623fc099beb8456fbfdb332567013a3131ad138d8633cb19c50a8b77df3990d67500af896cada8b6f698b4",
"algorithm": "sha512",
"filename": "file2.pdf"
},
{
"size": 3017536,
"digest": "630d01a329c70aedb66ae7118d12ff7dc6fe06223d1c27b793e1bacc0ca84dd469ec1a6050184f8d9c35a0636546b0e2e5be08d9b51285e53eb1c9f959fef59d",
"algorithm": "sha512",
"filename": "file1.pdf"
},
{
"size": 3420686,
"digest": "931eb84f798dc9add1a10c7bbd4cc85fe08efda26cac473411638d1f856865524a517209d4c7184d838ee542c8ebc9909dc64ef60f8653a681270ce23524e8e4",
"algorithm": "sha512",
"filename": "file3.pdf"
}
]

The simplest usecase for tooltool is to provide a manifest and the url of a tooltool server (this url is not longer used, the last modified file is from 13-Oct-2015) , and run a "fetch" command to download the files mentioned in the manifest:

python tooltool.py fetch -m my-manifest.tt

If a manifest name is not provided, tooltool will default to manifest.tt.

Tooltool is build on https://mozilla-releng.net/tooltool/ and to find the uploaded file you can search after : Filename,SHA512 hash,Upload author or Upload message (including bug number).

example: search after "mitmproxy"

The tooltool download servers are simply apache server folders with a flat structure (all files are stored at root level with no subfolders).

There are global options and command arguments. All terminal interactions after the option parser finishes is done through the Python logging API. The default is to print logging.INFO and higher messages. Currently, the following global options exist:

-q/--quiet tells Tooltool to print only logging.ERROR and higher messages
-v/--verbose specifies to print logging.INFO and higher
-m/--manifest <file> instructs Tooltool to reference a manifest file located at the specified path
-d/--algorithm <algorithm> instructs Tooltool to use the specified algorithm
-o/--overwrite tells Tooltool to overwrite a local file if the filename matches the manifest but the hash value is different to the manifest
--url specifies the base url to be used for remote operations

= Where's tooltool code? =

Tooltool's main dev repo is [https://github.com/mozilla/build-tooltool https://github.com/mozilla/build-tooltool].

The version of tooltool which is actually deployed in our infrastructure is in https://github.com/mozilla/build-puppet/blob/master/modules/packages/templates/tooltool.py. The tooltool.py file in the puppet repo should be in sync with the main dev repo (they should be identical, apart from the she-bang line).

= Listing and Validating =

The two most basic commands list a manifest and validate the local files against the manifest. The list command lists out all of the files in the manifest as well as whether they are present and/or valid. The return code from listing is zero unless there was an error in listing the files. Absent or invalid files will still result in an exit code of zero if there was no error in the listing process. The validate command is used to check if all the files in the manifest are present and valid. The exit code for validating is zero if all files in the manifest are present and their hash matches the manifest. It is non-zero if any file is missing locally or the file does not have the same hash as the manifest.

= How To Upload To Tooltool =

{{note|Make sure you're using the latest tooltool.py from https://github.com/mozilla/build-tooltool}}

First, plan out your upload. Are all of the files you need to upload public? If not, tooltool also offers storage of "internal" files, which are not made publicly available but can be downloaded by those with proper permissions. Internal files might be under a non-redistribution license (e.g., Google SDKs, Microsoft DLLs), but '''must not include secrets''' such as passwords or private keys.

Use the `tooltool.py` client to build a manifest containing the files you would like to upload, annotating each file with its visibility level:

python tooltool.py add --visibility public gcc.tar.gz

Next, you will need credentials for the upload. Tooltool is hosted on RelengAPI, so the credentials take the form of a RelengAPI token. To get such a token, go to https://mozilla-releng.net/tokens, log in, click on "Tokens", and issue a user token with the permissions ''tooltool.upload.internal'' and/or ''tooltool.upload.public'', depending on the nature of your upload. If you don't have internal permission, then the option won't be available to you. When you click "Issue", you'll get a long string which you should copy/paste into a (protected) file on your system, and pass to tooltool.py with ''--authentication-file''.

Once all of that is in place, you're ready to perform the upload. Just like for a Git or Mercurial commit, you'll need to provide a message to indicate why you're making this upload. Including a bug reference is always helpful.Also make sure the "file" is in the same repository from where you run tooltool.

python tooltool.py upload --authentication-file=~/.tooltool-token --message "Bug 1234567: add new frobnicator binaries"

The tool will begin uploading and, when it's finished, may wait a minute or two longer until the files are verified by RelengAPI. If your upload fails for some reason, just restart it. Any files already completely uploaded will not need to be uploaded again.

You can view the available files and uploads at https://mozilla-releng.net/tooltool.

= Other tooltool features =

== The tooltool cache ==

This has been implemented in bug [https://bugzilla.mozilla.org/show_bug.cgi?id=858635 https://bugzilla.mozilla.org/show_bug.cgi?id=858635]
All the bellow examples are used for the older packages that are on "http://tooltool.pub.build.mozilla.org/temp-sm-stuff" url.
When connecting to a tooltool server to fetch a file, it is now possible to use a local cache specifying the local cache folder with the -c option, e.g.:

python tooltool.py fetch --url http://tooltool.pub.build.mozilla.org/temp-sm-stuff -c ~/cache

In the previous example, folder ~/cache will be inspected for artifacts before connecting to the specified tooltool server. If the specified cache folder does not exist, tooltool will create it (if the user running tooltool has permissions to do so!).
Purging the cache with the purge command

A mechanism to purge the tooltool cache has been implemented as a separate command. Let's see some examples of purge commands:

python tooltool.py purge -c ~/cache
python tooltool.py purge -c ~/cache -s 34
python tooltool.py purge -c ~/cache --size 34

In the first example, we are cleaning up the tooltool cache completely, i.e.: all the content will be wiped out

In the second and third example the extra parameter -s (or, in extended form, --size) is provided, which specifies the number of Gigs we want to be free at the end of the purging process. Tooltool will delete files in the cache folder, starting with the oldest ones, until the specified amount of gigs is free on disk. When there is more free space that the specified gigs when the command is invokes, no files will be deleted at all.

=== Enabling automated tooltool cache purge within purge_build.py ===

Since the purge_builds.py script (within the build-tools repo) is already responsible of clean-up operations, the option of cleaning the tooltool cache has been added to that script as well.

In order for the automated cleanup to take place, two environment variables need to be setup when purge_builds.py is invoked, and namely:

TOOLTOOL_HOME: the folder containing tooltool.py
TOOLTOOL_CACHE: the local folder being used as tooltool cache (which will be cleaned up)

= How Tooltool is invoked in build/test jobs =

Two wrapper scripts exist which are usually being used to invoke tooltool in build/test jobs:
* tooltool_wrapper.sh: http://hg.mozilla.org/build/tools/file/327616201e98/scripts/tooltool/tooltool_wrapper.sh
* TooltoolMixin class for mozharness calls: https://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/mozharness/mozilla/tooltool.py (despite being an Homonym of tooltool itself, this is a python module defining TooltoolMixin class to be used in mozharness calls).

= Tooltool uploads: the old school =

'''''This is the old-school, deprecated, manual way to upload artifacts to tooltool. Please use the official procedure described above whenever possible, in order to keep information about what is uploaded, by whom, and why (the manual procedure doesn't allow to store any metadata about uploads).'''''

If you don't want to upload from your own laptop (because, eg, you have a slow uplink) you can do this from cruncher.

Access cruncher with your credentials:
ME="your_short_ldap_username" # or `whoami`
ssh -A $ME@cruncher.build.mozilla.org
Download the file and then:
scp filename.tar.xz $ME@relengwebadm.private.scl3.mozilla.com:
Login to relengwebadmn:
ssh $ME@relengwebadm.private.scl3.mozilla.com
And deploy the file to tooltool:
<pre>
FILE=~/emulator.zip # or whatever you're uploading
export SHA512=`openssl sha512 $FILE | cut -d' ' -f2`
sudo mv -i $FILE /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
sudo chmod 644 /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
ls -l /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
</pre>

* Add the filename, filesize, and sha512 digest to the bug you are working on. These can be added to the tooltool manifests later.

ReleaseEngineering/How To/VCSSync

2018-06-30T02:12:23Z

Garbas: new mapper url

{{Release Engineering How To|VCS Sync}}
{{Release Engineering Deployment|VCS Sync}}
This is documentation on the mozharness-based vcs sync processes.

<div id="legacy"></div>
See [http://people.mozilla.org/~hwine/tmp/vcs2vcs/ legacy docs] for the legacy (aka "old") process.

= Quick notes =

"VCS-sync" is a system which has evolved from it's original purpose to being the tool used whenever "this repository needs to be publicly visible 'over here'", perhaps with modifications, and a few exceptions on "publicly visible". The current (2016 Q1) business needs it addresses are:

# Support for gecko developers who prefer to work with git:
#* All gecko primary development and release branches are merged into a single git repository (with CVS history) in "gecko-dev.git".
#* All gecko twig and misc branches are merged into a single git repository (with CVS history) in "gecko-projects.git".
# Support for Mozilla development tools and processes which (for operational needs (e.g. travis-ci), or developer preference, should appear in both Mercurial and Git repositories.
#* These are "mirrors" from one system to the other.
#* One repo is always designated the RoR (Repository of Record), and is the only one where commits will "stick".
#* Refer to the system configurations for each mirror to determine the RoR. (It can, and has, changed over time for some repositories.)
# Support for Partner facing FxOS development processes.
#* The URL's for these repositories can not change, as they are referenced in manifest files which comprise the delivery to Partners.
#* These repositories on git.mozilla.org satisfy various contractual needs with FxOS partners.
#* These repositories must not contain either object deletions, or non-fast-forward commits.
#* These repositories only contain content that was "delivered" to Partners, with branches named according to FxOS conventions.
#* These repositories include any "Mozilla Contributions" to FxOS, which includes code beyond gecko.git and gaia.git.
# Support for internal FxOS development processes.
#* Many Android repositories have been mirrored to git.mozilla.org for the purpose of having a known-good-repository for the build systems to clone from.
#* Some tests still require translation from Git → Mercurial format. Each such gaia.git branch is mapped to a distinct Mercurial repository.

== What's running where ==

Anything not listed below is probably running in the new system. The system pushing the production content is listed first. If the other system is listed, we're in parallel operations mode and changes should be made to both systems.

=== Repositories needed to Build Mozilla Products ===
{| class="fullwidth-table sortable"
| style="background:#cccccc" | '''Repository'''
| style="background:#cccccc" | '''Old/New'''
| style="background:#cccccc" | '''Web views at'''
| style="background:#cccccc" | '''Notes'''
|-
| gecko-dev.git
| new
| [https://github.com/mozilla/gecko-dev.git github], [http://git.mozilla.org/?p=integration/gecko-dev.git;a=summary Moz]
|
|-
| gecko-projects.git
| new
| [https://github.com/mozilla/gecko-projects.git github], [http://git.mozilla.org/?p=integration/gecko-projects.git;a=summary Moz]
|
|-
| releases/gecko.git
| old, new
| [http://git.mozilla.org/?p=releases/gecko.git;a=summary Moz]
| Partner facing FxOS <nowiki>[1]</nowiki>
|-
| releases/gaia.git
| old, new
| [http://git.mozilla.org/?p=releases/gaia.git;a=summary Moz]
| Partner facing FxOS <nowiki>[1]</nowiki>
|-
| releases/l10n/<locale>/{gecko,gaia}.git
| old, new
| [http://git.mozilla.org/ Moz] (browse from there)
| Partner facing FxOS <nowiki>[1]</nowiki>
|-
| integration/gaia*
| old
| [http://hg.mozilla.org/integration/ Moz] (browse from there)
| Hg versions of gaia branches to support older tooling
|-
| b2g mirrors
| old
| [http://git.mozilla.org/ Moz] browse <tt>external</tt> and <tt>b2g</tt> from there (r/o)
| Local copies to ensure availability during builds.
|}

Notes:
# Partner facing repositories are configured to:
#* not accept non fast forward commits
#* not accept deletes
#* only contain branches related to shipping versions of FxOS

=== Repositories Supporting Mozilla Web Products ===
{| class="fullwidth-table sortable"
| style="background:#cccccc" | '''Repository'''
| style="background:#cccccc" | '''Old/New'''
| style="background:#cccccc" | '''Web views at'''
| style="background:#cccccc" | '''Notes'''
|-
| bugzilla-{bugzilla,qa}.git
| old
| [http://git.mozilla.org/ Moz] (r/w), [https://github.com/bugzilla/ github] (r/o)
| Only the bugzilla.git & qa.git repos are mirrored to github
|-
| webtools-bmo-{bugzilla,qa}.git
| old
| [http://git.mozilla.org/ Moz] (r/w), [https://github.com/mozilla/ webtools-bmo-* on github] (r/o)
|
|}

=== Repositories Supporting Infrastructure, Tooling, and Miscellaneous ===
See also [[ReleaseEngineering/Repositories|full list of RelEng repositories]].
{| class="fullwidth-table sortable"
| style="background:#cccccc" | '''Repository'''
| style="background:#cccccc" | '''Old/New'''
| style="background:#cccccc" | '''Web views at'''
| style="background:#cccccc" | '''Notes'''
|-
| build-releng-api.git
| old
| [http://github.com/mozilla/build-releng-api.git github] (r/o), [http://hg.mozilla.org/build/releng-api Moz] (r/o)
|-
| automation/fuzz-tools.git
| old
| [http://git.mozilla.org/?p=automation/fuzz-tools.git Moz] (r/w), [https://github.com/mozilla/automation-fuzz-tools.git github] (r/o)
|
|-
| qa/mozmill-tests
| old
| [http://hg.mozilla.org/qa/mozmill-tests Moz] (r/w), [https://github.com/mozilla/qa-mozmill-tests github] (r/o)
|
|}

== Machines ==
2013.10.14: We're currently using the following machines:

{| class="fullwidth-table sortable"
| style="background:#cccccc" | '''Machine'''
| style="background:#cccccc" | '''Process'''
| style="background:#cccccc" | '''Location'''
| style="background:#cccccc" | '''User'''
|-
| vcssync1.srv.releng.usw2.mozilla.com
| beagle (gecko-dev)
| /opt/vcs2vcs
| vcs2vcs
|-
| vcssync2.srv.releng.usw2.mozilla.com
| project-branches (gecko-projects)
| /opt/vcs2vcs
| vcs2vcs
|-
| github-sync2.dmz.scl3.mozilla.com
| build repos (autoland, buildapi, buildbot-configs, buildbotcustom, cloud-tools, mozharness, opsi-package-sources, partner-repacks, preproduction, puppet, puppet-manifests, rpm-sources, talos, tools)
| /opt/vcs2vcs
| vcs2vcs
|-
| github-sync4.dmz.scl3.mozilla.com
| gecko l10n and gaia l10n
| /opt/vcs2vcs
| vcs2vcs
|}

Plus a [http://people.mozilla.org/~hwine/tmp/vcs2vcs/trouble_shooting.html#accessing-the-machines few other github-sync machines] for legacy processes.

== Branch ==
2013.10.14: We're based off the [http://hg.mozilla.org/build/mozharness mozharness production branch].

= Specific instructions =

== Emails ==

=== Successful noop emails ===

<pre>
From: vcs2vcs@vcssync2.srv.releng.usw2.mozilla.com
Subject: [vcs2vcs] Successful conversion for project-branches (12s) <EOM>
</pre>

These are pretty much there to tell you that the job's running, everything's fine, and the noop time was (in this case) 12 seconds. The <EOM> tells you that the body of the email is empty (end of message).

The noop time varies depending on hg.m.o load, as well as log upload times.

See [https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo#How_to_adjust_email_notifications here] for enabling/disabling these... they can be pretty spammy.

=== Successful emails with messages ===

<pre>
From: vcs2vcs@vcssync1.srv.releng.usw2.mozilla.com
Subject: [vcs2vcs] Successful conversion for beagle (72s)

Summary is non-zero:

info - Successfully pushed mozilla-aurora.
error - Error getting changes for mozilla-inbound; skipping!

02:16:08 ERROR - abort: HTTP Error 500: Internal Server Error
02:16:08 ERROR - Automation Error: hg not responding
02:16:08 ERROR - Return code: 65280
02:16:08 ERROR - Error getting changes for mozilla-inbound; skipping!
</pre>

Whenever the summary of a job is non-zero, or the error log is non-zero, we'll get a message body in the success email. This is usually a successful push notification, but can include the contents of the error log or any other information we put in the summary. (At some future point we may add l10n repo creation on git.m.o in here).

In this case, the ISE from hg.m.o is an intermittent issue. In general, the script should auto-fix by clobbering+recloning, and the conversion will happen later, though it'll be delayed by this action.

=== Failure emails ===

<pre>
From: vcs2vcs@vcssync-dev.build.mozilla.com
Subject: [vcs2vcs] Failed conversion for beagle

Unable to push these repos:
mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
This was a test push that failed; not proceeding any further with mozilla-beta!

Summary is non-zero:

error - Unable to push mozilla-beta. failed.

14:32:30 ERROR - remote: error: denying non-fast-forward refs/heads/GECKO90_2011121217_RELBRANCH (you should pull first)
14:32:30 ERROR - ! [remote rejected] GECKO90_2011121217_RELBRANCH -> GECKO90_2011121217_RELBRANCH (non-fast-forward)
14:32:30 ERROR - error: failed to push some refs to '/opt/vcs2vcs/build/target/beagle/.git'
14:32:30 ERROR - Return code: 256
14:32:30 ERROR - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
14:32:30 ERROR - This was a test push that failed; not proceeding any further with mozilla-beta!
14:32:30 ERROR - Unable to push mozilla-beta. failed.
14:32:30 FATAL - Unable to push these repos:
14:32:30 FATAL - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
14:32:30 FATAL - This was a test push that failed; not proceeding any further with mozilla-beta!
14:32:30 FATAL -
14:32:30 FATAL - Running post_fatal callback...
</pre>

This is a failure email involving [https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo#How_to_deal_with_GECKO90_2011121217_RELBRANCH this issue]. Luckily, we have a fix for that one already.

Failure emails will be sent for every failure currently, so will be very spammy until they're fixed.

== Maintenance ==
=== How to add a repo to gecko-git ===
'''''CAUTION:''' production gecko.git is still handled by legacy vcs-sync - see [http://people.mozilla.org/~hwine/tmp/vcs2vcs/cookbook.html#adding-a-branch-to-gecko-git legacy docs].'' 
This shouldn't be done lightly. Gecko.git is our partner-oriented repo and should be treated with kid gloves.
=== How to add another repo to mirror for b2g OS ===
Mirroring repositories for purpose of b2g os builds is handled by the legacy docs. See [http://people.mozilla.org/~hwine/tmp/vcs2vcs/cookbook.html#mirroring-a-git-repository here].
=== How to add a repo to gecko-dev ===
gecko-dev is our developer focused repository with all gecko branches related to release-trains and inbound-branches only.

You should follow the example [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l96 here]. Annotated:

<pre>
"repo": "https://hg.mozilla.org/releases/mozilla-beta", # where we should pull the repo from.
"revision": "default", # what revision to pull
"repo_name": "mozilla-beta", # the name we'll know this repo by. This should be unique within this job.

"targets": [{
"target_dest": "beagle/.git", # This is a location on disk (test_push is True)
"vcs": "git", # This was used for non-fastforward detection; might be overkill at this point.
"test_push": True,
}, {

"target_dest": "gitmo-beagle", # This is a location specified in the remote targets
# http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l332
}, {

"target_dest": "github-beagle", # Also in remote_targets
}],

"vcs": "hg", # The "vcs" settings are kind of glossed over at the moment,
# but when we get git<->git and git->hg going it'll matter.

"branch_config": {
"branches": {
"default": "beta", # the 'default' hg branch should become a 'beta' git branch.
# This is really needed by the all-in-one repos where there are multiple
# 'default' branches from hg that need to each be their own unique branch.
},
"branch_regexes": [
"^GECKO[0-9_]*RELBRANCH$", # Whitelist the gecko and mobile release branches without having to
"^MOBILE[0-9_]*RELBRANCH$", # list them all by name.
],
},
"tag_config": {
"tag_regexes": [
"^(B2G|RELEASE_BASE)_", # Whitelist the B2G and RELEASE_BASE tags.
# Explicitly avoid the release tags since they currently move in hg-land.
],
},
</pre>

=== How to add a project to project-branch conversion ===
If the project is under hg.m.o/projects, add it [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/project-branches.py#l5 here].

If the project isn't under hg.m.o/projects, add it [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/project-branches.py#l32 here].

=== How to add a project cloning to a custom github account/project ===

* Key setup
** Generate an ssh key for pushing to the github project (no passphrase)
** Add the key to github/org-name/project-name as a deploy key
** Add the key to the vcs2vcs account on the vcssync server
* Add the project to mapper for hg to git hash mapping
** Get a token from https://mozilla-releng.net/tokens
** POST to /project-name on https://mapper.mozilla-releng.net
* Filesystem setup on the vcssync server
** Clone the source hg repo to /opt/vcs2vcs/vcs_sync_build/build/stage_source/
** Clone the hg repo from stage_source to conversion <pre>hg clone /opt/vcs2vcs/vcs_sync_build/build/stage_source/$PROJECT /opt/vcs2vcs/vcs_sync_build/build/conversion/$PROJECT
</pre>
* Add to a vcs-sync configuration file. For example: https://hg.mozilla.org/build/mozharness/rev/57a550a9e307863ab125f1cd1c381de974caf79d
** Set the source location
** Set the destination org/project
* Monitor results and fix/tweak
** Join and watch the error emails in https://groups.google.com/a/mozilla.com/forum/?hl=en&pli=1*!forum/releng-ops-trial
** Check the github project for the commits to appear
** Compare the github commits with the hg commits

=== How to add a locale to l10n conversion ===

L10n conversions are based on [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/l10n.py#l82 this config file].

These point at gecko all-locales and gaia languages_dev.json files. To add a locale, update the appropriate all-locales or languages_dev.json file.

If the b2g versions are changing against gecko trains (as they will every 6 weeks), you need to update [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/l10n.py#l6 this section].

=== How to adjust email notifications ===

The notify_config's are in each config file, like [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/beagle.py#l375 this].

This is a list-of-dictionaries. Each dictionary has a "to" email address.

If <code>failure_only</code> is set to True, no emails with a successful outcome will be sent to this address.

If <code>skip_empty_messages</code> is set to True, no emails without a message body will be sent (these should only be successful noop runs with no warnings).

Allowing both of these to be False will result in a lot of email, approximately one per minute per process. Just setting <code>skip_empty_messages</code> to True will send email per successful push, any sort of warning, or failures.

=== How to force the process to pull/bookmark/convert/push a repo, even if nothing's changed ===

There is a <code>--no-check-incoming</code> commandline option to the vcs_sync.py script.

This can also be set in the config file, as <code>check_incoming</code> set to True.

This can also be set in the config file per-repository, as <code>check_incoming</code> inside each of the specific <code>conversion_repos</code> where we want to skip the incoming check.

To force a push manually:
* cd into project-branches jobs top level directory
* wait for projects.lock to disappear (from cron job)
* touch projects.lock # grab lock to avoid race with automation
** or: <code>touch my_user_name.lock; while ! ln my_user_name.lock projects.lock ; do sleep 10 ; done</code>
** wait for return of command prompt
* get the command line from the run_*.sh file: grep python run_*.sh
* add the '--no-check-incoming' option to the command
* wait for processing to complete
** '''NOTE''': the log output is only to your screen. The usual summary, "<code>projects.log</code>" is only updated via the <code>run_*</code> script. ({{bug|1279008|see also}})
*** The official logs from the last run are in the "<code>./logs/</code>" directory.
*** Older logs are in the "<code>./build/upload/logs/</code>" directory.
* rm 'projects.lock' # let normal processing resume

=== Where to find the keys ===

The keys are the same as the keys on vcs2vcs@gd{0..4}, and has been shared on Google Drive to various RelEngers... except the passphrase has been stripped from them: <code>ssh-keygen -p -f FILE</code>

=== How to move a VCS Sync process ===

* pause the cron job
* move the work dir (or rsync, if across machines).
** You need to move build/conversion or you'll start from scratch, which is definitely a no-no for mozilla-central based repos (see https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo#How_to_start_a_mozilla-central_based_repo_conversion_from_scratch)
** Moving build/stage_source will save a lot of cloning time, especially for processes with a lot of large repos like the project-branches process.
** If the path changes, you need to delete (or not move) build/venv.
** If you're trying to move it cleanly, don't move build/target or build/upload. If you want to preserve the old history, move them.
* If you're on a new machine, you need to make sure it's set up properly. (Should be easier once it's puppetized)
* make sure the cron job script is pointing at the right location
* restart the cron job

=== How to start a mozilla-central based repo conversion from scratch ===

* First, you need a copy of initial3.tar.bz2. This is mozilla-central-with-cvs-history, already converted to git for you, up to Spring, 2013. Without this file, you will need to run the [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/scripts/vcs-sync/initial_beagle.py initial conversion] with [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/gecko-git.py this config file] (or beagle) and let it run for about a week.
** The sha512 sum is <code>0a3243fe5a6c8ffa4e47131e0eb0243e1f5676ea3cacd535d11b424f8f601511130a2e941670950b63c0b00726dafa9bd30bf7c3d040752fce824158021ef014</code>
** This file exists on vcssync{1,2} at /opt/vcs2vcs/initial3.tar.bz2 , on github-sync{2,4} at /home/asasaki/initial3.tar.bz2 , and on Aki's laptop.
* Second, you need a config file for the conversion type. This is probably covered with the beagle, gecko-git, and project-branches config files [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync here].
* Clone the appropriate mozharness repo + branch (see [https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo#Branch here], unless you're working off a separate development branch). This will go into the base_work_dir. For instance, if you want to use <code>/opt/vcs2vcs</code> as the base_work_dir, this will be <code>/opt/vcs2vcs/mozharness</code> .
* Extract the contents of initial3.tar.bz2 into the appropriate location. For instance, if I want to use /opt/vcs2vcs/ as my base_work_dir, and I'm using [http://hg.mozilla.org/build/mozharness/file/7d9425c91051/configs/vcs_sync/gecko-git.py#l21 this config file] (which specifies "gecko-git" as my conversion_dir), I'll need to move some files around.
<pre>cd /opt/vcs2vcs
tar xjvf initial3.tar.bz2 # this creates conversion/beagle
mkdir -p build/conversion
# move the extracted directory from initial3.tar.bz2 to where
# the mozharness script expects the gecko-git conversion dir to be
mv conversion/beagle build/conversion/gecko-git
rmdir conversion
</pre>
* Run the conversion once if you want to make sure.
<pre>
cd /opt/vcs2vcs
# For example,
python mozharness/scripts/vcs-sync/vcs_sync.py -c mozharness/configs/vcs-sync/CONFIGFILE.py # [--no-push] [--no-upload] [--no-notify] if desired
</pre>
** If you want to avoid certain actions, you can specify that you want to skip them. <code>--no-push</code>, <code>--no-upload</code>, and <code>--no-notify</code> might be some options you want to use, depending on the situation. <code>--help</code> and <code>--list-actions</code> should be helpful as well.
* Run in cron like the other boxen.

== Troubleshooting ==
=== How to deal with non-ffwd ===

# Determine what happened. This can generally be found with <pre>hg heads</pre>. If there are multiple heads for a branch, you'll need to follow the below instructions.
# Determine if you want to discard one of the heads, or merge them. (Do you want the changes from both heads, or do you want one head's changes to stick and discard the other one?)
#* If you want to merge the two heads, you can do so with <code>hg merge</code>, <code>hg commit</code>, and <code>hg push</code>
#* If we want to discard one,

<pre>
# we're on good revision X
# we have old [bad] revision Y, which is a different head
hg debugsetparents X Y; hg commit -m "merging X and Y via debugsetparents" # Create a new changeset that has parents X and Y
hg glog | more # see that the history has merged
hg diff -r X # see that the code is the same as revision X, no diffs
hg push
</pre>

If you hit a relbranch issue you may need to follow [https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo#How_to_deal_with_GECKO90_2011121217_RELBRANCH these instructions] instead.

=== How to deal with project branch reset ===

To some degree, the integration-gecko-projects repo should be self-healing. If the incoming changesets to a repo are incompatible (so an <code>hg pull</code> doesn't work), the script should blow away the source repo and reclone. The integration-gecko-projects target will shortly have <code>force_push</code> set to True, so it should be pushing with <code>git push -f</code>.

However, project branch resets may become an issue; time will tell.

We can probably:

* pause the project branches job (vcs2vcs@vcssync2 cron),
* delete the branch on github https://github.com/mozilla/integration-gecko-projects/branches
* delete the branch on disk (vcs2vcs@vcssync2:/opt/vcs2vcs/build/stage_source/BRANCH)
* restart the job

and it'll clone a new copy of the branch, convert, and push.

If there are still issues, we can debug further to try to get it to work. A final nuke-the-site-from-orbit solution is documented [https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo#How_to_deal_with_completely_resetting_gecko-projects here], but may take the repo down for a day or so.

=== How to deal with completely resetting gecko-projects ===

# Make the decision that this is the solution to the problem.
# Make sure people are aware this is going to happen.
# Stop the conversion of gecko-projects. This is currently on vcs2vcs@vcssync2, and is run via cron.
# Save a copy of https://raw.github.com/mozilla/integration-gecko-projects/master/README.md
# Delete the repo! This option is available in the "Danger Zone" area of https://github.com/mozilla/integration-gecko-projects/settings
# Recreate the repo. There's a "Create a new repo" icon in the top right. This will need the same name, should be public, and you want to initialize the repo with a README. (gecko-projects is the only repo where we want the README: essentially, we want an empty master branch because github doesn't allow you to delete the first populated branch in a repo.)
# Edit the README with the contents you downloaded in step 4.
# Start the conversion back up. You may only need to delete the build/target and build/stage_source/BAD_REPO directories. Or you may need to blow away the whole thing from orbit to be sure, in which case you should follow the docs for [https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo#How_to_start_a_mozilla-central_based_repo_conversion_from_scratch how to start a mozilla-central based repo conversion from scratch].
#* Note, just cloning the hg repos for the projects repo can take many many hours, so this will take the gecko-projects repo down for a long time.

=== How to deal with GECKO90_2011121217_RELBRANCH ===

We were getting email with the following failure:

<pre>
11:14:01 ERROR - remote: error: denying non-fast-forward refs/heads/GECKO90_2011121217_RELBRANCH (you should pull first)
11:14:01 ERROR - ! [remote rejected] GECKO90_2011121217_RELBRANCH -> GECKO90_2011121217_RELBRANCH (non-fast-forward)
11:14:01 ERROR - error: failed to push some refs to '/opt/vcs2vcs/build/target/beagle/.git'
11:14:02 ERROR - Return code: 256
11:14:02 ERROR - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git!
11:14:02 ERROR - This was a test push that failed; not proceeding any further with mozilla-beta!
</pre>

To debug:

<pre>
# vcs2vcs@vcssync1
cd /opt/vcs2vcs/build/target/beagle
git show GECKO90_2011121217_RELBRANCH # this gave 1003ec79451969335008880ad82e305d93b89642
cd /opt/vcs2vcs/build/conversion/beagle
git show GECKO90_2011121217_RELBRANCH # this gave fac7279c040d643fb4c35105fa85b9335ba2c2f9
git merge-base fac7279c040d643fb4c35105fa85b9335ba2c2f9 1003ec79451969335008880ad82e305d93b89642
</pre>

The merge-base returned <code>fac7279c040d643fb4c35105fa85b9335ba2c2f9</code>. That means that <code>fac727</code> is a parent to <code>1003ec</code>. No sha divergence, only stale history.

I noticed that http://hg.mozilla.org/releases/mozilla-beta/rev/GECKO90_2011121217_RELBRANCH pointed at <code>a9221b332d8a</code> (dec 15) and http://hg.mozilla.org/releases/mozilla-release/rev/GECKO90_2011121217_RELBRANCH pointed at <code>4e309e63c279</code> (jan 3); Calendar must have done a release off the relbranch off mozilla-release. Since the branch name is shared across mozilla-beta and mozilla-release, converting the branch from mozilla-beta effectively does a non-fastforward push and is rejected.

To fix, I exported the two revisions from mozilla-release to mozilla-beta:

<pre>
cd mozilla-release
hg export -r 79350 > ../cal2
hg export -r 79349 > ../cal1
# Doublecheck to make sure those look good
cd ../mozilla-beta
hg pull -u
hg up -r GECKO90_2011121217_RELBRANCH
hg import ../cal1
hg ident # sha matched m-r
hg import ../cal2
hg ident # sha matched m-r
hg out
hg push
</pre>

Then update https://wiki.mozilla.org/ReleaseEngineering/VCSSync/History#Relbranch_issues .

==== Hints ====

<pre>
hg glog 0..REVISION
</pre>

QA/Ondemand Update Tests

2018-06-30T02:01:45Z

Garbas: new treestatus url

= Summary =
This page describes in detail the steps you need to take to run Ondemand Update tests for various Firefox builds. The setup below assumes you have access to [http://mm-ci-production.qa.scl3.mozilla.com:8080 Jenkins], either from within the internal network or via [https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=30769829 VPN].

Complementary to update tests on the '''"release"''' channel, you may need to also run [https://wiki.mozilla.org/QA/Throttling_Tests Throttling Tests]. Visit the link for instructions on how to run these.

= Setting up the config file =
# <big>Configuration data should be readily available prior to builds coming out</big>
# <big>'''Steps to find/update configuration data, and set up the config file:'''</big>
## Search Mozilla wiki for the Test Plan page for the version you are testing - e.g. if you need to test update for 52.0.2, then go to the Firefox 52 Test Plan page - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan
## Look for the section dedicated to the exact version that you need to test
### E.g. if you need to test update for 52.0.2, it should be under '''Milestones -> Release Candidates -> 52.0.2''' (see https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#52.0.2)
### E.g. if you need to test update for Beta 6, it should be under '''Milestone -> Betas -> Beta 6''' (see https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#Beta_6)
## <big>'''If there is a section dedicated to the exact version that you need to test'''</big>, do the following:
### Under '''"Testing"->"Automated"''' there should be a link that says '''“configs”''' - click on it
### The Config page that opens (e.g. https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b6), will display information on update restrictions and watersheds, and below you should see the contents needed in the config files to run the tests - usually there will be two sections: one for the Win 64 builds config file (the short one on the bottom), and a larger one for all the other platforms
### Copy each section of pre-formatted text (starting with [testrun]) in a '''“config.ini”''' file on your computer - you can name the files any way you want but it's recommended you give it a name that will help you keep track of it (e.g. '''"52b6_build1_beta-cdntest_win64.ini"''' for a config file used to test 52 Beta 6 build 1, with a 64 bit Windows build, on the beta-cdntest channel)
### Make sure that the “channel” value is the one that you want to test - notice how the order to test the channels is specified in each corresponding test plan:
#### Beta - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#Beta_6 -> Testing -> Automated = order is: “beta-cdntest”, then “beta”
#### Release Candidate - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#52.0_RC2 -> Testing -> Automated = order is: “beta-cdntest”, then "beta", then “release-cdntest”, then “release”
#### Dot Release - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#52.0.2 -> Testing -> Automated = order is: “release-localtest”, then “release-cdntest”, then “release”
#### ESR - https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan#Firefox_52.1.0_ESR -> Checklist = order is: “esr-localtest”, then “esr-cdntest”, then “esr”
### Save the files locally - they are now ready to be used to run the tests
## <big>'''If there is NO section dedicated to the exact version that you need to test'''</big>, do the following:
### Look for the section dedicated to the previous build: e.g. if you are testing 52 Beta 6, look for the 52 Beta 5 section in the Test Plan (https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#Beta_5)
### Under the '''"Testing"->"Automated"''' section there should be a link that says '''“configs”''' - click on it
### The Config page that opens (e.g. https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b5), will display information on update restrictions and watersheds, and below you should see the contents needed in the config files to run the tests - usually there will be two sections: one for the Win 64 builds config file (the short one on the bottom), and a larger one for all the other platforms
### Copy each section of pre-formatted text (starting with [testrun]) in a '''“config.ini”''' file on your computer - you can name the files any way you want but it's recommended you give it a name that will help you keep track of it (e.g. '''"52b6_build1_beta-cdntest_win64.ini"''' for a config file used to test 52 Beta 6 build 1, with a 64 bit Windows build, on the beta-cdntest channel)
### Open each config file for editing
### Modify the “target-version” value to the version corresponding to the build you are testing, making sure the correct format is used (see examples below):
#### Beta - 52.0b6#1 - means 52 Beta 6, build 1 (found here: https://archive.mozilla.org/pub/firefox/candidates/52.0b6-candidates/)
#### Release/Dot release - 52.0.1#2 - means 52.0.1, build 2 (found here: https://archive.mozilla.org/pub/firefox/candidates/52.0.1-candidates/)
#### ESR - 52.1.0esr#3 - means ESR 52.1.0, build 3 (found here: https://archive.mozilla.org/pub/firefox/candidates/52.1.0esr-candidates/)
### <big>Make sure to always use the latest available build, as that's the one Firefox should be updating to</big>
### Modify the “channel” value to the channel that you want to test (notice how the order to test the channels is specified in each corresponding test plan:
#### Beta - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#Beta_6 -> Testing -> Automated = order is: “beta-cdntest”, then “beta”
#### Release Candidate - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#52.0_RC2 -> Testing -> Automated = order is: “beta-cdntest”, then "beta", then “release-cdntest”, then “release”
#### Dot Release - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#52.0.2 -> Testing -> Automated = order is: “release-localtest”, then “release-cdntest”, then “release”
#### ESR - https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan#Firefox_52.1.0_ESR -> Checklist = order is: “esr-localtest”, then “esr-cdntest”, then “esr”
### Modify the config files:
#### At a minimum make sure that the previous version is updated (e.g. you'll notice that on each platform we test the previous version with en-US and/or another locale - in https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b5 you'll see that the previous version - 52.0b4 - is the first one to be covered for each platform - if you are updating the file to be used for 52 Beta 6, then, at a minimum, you need to change 52.0b4 everywhere to 52.0b5)
#### For a complete update of the config data, we aim at covering for each version:
##### all builds of version V, with one locale each - e.g. for https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b5, under [windows xp 32bit], there is one locale for each of 52.0b4, 52.0b3, 52.0b2, 52.0b1 (with en-US for 52.0.b4) - if we cannot cover all builds of version V under a platform, then we aim at splitting them as evenly as possible among various similar platforms (Windows, Mac, Linux)
##### a minimum of one locale for V-1, V-2, V-3 versions - e.g. for https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b5, we cover a random version of each of 51b, 50b, and 49b for Windows, but because we afford it, we cover multiple versions for 51b on Linux
#### <big>NOTE: The configuration data is designed to cover ALL existing locales exactly once on any platform (except for en-US), distribute the tests as evenly as possible between Windows, Mac and Linux, cover versions only up to V-3 (except for ESR), and focus more on the more recent previous versions, and less on the older previous versions</big>
### Once you've modified the data in your config files, save the files locally (they're now ready to use to run the tests), then also create a new Wiki page for the new config data, using the wiki page from the previous config (e.g. for 52 Beta 6, go to https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b6 to create it, and add the info from https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b5, with the modified config file content)

== Notes ==
# Example of test plans for Firefox and ESR (just replace the version number in the URL to get to the wiki you need):
## https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan
## https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan

= Running the tests with the config file =
# Connect to MozillaVPN
# Go to http://mm-ci-production.qa.scl3.mozilla.com:8080/
# Login with valid credentials
# Click on the [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/trigger-ondemand/ trigger-ondemand] link
# Click on [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/trigger-ondemand/build?delay=0sec Build with Parameters] on the left side of the page
# Click on the '''“Choose File”''' button and select your config file for the run
# Click the '''“Build”''' button -> The job will start to load in the panel on the left
# As you will usually use two config files, repeat the steps above to start the tests with the second file as well (you can do it immediately, the system will know to start the jobs from the second file, once the ones in the first file are all started)
# Monitor progress in: http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/
# Monitor test results in Treeherder:
## <big>'''If the Test Plan section already exists for your version'''</big> - Click on the '''"report"''' link corresponding to the channel that you are testing
## <big>'''If the Test Plan section doesn't/didn't already exist for your version'''</big> - Follow the next steps:
### Open the report for the previous version (see examples below):
#### Beta/Release/Dot Release - https://wiki.mozilla.org/Releases/Firefox_46/Test_Plan#Beta_4 '''-> Testing -> Automated -> “report” link =>''' https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=2f6f69a19150&filter-tier=3&filter-searchStr=Fxup-beta-cdntest(
#### ESR - https://wiki.mozilla.org/Releases/Firefox_45_ESR/Test_Plan#Firefox_45.0.1_ESR '''-> Checklist -> Update automation on esr-localtest -> “report” link =>''' https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr45&revision=32eb8c423d67&filter-tier=3&filter-searchStr=Fxup-esr-localtest(
### In the URL of the report for the previous version, just replace the revision string with the one for the current version (that you are testing now) - to find this value do the following:
#### Go to the location of your build, and select any platform and any language - e.g. for 44.0.2 you could go to: http://archive.mozilla.org/pub/firefox/candidates/44.0.2-candidates/build3/win32/en-US/
#### Click on the txt file at this location to view it - e.g. in this case: http://archive.mozilla.org/pub/firefox/candidates/44.0.2-candidates/build3/win32/en-US/firefox-44.0.2.txt
#### Navigate to the URL in the txt file - e.g. in this case: https://hg.mozilla.org/releases/mozilla-release/rev/60e96806ff1c9fb51f3df31ccd1db33f5f320e75
#### Note the text on the top page - e.g. in this case: '''“Mercurial > releases > mozilla-release / changeset / 60e96806ff1c”'''
#### The last string is the value to use in the URl of the report for '''“revision”''' - e.g. in this case the value will be '''“60e96806ff1c”'''
### Also make sure that the '''“filter-searchStr”''' value in the URL corresponds to the channel you are testing
### You should now be able to see the results in real time, similar to the pic below:
[[File:Tree1.png]]

= Analyzing the results =

== Green = PASS ==
# No action

== Orange = TEST FAILURE ==
# If there’s a star displayed next to the text then it’s already starred
# If there is no star then you need to classify the failure:
## Click on the string -> The '''“Pinboard”''' will open
## See more details in section '''4. Classifying failures''' below
## Rebuild:
### In the pinboard (picture below) click on the '''“Job details”''' tab
### Click on the link next to '''“Inspect Jenkins Build (VPN required):”'''
### This will bring you to the Jenkins page for that test - e.g. http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/23245/
### On this Jenkins page click the '''“Rebuild”''' link on the left of the page
### Now click the '''“Rebuild”''' button on the bottom of the page
### Several seconds after the test was rebuilt you should see a new entry on the Treeherder [https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr45&revision=32eb8c423d67&filter-tier=3&filter-searchStr=Fxup-esr-localtest report] page with the same name as the one that failed (e.g. “lij-1”) but in gray (meaning it’s running) - if the rebuild passes you will then see it in green and are all done

== Red = BUSTAGE ==
# The exact same actions apply as for Orange

== Pink = CANCELLED ==
# Possible causes:
## Manual cancellation - [https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=078baf501b55&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release&selectedJob=190668 Example]
### HOW - you can manually cancel jobs in Jenkins by clicking the '''"x"''' icon next to a job in http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/, or clicking on the link for a job and then clicking the '''"x"''' icon on top right of the page
### ACTIONS - just classify all pink jobs as '''"expected fail"''' with a comment as to why you've cancelled them, and then either rebuild all pink jobs (if just a few) or re-trigger the whole ondemand job using the config.ini file
## Network issues - [https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=2991f214d4f4&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta&selectedJob=1074174 Example]
### HOW - network issues may cause the test machines to not be able to setup the environment, so tests won't even be start
### ACTIONS - just classify all pink jobs as '''"infra"''' with a comment stating that they were caused by an internal network issue, and then either rebuild all pink jobs (if just a few) or re-trigger the whole ondemand job using the config.ini file

== '''Classifying failures''' ==
# '''Issue is in the "Failure Summary"/"Failure Classification" list'''
## Look for all errors for that test - in the picture below there is only one error: '''“TEST-UNEXPECTED-FAIL | test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: The staged update to buildid 20170504103226 has not been applied”''' - which shows on the first line under '''"Failure Classification"''' or '''"Failure Summary'''" if you select it
## Look underneath the error for the bug number that best corresponds to your error - in the picture below that would be: '''“1355818 - [tier-3] Intermittent test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: The staged update to buildid 20170412030252 has not been applied”''' - which is the very first line under the error (most of the times, the first line will also be the bug that you're looking for)
## Click the pin icon in front of the bug number ('''Note: you need to be logged in''')
## Click save - you should get a notification on the top left of the page that the error has been classified, and the job will now display a '''star''' on the top right [[File:ondemand_fail_1.png]]
## '''Note: this also works if you have multiple jobs selected with Ctrl+click (if all have failed with the same error)'''
# '''Issue is NOT in the Failure Summary'''
## Search this wiki or bugzilla for the error (search Product=Testing, Component=Firefox UI Tests - [https://bugzilla.mozilla.org/buglist.cgi?list_id=13011155&short_desc=Unable%20to%20locate%20window%3A&query_format=advanced&short_desc_type=allwordssubstr&component=Firefox%20UI%20Tests&product=Testing example of query])
### If found:
#### Ctrl+Click on the failed job OR click on '''“Open pinboard ^”''' (additionally you can Ctrl+Click on all jobs that failed with the same error)
#### Under “classification” select '''“intermittent”'''
#### Add the bug number in the '''"+ click to add a related bug"''' field on the left of the "classification" field, and press ENTER - if this doesn't work, add it in the comment field
#### Click '''“save”''' (if '''“save”''' button is disabled, CTRL+Click the same test again - you should see it added on the left, and '''“save”''' is enabled) - classify the issue as "intermittent" with the bug number
### If not found file a new bug:
#### Via Treeherder (the quick way to do it)
##### In the URL bar add '''"&bugfiler"''' parameter to the URL and press '''"Enter"'''
##### Click on the job that failed
##### Select the '''Failure Summary''' tab (the second one, in between '''Job Details''' and '''Failure Classification''')
##### Click the small '''bug icon''' in front of the failure that you want to file (the line that says: "TEST-UNEXPECTED-ERROR | ...") -> This should open the '''Intermittent Bug Filer''' pop-up window
##### In the Product entry field add '''Testing :: Firefox UI Tests''' and click '''"Find Product"''' -> Make sure the component is found, and the radio box in front of it appears as selected
##### Double-check the summary to make sure it is the desired one
##### Leave all three check-boxes selected:
###### '''Include Parsed Log Link'''
###### '''Include Full Log Link'''
###### '''This is an intermittent failure''' (unless you are hitting a permanent failure, in which case you need to uncheck it)
##### Add any additional comments, as you consider necessary (e.g. where it was found, how many times it was seen, etc.)
##### Click '''"Submit Bug"'''
##### The failure should now appear as classified, and within several seconds you should also see the link to the newly filed bug
#### Via Bugzilla
##### Choose to create a [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Firefox%20UI%20Tests new bug] for Testing::Firefox UI
##### Set '''"Version"=<version_that_you've_run_the_tests_for'''
##### Set appropriate '''"Platform"''' values (or leave '''Unspecified''')
##### In the '''"Summary"''' field enter the error encountered (the error that shows in [https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=5bbf2e7c2fc6&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta(&selectedJob=1025489 Treeherder] under '''"Failure summary"''' - e.g. '''"TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update, test_page_info_window.py TestPageInfoWindow.test_close_window | MarionetteException: win.document.documentElement is null"''')
##### In the '''"Description"''' field add the following information:
###### Where was the error seen (with link to the [https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=5bbf2e7c2fc6&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta(&selectedJob=1025489 Treeherder] job) - for [https://bugzilla.mozilla.org/show_bug.cgi?id=1268087 example]: "Seen once while running ondemand update tests for 47.0b1 (https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=5bbf2e7c2fc6&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta(&selectedJob=1025489)."
###### Traceback log - to get this do the following:
####### For the [https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=5bbf2e7c2fc6&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta(&selectedJob=1025489 Treeherder job] that failed, click on the '''"Job details"''' tab
####### Click the '''"report.html"''' link, to open the report in a new tab
####### Copy the contents displayed under the '''"Results"''' section (e.g. "Traceback (most recent call last): ..."
####### This will be your Traceback log to paste in the bug description - see the example [https://bugzilla.mozilla.org/show_bug.cgi?id=1268087 here]
# '''No logs available in the "Failure Summary"'''
## This usually indicates a problem with the Mozilla infrastructure
## Actions to take:
### If [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/ Jenkins] indicates only a few failure
#### Rebuild and see if it passes (if it doesn't then it may be a real problem)
#### Once rebuilt and passed classify the issue as "infra" (comment is not mandatory)
### If [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/ Jenkins] indicates a lot of failures
#### Reverify the config file
#### If config file is OK, contact :whimboo on #automation - he should know/investigate such issues
#### Once notified that the problem was remedied:
##### Classify all failures in treeherder as "infra" (comment is not mandatory)
##### Start the entire run once again (could rebuild individual jobs that failed, but that takes a much longer time)
##### Examples: [https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=2991f214d4f4&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta&selectedJob=1074194 47 Beta 4]

== '''Common issues''' ==
# Logs not fully parsed, please wait
## This is an infrastructure issue - no need to wait as the logs won't be parsed
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Open pinboard (or Ctrl+Click on the job), in the '''classification''' drop-down select '''infra''', then click '''“save”'''
# TEST-UNEXPECTED-FAIL | test_direct_update.py TestDirectUpdate.test_update | AssertionError: Available update has been found
## This usually indicates that Firefox has '''NOT''' found any updates available
## Actions:
### Check the results for patterns showing that only some versions get this, and double check for existing watersheds (there may be an existing watershed or rule in place that does not allow update of specific versions - #releng can check for this)
#### If a watershed is in place then all tests for the same version should fail in the exact same way - if you only have a few failures then this is likely [https://bugzil.la/1316567 bug 1316567] - Actions to take in case of bug:
##### [[#Rebuilding|Re-build]] - If tests pass on rebuild, then this confirms you are dealing with [https://bugzil.la/1316567 bug 1316567]
##### If/after the test passes on re-build, '''Classify the error''':
###### Click the pin icon in front of [https://bugzil.la/1316567 bug 1316567] + Click '''“save”''' ***OR***
###### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
#### If ALL tests fail, and most failures are with this error message and the one below, then it means that the server is not properly configured - Actions to take:
##### Notify people on #release-drivers or #releng about this, and ask them to check the server settings
#####'''Classify the error''' as "expected fail" with a comment explaining that this is due to the server not being set properly:
###### Ctrl+Click all failures, in the classification drop-down select "expected fail", in the field below add a comment explaining that this is due to the server not being set properly, press ENTER, and then Click '''“save”'''
##### After the rules on the update the server are fixed, re-run all jobs (trigger aggain with the same config files)
# TEST-UNEXPECTED-FAIL | test_direct_update.py TestDirectUpdate.test_update | AssertionError: u'20160427030215' != u'20160425205003'
## This usually indicates that Firefox was updated to a different version than the expected one
### The first string is the '''BuildID''' of the '''ACTUAL''' version that Firefox got updated to
### The second string is the '''BuildID''' of the '''EXPECTED''' version that Firefox was supposed to get updated to
## Actions:
### Check the results for patterns showing that only some versions get this, and double check for existing watersheds (there may be an existing watershed or rule in place that does not allow update directly to your expected version, but instead force update to an intermediate version - #releng can check for this)
#### If a watershed is in place then all tests for the same version should fail in the exact same way - if you only have 1-2 failures then this is likely [https://bugzil.la/1285340 bug 1285340] - Actions to take in case of bug:
##### [[#Rebuilding|Re-build]]
##### If/after the test passes on re-build, '''Classify the error''':
###### Click the pin icon in front of [https://bugzil.la/1285340 bug 1285340] + Click '''“save”''' ***OR***
###### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
#### If ALL tests fail, and most failures are with this error message and the one above, then it means that the server is not properly configured - Actions to take:
##### Notify people on #release-drivers or #releng about this, and ask them to check the server settings
#####'''Classify the error''' as "expected fail" with a comment explaining that this is due to the server not being set properly:
###### Ctrl+Click all failures, in the classification drop-down select "expected fail", in the field below add a comment explaining that this is due to the server not being set properly, press ENTER, and then Click '''“save”'''
##### After the rules on the update the server are fixed, re-run all jobs (trigger aggain with the same config files)
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: 'beta-cdntest' != u'nightly'
## This means that the machine running the test is stuck on Nightly or Aurora - ISSUE USED TO BE WIDESPREAD BUT SHOULD NOW BE FIXED
## To confirm click the '''“Job details”''' tab then the '''“report.html”''' link - if you see screenshots showing an open Nightly/Aurora window, then it’s confirmed that the machine is stuck on Nightly or Aurora - If you don't see a Nightly or Aurora build, but instead get the expected build, then just ignore the error and classify the other errors (if multiple errors), or classify as "infra" if it's the only error
## Actions:
### Make sure your VPN is connected
### Open a VNC Viewer app on Windows (e.g. VNC Viewer, Tight VNC Viewer)
### In the “VNC Server” or “Remote Host” field enter the name of the machine where the failure occurred (you can find it on the left of the expanded pinboard - e.g. '''"Machine: mm-win-7-64-2.qa.scl3.mozilla.com"''')
### Enter the password to connect to the machine (if you do not know the password you can ask in #automation)
### You should now be connected to that machine and see an open Nightly/Aurora window
### Note down: name of the machine, version of Firefox that was open (e.g. from "about:support" -> "49.0a1, BuildID=20160512205003"), and page that was open in the active window (e.g. "about:home")
#### '''Do this for all other machines in this case - you should now have a list of affected machines, firefox versions, and open pages'''
### Close the Firefox window and exit from the VNC Viewer app - '''NOTE: Make sure you close ONLY the FIREFOX windows'''
#### '''In case of multiple machines in this situation do the following'''
##### '''Close ONLY the Firefox window on all of them'''
##### '''Take at least one machine offline (see "[[#Taking_node_offline|Taking node offline]]" at the end of this wiki page'''
##### '''Send email to mozmill-ci@mozilla.org, stating that there are multiple machines stuck on Nightly/Aurora, and list the details picked in the previous steps (machine name, firefox version that was stuck, with BuildID, page that was open)'''
##### '''Proceed then to classify issues and rebuild'''
### [[#Rebuilding|Re-build]]
### Classify the issue:
#### Click on the failed test
#### Click on '''“Open pinboard ^”'''
#### If '''“save”''' button is disabled, CTRL+Click the same test again (you should see it added on the left, and '''“save”''' is enabled)
#### Under “classification” select '''“infra”'''
#### Add a comment like: "Machines stuck on Nightly/Aurora"
#### Click '''“save”'''
#### Note: you can classify multiple issues of the same kind by using CTRL+Click for multiple selection
## '''NOTE: These errors can also show in case of expected failures, caused by:'''
### '''Bad config.ini - For example [https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=076bf6a0ac85&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta&selectedJob=1012101 here] a config.ini file was used where the target build was 46.0b11, but for some locales the source for the update was also 46.0b11 - obviously a build will not update to itself'''
### '''Watershed rules - are rules set so that some older versions will NOT update beyond some version, for example:'''
#### '''44.0b1 is an update "watershed" for Windows, no updates from before go to builds beyond it. This means that if you set up a test to update 43.0b7 for example to the latest Beta version (e.g. 47.0b4), then the update test will fail, as 43.0b7 would update to 44.0b1'''
#### '''44.0rc3 is an update "watershed" for Linux, 44.0b4 and earlier only get updates to that version.'''
### '''Blocked updates - temporary/partially disabled updates - For example [https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=0b8492c110be&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release&selectedJob=191052 here] a lot of tests failed because when releasing 46.0.1 to the release channel, Release Engineering decided that updates from 45.0.2 and lower should be temporarily disabled ([https://bugzil.la/1269848 bug 1269848]), which naturally caused all update tests to fail, except for the ones that tested update from 46.0'''
#### '''Basically in this case, the failures confirmed the fix for [https://bugzil.la/1269848 bug 1269848]'''
#### '''These failures were NOT rebuilt since they were never going to pass (until the change from [https://bugzil.la/1269848 bug 1269848] would have been reverted)'''
#### '''All these failures were marked as "expected fail"'''
# TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | AttributeError: 'dict' object has no attribute 'type'
## This is [https://bugzil.la/1247596 bug 1247596]
## This is '''NOT''' necessarily a problem with Firefox
## This usually indicates there are some watersheds in play - to find out more information:
### Click on a job and then on '''"Job details"'''
### Click on the '''"log_info.log"''' link
### Go through the log trying to figure out:
#### What might have caused a failure
#### What's the actual build that Firefox got updated to (look for '''"build_post"''' in the logs)
## Re-building jobs is unlikely to help in this case, but you can try to do it at least once
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | NoSuchWindowException: NoSuchWindowException: No window found for '<function <lambda> at 0x03226830>'
## This is [https://bugzil.la/1274943 bug 1274943] - should show up in the '''"Failure summary"'''
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1274943 bug 1274943] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | error: [Errno 10054] An existing connection was forcibly closed by the remote host
## This is [https://bugzil.la/1314627 bug 1314627] - should show up in the '''"Failure summary"'''
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1314627 bug 1314627] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | IOError: process died with returncode 0
## This is [https://bugzil.la/1289343 bug 1289343] - should show up in the '''"Failure summary"'''
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1289343 bug 1289343] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-FAIL | test_direct_update.py TestDirectUpdate.test_update | AssertionError: <firefox_puppeteer.ui.about_window.deck.DownloadFailedPanel object at 0x02830CD0> == <firefox_puppeteer.ui.about_window.deck.DownloadFailedPanel object at 0x02830650>
## This is [https://bugzil.la/1249466 bug 1249466], which is in fact a duplicate of [https://bugzil.la/1249467 bug 1249467] - should show up in the '''"Failure summary"'''
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1249467 bug 1249467] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-FAIL | test_direct_update.py TestDirectUpdate.test_update | AssertionError: <firefox_puppeteer.ui.about_window.deck.DownloadingPanel object at 0x032BD9F0> != <firefox_puppeteer.ui.about_window.deck.CheckForUpdatesPanel object at 0x032BD770>
## This is [https://bugzil.la/1233679 bug 1233679]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1233679 bug 1233679] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | IOError: Timed out waiting for port! '''*** OR ***''' TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | IOError: Timed out waiting for port!
## This is [https://bugzil.la/1202375 bug 1202375] - should show up in the '''"Failure summary"'''
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1202375 bug 1202375] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | MarionetteException: MarionetteException: win.document.documentElement is null '''*** OR ***''' TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | MarionetteException: MarionetteException: win.document.documentElement is null
## This is [https://bugzil.la/1268087 bug 1268087]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1268087 bug 1268087] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | TimeoutException: TimeoutException: Timed out after 5.1 seconds '''*** OR ***''' TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | TimeoutException: TimeoutException: Timed out after 5.1 seconds with message: Cannot get window type for chrome window handle "1"
## This is [https://bugzil.la/1256425 bug 1256425] - should show up in the '''"Failure summary"'''
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1256425 bug 1256425] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | JavascriptException: JavascriptException: TypeError: ums.activeUpdate is null
## This is [https://bugzil.la/1260383 bug 1260383]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1260383 bug 1260383] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | UnknownException: UnknownException: Error loading page '''*** OR ***''' TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | UnknownException: UnknownException: Error loading page
## This is [https://bugzil.la/1262122 bug 1262122]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1262122 bug 1262122] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | TimeoutException: TimeoutException: Timed out after 360.2 seconds with message: Download has been completed.
## This is [https://bugzil.la/1238002 bug 1238002] - should show up in the '''"Failure summary"'''
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1238002 bug 1238002] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | NoSuchWindowException: NoSuchWindowException: Unable to locate window: [1|3]
## This is [https://bugzil.la/1207042 bug 1207042]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1207042 bug 1207042] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | TimeoutException: TimeoutException: Timed out after 30.2 seconds with message: Check for updates has been finished.
## This is [https://bugzil.la/1288731 bug 1288731]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1288731 bug 1288731] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Connection timed out after 60s)
## This is [https://bugzil.la/1311987 bug 1311987]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1311987 bug 1311987] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Timed out waiting for port 2828!)
## This is [https://bugzil.la/1303834 bug 1303834]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1303834 bug 1303834] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | UnknownWindowError: UnknownWindowError: Window with handle "1" does not exist
## This is [https://bugzil.la/1323174 bug 1323174]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1323174 bug 1323174] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | IOError: Process has been closed (Exit code: 0) (Reason: [Errno 61] Connection refused)
## This is [https://bugzil.la/1324059 bug 1324059]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1324059 bug 1324059] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-FAIL | test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Additional update found due to watershed release 50.1.0
## This is [https://bugzil.la/1316564 bug 1316564]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1316564 bug 1316564] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | MarionetteException: Please start a session
## This is [https://bugzil.la/1322109 bug 1322109]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1322109 bug 1322109] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | NoSuchWindowException: No window found for '1'
## This is [https://bugzil.la/1377092 bug 1377092]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1377092 bug 1377092] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Timed out waiting for connection on localhost:2828!)
## This is [https://bugzil.la/1362293 bug 1362293]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1362293 bug 1362293] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-FAIL | test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: The staged update to buildid 20170504103226 has not been applied
## This is [https://bugzil.la/1355818 bug 1355818]
## Actions:
### On localtest or cdntest channels ONLY - [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1355818 bug 1355818] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
## NOTE: This error causes repeated failures on WINDOWS, that pass after multiple re-runs, and seriously affect the official channels (beta, release, esr), to the point where you might even end up re-building the same job up to 100 times until it passes - this error was not reported in the real world - current agreement is that WE DO NOT RE-BUILD JOBS THAT FAILED WITH THIS ERROR ON OFFICIAL CHANNELS (beta, release, esr) - we only re-build on localtest or cdntest channels
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Timed out waiting for connection on localhost:2828!)
## This is [https://bugzil.la/1303834 bug 1303834]
## Actions:
### On localtest or cdntest channels ONLY - [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1303834 bug 1303834] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
## NOTE: This error causes repeated failures on LINUX, that pass after multiple re-runs, and seriously affect the official channels (beta, release, esr), to the point where you might even end up re-building the same job up to 100 times until it passes - this error was not reported in the real world - current agreement is that WE DO NOT RE-BUILD JOBS THAT FAILED WITH THIS ERROR ON OFFICIAL CHANNELS (beta, release, esr) - we only re-build on localtest or cdntest channels
# TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | TimeoutException: Timed out after 5.1 seconds with message: No new chrome window has been opened.
## This is [https://bugzil.la/1372982 bug 1372982]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1372982 bug 1372982] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''
# TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | JavascriptException: ReferenceError: log is not defined
## This is [https://bugzil.la/1383032 bug 1383032]
## Actions:
### [[#Rebuilding|Re-build]]
### If/after the test passes on re-build, '''Classify the error''':
#### Click the pin icon in front of [https://bugzil.la/1383032 bug 1383032] + Click '''“save”''' ***OR***
#### Open pinboard (or Ctrl+Click), manually add the bug number (in the '''"+ click to add a related bug"''' field on the left of the "classification" field), press ENTER + Click '''“save”'''

= Reporting =
See below a series of emails sent when signing off various builds, on various channels. Note that the procedure can vary on the same channel at different phases.

== BETA - sign off on beta-cdntest ==
=== BETA 1 ===
'''QA signs off, but Release Management asks for the push to "beta" channel (as the release notes need to be ready)'''
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: release-drivers [mailto:release-drivers-bounces@mozilla.org] On Behalf Of Robert Kaiser
Sent: Thursday, January 28, 2016 7:21 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] QE sign-off for Firefox 45.0b1

QE has finished testing Firefox 45.0b1 build1 candidates and updates on the beta-cdntest channel.

Automated testing looks OK, manual testing looks good as well, the larger issues found are either not regressions or in default-off code.

I think we're good to go live with 45.0b1. Pushing the build to beta (and DevEdition to aurora) is gated on release management's call.
</pre>
=== BETA 2-last ===
'''QA signs off, and asks for the push to "beta" channel (no need to wait on Release Management)'''
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: release-drivers [mailto:release-drivers-bounces@mozilla.org] On Behalf Of Florin Mezei
Sent: Tuesday, May 10, 2016 5:28 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] Please push firefox and devedition 47.0b4 to beta and aurora, respectively

Drivers,

QE has finished testing Firefox 47.0b4 (build1), and updates on the beta-cdntest channel. Regression and manual testing efforts found one new issue (https://bugzil.la/1271642), which is Linux specific and not a recent regression, so we consider the build ready for release to Beta.

Please make the build live on the beta channel - and the corresponding DevEdition build live on the aurora channel.

For more information on testing of 47.0b4, see our test plan at
https://wiki.mozilla.org/Releases/Firefox_47/Test_Plan#Beta_4
</pre>

== BETA - sign off on beta ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: release-drivers [mailto:release-drivers-bounces@mozilla.org] On Behalf Of Michelle Funches
Sent: Tuesday, May 10, 2016 7:42 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] Firefox 47.0b4 - Updates on Beta Channel signed off by QA

Drivers,

Update tests on the beta channel look good. Results are here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=2991f214d4f4&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta(
</pre>

== RC - sign off on beta-cdntest ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: Florin Mezei [mailto:florin.mezei@softvisioninc.eu]
Sent: Wednesday, June 01, 2016 5:24 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] Preliminary sign-off for Firefox 47.0RC on the beta-cdntest channel

Drivers,

QE has finished testing the Firefox 47.0RC (build1) candidate, and updates on the beta-cdntest channel. Manual testing efforts and automated update tests found no blocking issues for pushing to beta.

From the QE side, we are ready to make the builds live on the beta channel, so handing over to RelMan to give the "go" on that.

For more information on testing of 47.0RC build1, see our test plan at
https://wiki.mozilla.org/Releases/Firefox_47/Test_Plan#47.0_RC
</pre>

== RC - sign off on beta ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: Florin Mezei [mailto:florin.mezei@softvisioninc.eu]
Sent: Wednesday, June 01, 2016 7:26 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] Firefox 47.0 RC updates on beta channel signed off by QE

Update tests on the beta channel look good. Results are here.

Signing off for 47.0 RC (build 1) on this channel.
</pre>

== RC - sign off on release-cdntest ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: release-drivers [mailto:release-drivers-bounces@mozilla.org] On Behalf Of Florin Mezei
Sent: Tuesday, June 07, 2016 5:09 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] Firefox 47.0 (build 3) signed off on the release-cdntest channel by QE

Update tests on the release-cdntest channel look good. See results at: https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=b0310cb90fd0&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release-cdntest(.

Test Plan can be found at: https://wiki.mozilla.org/Releases/Firefox_47/Test_Plan#47.0_RC3

Pending Release Management decision, we can go forward with releasing this one.
</pre>

== RC - sign off on release ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: release-drivers [mailto:release-drivers-bounces@mozilla.org] On Behalf Of Florin Mezei
Sent: Tuesday, April 26, 2016 8:18 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] QE signing off on 46.0 release

Hi,

Update tests look good on the release channel.

More details in the Test Plan: https://wiki.mozilla.org/Releases/Firefox_46/Test_Plan#Release_46.0.
</pre>

== DOT RELEASE - sign off on release-localtest ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: release-drivers [mailto:release-drivers-bounces@mozilla.org] On Behalf Of Florin Mezei
Sent: Tuesday, May 03, 2016 9:29 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] Firefox 46.0.1 signed off on the release-localtest channel by QE

Hi,

Update tests on the release-localtest channel look good, manual testing showed no new issues, and bug https://bugzil.la/1267318 has been verified.

With that, I'm signing off for QE on release-localtest and handing over to release management.

For more information on testing of 46.0.1, see our test plan at:
https://wiki.mozilla.org/Releases/Firefox_46/Test_Plan#46.0.1_Build_1
</pre>

== RELEASE & DOT RELEASE - sign off on release-cdntest ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: release-drivers [mailto:release-drivers-bounces@mozilla.org] On Behalf Of Florin Mezei
Sent: Tuesday, May 03, 2016 10:32 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] Firefox 60.0.2 signed off on the release-cdntest channel by QE

Update tests on the release-cdntest channel look good. See results at: https://public.etherpad-mozilla.org/p/Firefox_60.0.2-build1.

Test Plan can be found at:
https://wiki.mozilla.org/Releases/Firefox_60/Test_Plan#60.0.2_Build_1

Pending Release Management decision, we can go forward with releasing this one.
</pre>

== RELEASE & DOT RELEASE - sign off on release ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: Florin Mezei [mailto:florin.mezei@softvisioninc.eu]
Sent: Wednesday, May 04, 2016 12:15 AM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] QE signing off on 60.0.2 release

Hi,

Update tests (https://public.etherpad-mozilla.org/p/Firefox_60.0.2-build1) look good on the release channel.

More details in the Test Plan:
https://wiki.mozilla.org/Releases/Firefox_60/Test_Plan#Release_60.0.2
</pre>

== ESR - sign off on esr-localtest ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: Florin Mezei [mailto:florin.mezei@softvisioninc.eu]
Sent: Tuesday, April 26, 2016 3:46 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] QE Signs off on Firefox 52.9.0 ESR on the "esr-localtest" channel

Hi,

We have completed the following manual update testing using "esr-localtest" channel for Firefox 52.9.0 ESR (build1) and found no new issues:
o Results: https://public.etherpad-mozilla.org/p/esr-52.9.0-build1
o Test Plan: https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan#Firefox_52.9.0_ESR
</pre>

== ESR - sign off on esr-cdntest ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: Florin Mezei [mailto:florin.mezei@softvisioninc.eu]
Sent: Tuesday, April 26, 2016 4:46 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] QE Signs off on Firefox 52.9.0 ESR build1 on the "esr-cdntest" channel

Hi,

We have completed the following manual update testing using "esr-cdntest" channel for Firefox 52.9.0 ESR (build1) and found no new issues:
o Results: https://public.etherpad-mozilla.org/p/esr-52.9.0-build1
o Test Plan: https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan#Firefox_52.9.0_ESR
</pre>

== ESR - sign off on esr ==
<pre style="white-space:-moz-pre-wrap; white-space:-pre-wrap; white-space:-o-pre-wrap; white-space:pre-wrap; word-wrap:break-word;">
From: Florin Mezei [mailto:florin.mezei@softvisioninc.eu]
Sent: Tuesday, April 26, 2016 7:42 PM
To: 'release-drivers' <release-drivers@mozilla.org>
Subject: [desktop] QE Signs off on Firefox 52.9.0 ESR on the "esr" channel

Hi,

We have completed the following manual update testing using "esr" channel for Firefox 52.9.0 ESR (build1) and found no new issues:
o Results: https://public.etherpad-mozilla.org/p/esr-52.9.0-build1
o Test Plan: https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan#Firefox_52.9.0_ESR
</pre>

= Troubleshooting =
# All tests fail
## Usually it’s a sign of:
### Incorrect settings in the config file:
#### Action: check the config file again and make sure the correct values are set for “target-version”, and “channel”
### Some massive problem with Jenkins/Marionette (e.g. [https://bugzilla.mozilla.org/show_bug.cgi?id=1271661 Network issues])
#### Actions:
##### Check https://mozilla-releng.net/treestatus/ to see if '''mozilla-beta''', '''mozilla-release''', or '''mozilla-esrXX''' are OPEN or CLOSED
##### Ask on #automation or #releng if still unclear if there is a problem or not
# Many test failures
## Whenever ~25% or more of the tests fail it’s a sign of one of the following:
### Some problem in the config file
#### Action: check again the config file
### There is some watershed rule in place that causes older versions to not update directly to the latest version, but instead go through an intermediary version (e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1234277#c4)
#### Action: classify failures as "expected fail" with comment stating that it's due to watershed rules from bug X
### Some problem on Jenkins/Marionette (haven’t encountered such a situation)
#### Action: contact :whimboo, :AutomatedTester, :maja_zf (#automation, #releng, #release-drivers), or write directly on #automation, or #releng and ask for help
# Few test failures
## A low number of test failures can be caused by all sorts of intermittent bugs
### Action: pin the failure and rebuild
### If the test still fails after rebuilding it’s possible that the localized build does not exist, or there is a real issue so notify people on #release-drivers (especially whimboo if available as he can check what the problem is)

== Rebuilding ==
If a run failed or you want to run it again for a platform, all you have to do is to rebuild it:
# In the pinboard click on the '''“Job details”''' tab
# Click on the link next to '''“Inspect Jenkins Build (VPN required):”'''
# This will bring you to the Jenkins page for that test - e.g. http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/23245/
# On this Jenkins page click the '''“Rebuild”''' link on the left of the page
# Now click the '''“Rebuild”''' button on the bottom of the page
# Several seconds after the test was rebuilt you should see a new entry on the Treeherder [https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr45&revision=32eb8c423d67&filter-tier=3&filter-searchStr=Fxup-esr-localtest report] page with the same name as the one that failed (e.g. “lij-1”) but in gray (meaning it’s running), right next to the previously failed job - if the rebuild passes you will then see it in green and are all done

== Taking node offline ==
If a run is aborted without clear reason (or tests fail because of some issues), you may need to take some node(s) offline, so people on the infra team can fix the problem and then take them back online. To take a node offline:
# In the pinboard (picture below) click on the '''“Job details”''' tab
# Click on the link next to '''“Inspect Jenkins Build (VPN required):”'''
# This will bring you to the Jenkins page for that test - e.g. http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/23245/
# Click on the machine link on the right of the page (e.g. [http://mm-ci-production.qa.scl3.mozilla.com:8080/computer/mm-win-81-64-3/ mm-win-81-64-3])
# Click “Make this node temporarily offline”

== Cancel runs ==
In some cases (e.g. run triggered with bad config.ini) you may want to cancel the run, or individual jobs one by one:
# Cancel individual jobs - '''TESTED''' (should work properly)
## Go to Jenkins [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/ ondemand update] page
## In the Job Queue (left side) you should see which jobs are '''done''' (full green/red bar), which '''in progress''' (in progress green/red bar), and which '''pending''' (text)
## To cancel a job (In Progress or Pending):
### Click on the close ('''x''') button on the right of the job link
### Click on the link for a job and then click the '''"x"''' icon on top right of the page
# Cancel the run - '''UNTESTED''' (may NOT work properly)
## Go to Jenkins [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/trigger-ondemand/ trigger-ondemand] page
## Look for the run you want to cancel in the Build Queue (left side)
## Click on the close ('''x''') button of the build you want to cancel

Mobile/Fennec/Android/Updating SDK on builders

2018-06-30T02:00:06Z

Garbas: new tokens and tooltool url

This page details how to update the remote builders' SDK versions for Nightly builds, Treeherder, etc.

== Legal notes ==
The Android SDK must have "internal" access rights because we are not allowed to distribute the SDK.

== Requirements ==
* A tooltool upload authorization token - create one at https://mozilla-releng.net/tokens/

== Procedure ==
(If you're updating Gradle dependencies instead of the SDK, you can run a [https://gecko.readthedocs.org/en/latest/build/buildsystem/toolchains.html#firefox-for-android-with-gradle TaskCluster job] to create the new build dependencies. Otherwise...)

* Download the [http://developer.android.com/sdk/index.html#Other latest Android SDK for Linux]

* Download the packages necessary for building. On Linux, this can be done via the `android` utility (in the downloaded SDK, not locally!). For other platforms, see below. You can verify which packages need to be installed by comparing with the android SDK currently available for the builders on tooltool (see the downloading section below). Currently (6/4/15), these tools are:
** Tools (the version supplied with the SDK may be outdated)
** Platform
** Platform tools
** Build tools
** Android Support Library
** Google Play services

* Place the downloaded packages into the downloaded SDK, following the pattern of an existing SDK (either from tooltool, or your own local install). Note that the packages often rename themselves after unzipping (e.g. mine were called "android-5.1). Currently (6/4/15), this looks like:
**build-tools/<version>
**extras/android/support
**extras/google/google_play_services
**platforms/android-<#>
**platform-tools/
**tools/

* Compress the SDK! The filename is currently android-sdk.tar.xz (i.e. compressed with xz, unclear if this is a requirement). To tar (because, you know, those flags): `tar -cJf android-sdk.tar.xz android-sdk-linux`. On OS X, tar does not support xz by default - you can download gnu-tar (`brew install gnu-tar`) and run the command above, substituting `gtar` for `tar`. Note: this can take a while.

* Upload the SDK via [https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool tooltool].
** Get (or update) tooltool: `git clone http://github.com/mozilla/build-tooltool`
** Add the SDK to the upload manifest: `python <path-to-tooltool>/tooltool.py add --visibility internal android-sdk.tar.xz`. This will output a new manifest.tt file or amend an existing one. Note that the `--visibility internal` is critical for compliance with Google's license!
** Get an upload authentication token via https://mozilla-releng.net/tokens/ (provided you have the permissions - ping garbas on IRC to get it)
** Upload the SDK: `python <path-to-tooltool>/tooltool.py upload --authentication-file=<path-to-auth-token> --message "Bug ###: <message>"`

* Update the builder configuration files to point at the new uploaded SDK: follow the model of [https://bugzilla.mozilla.org/show_bug.cgi?id=1165422 bug 1165422]. Use the information in the manifest.tt file update the configs.

* Consider if there's anything else to update (e.g. gradle build versions. see also [https://bugzilla.mozilla.org/show_bug.cgi?id=1165422 bug 1165422])

* (Maybe? I did) Add a CLOBBER (i.e. edit the CLOBBER file)

* Test it! Push it to try.

* Green? Land it!

== Downloading tools on non-Linux machines ==
You can't run the `android` binary locally so you'll have to download the tools from the URLs directly:
* Run `android list sdk` - it'll output it the source XML for the various tools it can install. For the primary packages, this is currently located [https://dl.google.com/android/repository/repository-10.xml here].
* Open the url (e.g. `curl <url> | less`) and look through the XML for the tools at the versions you need them. The XML tags follow a pattern:
** Tools: <sdk:tool>
** Platform: <sdk:platform>
** Platform tools: <sdk:platform-tool>
** Build tools: <sdk:build-tool>
** Android Support Library: (separate xml file) <sdk:extra>
** Google Play services: (separate xml file) <sdk:extra>
* Look for an <sdk:url> tag. If it's a full url, you're in! Otherwise, guess at the URL based on where you downloaded the SDK and look at some other urls in the file. See the [[#Example URLs|example URLs]].

=== Example URLs ===
* Platform: https://dl-ssl.google.com/android/repository/android-22_r02.zip
* Platform tools: https://dl-ssl.google.com/android/repository/platform-tools_r22-linux.zip
* Build tools: https://dl-ssl.google.com/android/repository/build-tools_r22.0.1-linux.zip
* Android Support Library: https://dl-ssl.google.com/android/repository/support_r19.1.zip
* Google Play Services: https://dl-ssl.google.com/android/repository/google_play_services_7571000_r25.zip

== Downloading previously uploaded files ==
Motivation: maybe you need to double-check the format of the previous SDK.

Note that you'll need an auth token because the access rights on the SDK are internal.
*For the most recent SDK, look [https://dxr.mozilla.org/mozilla-central/source/mobile/android/config/tooltool-manifests/android/releng.manifest here].
**Alternatively, for older files, go to [https://mozilla-releng.net/tooltool/ tooltool] and find the file you want to download. For the SDK, searching for the bug number seems most optimal. There may also be a download button on the page, in order to skip the steps through the tooltool script below.
*Write a manifest file using the info collected on tooltool (note: sha512 is always used), e.g.:
[
{
"size": 329217868,
"digest": "1cf54f12c271a2eea8f03caafa458207768ba12400cf53d742e3c4654a32b18fb75d8e0dece95e27691695751a6f6514ccb724b1a74958948c5f111c5c00da4f",
"algorithm": "sha512",
"filename": "android-sdk.tar.xz"
}
]
*Download the file! e.g.
python <path-to-tooltool>/tooltool.py fetch --url https://mozilla-releng.net/tooltool/ \
-m <path-to-manifest> --authentication-file=<path-to-auth-file>

ReleaseEngineering/Applications/Mapper

2018-06-30T01:54:42Z

Garbas: new mapper url

= What is it? =
When we convert hg repositories to git, and vice versa, the hg changeset SHA (the 40 character hexadecimal string that you get when you commit a change) is different to the git commit id (the equivalent SHA used by git).

In order to keep track of which hg changeset SHAs relate to which git commit SHAs, we keep a database of the mappings, together with details about the project the SHAs come from, and what time they were inserted into the database.

The vcs sync tool (checked into mozharness) is the tool which performs the conversion between hg repos and git repos, and this is documented separately. It is responsible for performing the conversion - this is outside the scope of mapper.

Mapper is a rest api, that allows:
# insertion of new mappings and projects (a "project" is essentially the name of the repo - e.g. build-tools) (HTTP POST)
# insertion of git/hg mappings for a given project (HTTP POST)
# retrieval of mappings for a given project (HTTP GET)

Behind the scenes, it is reading/writing from the database (using sqlalchemy).

Note: the vcs sync tool is a client of the mapper: it is vcs sync that inserts into mapper (i.e. uses the HTTP POST methods).
The other clients of mapper will be:
# people / developers - wanting to query mappings
# b2g_build.py - the build script for b2g - since this needs to lookup shas in order to reference frozen commit versions in manifests

Mapper is written as a RelEng API blueprint - please note RelEng API has its [https://docs.mozilla-releng.net/projects/releng-mapper.html own documentation] too.

= Mapper Development & Deployment =

The mapper is a project in [https://github.com/mozilla-releng/services mozilla-releng/services] repository, and so the details about how to deploy locally for development, and where the staging/production environments are, can be seen in the [https://docs.mozilla-releng.net/ RelEngAPI documentation].

= Mapper Data Feed =

VCS Sync is making HTTP Post requests to mapper, providing new mappings for mapper to insert into its database.

= API methods =

Below is a summary of the 4 unauthenticated API methods currently usable, together with example urls from the production environment.

For fuller details, and endpoints requiring authentication, the in process documentation is authoritative: [https://mapper.mozilla-releng.net/docs production] or [https://mapper.staging.mozilla-releng.net/docs staging] instances.

== GET Routes ==

=== Returns a mapping pair ===
* GET: /<project>/rev/<vcs_type>/<commit>
* Example: https://mapper.mozilla-releng.net/build-puppet/rev/git/69d64a8a18e6e001eb015646a82bcdaba0e78a24
* Example: https://mapper.mozilla-releng.net/build-puppet/rev/hg/68f1b2b9996c4e33aa57771b3478932c9fb7e161

=== Returns full mapfile for a given project ===
* GET: /<project>/mapfile/full
* Example: https://mapper.mozilla-releng.net/build-puppet/mapfile/full

=== Returns a subset of a mapfile, since a given date ===
* GET: /<project>/mapfile/since/<since>
* Example: https://mapper.mozilla-releng.net/build-mozharness/mapfile/since/29.05.2014%2017:02:09%20CEST

=== Returns list of projects ===
* GET: /projects
* Example: https://mapper.mozilla-releng.net/projects

= Staging =

The staging environment is available under https://mapper.staging.mozilla-releng.net (Look under "Documentation" to find details.)

ReleaseEngineering/Applications/Tooltool

2017-10-09T12:21:05Z

Garbas: don't suggest to use --url anymore

= Tooltool basics =

Some of the jobs in the releng infrastructure make use of generic binary artifacts, which are stored in dedicated artifacts repositories.

Tooltool is the client program, written in Python, which is used to retrieve these artifacts from the servers, and to perform integrity checks on them based on hashcode verification. Hence, the servers storing these artifacts are named "tooltool servers".

A set of files to be fetched by tooltool is specified via a tooltool manifest, usually a file in JSON format with tt extension.

The following is an example of a valid tooltool manifest:

[
{
"size": 139308,
"digest": "b2a463249bb3a9e7f2a3604697b000d2393db4f37b623fc099beb8456fbfdb332567013a3131ad138d8633cb19c50a8b77df3990d67500af896cada8b6f698b4",
"algorithm": "sha512",
"filename": "file2.pdf"
},
{
"size": 3017536,
"digest": "630d01a329c70aedb66ae7118d12ff7dc6fe06223d1c27b793e1bacc0ca84dd469ec1a6050184f8d9c35a0636546b0e2e5be08d9b51285e53eb1c9f959fef59d",
"algorithm": "sha512",
"filename": "file1.pdf"
},
{
"size": 3420686,
"digest": "931eb84f798dc9add1a10c7bbd4cc85fe08efda26cac473411638d1f856865524a517209d4c7184d838ee542c8ebc9909dc64ef60f8653a681270ce23524e8e4",
"algorithm": "sha512",
"filename": "file3.pdf"
}
]

The simplest usecase for tooltool is to provide a manifest and the url of a tooltool server (this url is not longer used, the last modified file is from 13-Oct-2015) , and run a "fetch" command to download the files mentioned in the manifest:

python tooltool.py fetch -m my-manifest.tt

If a manifest name is not provided, tooltool will default to manifest.tt.

Tooltool is build on https://api.pub.build.mozilla.org/tooltool/ and to find the uploaded file you can search after : Filename,SHA512 hash,Upload author or Upload message (including bug number).

example: search after "mitmproxy"

The tooltool download servers are simply apache server folders with a flat structure (all files are stored at root level with no subfolders).

There are global options and command arguments. All terminal interactions after the option parser finishes is done through the Python logging API. The default is to print logging.INFO and higher messages. Currently, the following global options exist:

-q/--quiet tells Tooltool to print only logging.ERROR and higher messages
-v/--verbose specifies to print logging.INFO and higher
-m/--manifest <file> instructs Tooltool to reference a manifest file located at the specified path
-d/--algorithm <algorithm> instructs Tooltool to use the specified algorithm
-o/--overwrite tells Tooltool to overwrite a local file if the filename matches the manifest but the hash value is different to the manifest
--url specifies the base url to be used for remote operations

= Where's tooltool code? =

Tooltool's main dev repo is [https://github.com/mozilla/build-tooltool https://github.com/mozilla/build-tooltool].

The version of tooltool which is actually deployed in our infrastructure is in https://hg.mozilla.org/build/puppet/file/default/modules/packages/templates/tooltool.py. The tooltool.py file in the puppet repo should be in sync with the main dev repo (they should be identical, apart from the she-bang line).

= Listing and Validating =

The two most basic commands list a manifest and validate the local files against the manifest. The list command lists out all of the files in the manifest as well as whether they are present and/or valid. The return code from listing is zero unless there was an error in listing the files. Absent or invalid files will still result in an exit code of zero if there was no error in the listing process. The validate command is used to check if all the files in the manifest are present and valid. The exit code for validating is zero if all files in the manifest are present and their hash matches the manifest. It is non-zero if any file is missing locally or the file does not have the same hash as the manifest.

= Other tooltool features =

== The tooltool cache ==

This has been implemented in bug [https://bugzilla.mozilla.org/show_bug.cgi?id=858635 https://bugzilla.mozilla.org/show_bug.cgi?id=858635]
All the bellow examples are used for the older packages that are on "http://tooltool.pub.build.mozilla.org/temp-sm-stuff" url.
When connecting to a tooltool server to fetch a file, it is now possible to use a local cache specifying the local cache folder with the -c option, e.g.:

python tooltool.py fetch --url http://tooltool.pub.build.mozilla.org/temp-sm-stuff -c ~/cache

In the previous example, folder ~/cache will be inspected for artifacts before connecting to the specified tooltool server. If the specified cache folder does not exist, tooltool will create it (if the user running tooltool has permissions to do so!).
Purging the cache with the purge command

A mechanism to purge the tooltool cache has been implemented as a separate command. Let's see some examples of purge commands:

python tooltool.py purge -c ~/cache
python tooltool.py purge -c ~/cache -s 34
python tooltool.py purge -c ~/cache --size 34

In the first example, we are cleaning up the tooltool cache completely, i.e.: all the content will be wiped out

In the second and third example the extra parameter -s (or, in extended form, --size) is provided, which specifies the number of Gigs we want to be free at the end of the purging process. Tooltool will delete files in the cache folder, starting with the oldest ones, until the specified amount of gigs is free on disk. When there is more free space that the specified gigs when the command is invokes, no files will be deleted at all.

=== Enabling automated tooltool cache purge within purge_build.py ===

Since the purge_builds.py script (within the build-tools repo) is already responsible of clean-up operations, the option of cleaning the tooltool cache has been added to that script as well.

In order for the automated cleanup to take place, two environment variables need to be setup when purge_builds.py is invoked, and namely:

TOOLTOOL_HOME: the folder containing tooltool.py
TOOLTOOL_CACHE: the local folder being used as tooltool cache (which will be cleaned up)

= How Tooltool is invoked in build/test jobs =

Two wrapper scripts exist which are usually being used to invoke tooltool in build/test jobs:
* tooltool_wrapper.sh: http://hg.mozilla.org/build/tools/file/327616201e98/scripts/tooltool/tooltool_wrapper.sh
* TooltoolMixin class for mozharness calls: https://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/mozharness/mozilla/tooltool.py (despite being an Homonym of tooltool itself, this is a python module defining TooltoolMixin class to be used in mozharness calls).

= How To Upload To Tooltool =

{{note|Make sure you're using the latest tooltool.py from https://github.com/mozilla/build-tooltool}}

First, plan out your upload. Are all of the files you need to upload public? If not, tooltool also offers storage of "internal" files, which are not made publicly available but can be downloaded by those with proper permissions. Internal files might be under a non-redistribution license (e.g., Google SDKs, Microsoft DLLs), but '''must not include secrets''' such as passwords or private keys.

Use the `tooltool.py` client to build a manifest containing the files you would like to upload, annotating each file with its visibility level:

python tooltool.py add --visibility public gcc.tar.gz

Next, you will need credentials for the upload. Tooltool is hosted on RelengAPI, so the credentials take the form of a RelengAPI token. To get such a token, to to https://api.pub.build.mozilla.org, log in, click on "Tokens", and issue a user token with the permissions ''tooltool.upload.internal'' and/or ''tooltool.upload.public'', depending on the nature of your upload. If you don't have internal permission, then the option won't be available to you. When you click "Issue", you'll get a long string which you should copy/paste into a (protected) file on your system, and pass to tooltool.py with ''--authentication-file''.

Once all of that is in place, you're ready to perform the upload. Just like for a Git or Mercurial commit, you'll need to provide a message to indicate why you're making this upload. Including a bug reference is always helpful.Also make sure the "file" is in the same repository from where you run tooltool.

python tooltool.py upload "file" --authentication-file=~/.tooltool-token --message "Bug 1234567: add new frobnicator binaries"

The tool will begin uploading and, when it's finished, may wait a minute or two longer until the files are verified by RelengAPI. If your upload fails for some reason, just restart it. Any files already completely uploaded will not need to be uploaded again.

You can view the available files and uploads at https://api.pub.build.mozilla.org/tooltool.

= Tooltool uploads: the old school =

'''''This is the old-school, deprecated, manual way to upload artifacts to tooltool. Please use the official procedure described above whenever possible, in order to keep information about what is uploaded, by whom, and why (the manual procedure doesn't allow to store any metadata about uploads).'''''

If you don't want to upload from your own laptop (because, eg, you have a slow uplink) you can do this from cruncher.

Access cruncher with your credentials:
ME="your_short_ldap_username" # or `whoami`
ssh -A $ME@cruncher.build.mozilla.org
Download the file and then:
scp filename.tar.xz $ME@relengwebadm.private.scl3.mozilla.com:
Login to relengwebadmn:
ssh $ME@relengwebadm.private.scl3.mozilla.com
And deploy the file to tooltool:
<pre>
FILE=~/emulator.zip # or whatever you're uploading
export SHA512=`openssl sha512 $FILE | cut -d' ' -f2`
sudo mv -i $FILE /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
sudo chmod 644 /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
ls -l /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
</pre>

* Add the filename, filesize, and sha512 digest to the bug you are working on. These can be added to the tooltool manifests later.

Webdev/Meetings/2017/May

2017-05-02T16:26:14Z

Garbas: my topic for the day

{{WebdevMeetingInfobox
|date=May 2nd, 2017
|airmo=https://air.mozilla.org/webdev-extravaganza-may-2017/
}}

Webdev Extravaganzas are open to the public and serve as a gathering point for anyone in the Mozilla community who is interested in web development and what Mozilla has been doing in it.

The meeting will be streamed and recorded on Air Mozilla: https://air.mozilla.org/webdev-extravaganza-may-2017/

== Shipping Celebration ==
What did we ship this month? Alternatively, what didn't we ship to spare the world the horror?

* (shobson) MDN's CSS code style guide is now documented in a [https://github.com/mozilla/kuma/blob/master/.stylelintrc%20 .stylelintrc] file, the [https://stylelint.io/ Stylelint.io] format. So it can be enforced on Pull Requests and supporting text editors can pop up friendly little error messages and warnings.
* (pmac) New library! [https://github.com/pmac/django-redirect-urls django-redirect-urls] was extracted from Bedrock. It was created to ease the task of replicating a lot of Apache URL rewriting rules in our code. It is now its own library so that other projects (most notably MDN) can potentially use it. Most other Django redirect helpers store redirects in the database.

* (garbas) Moved https://mozilla-releng.net from Elm 0.17 to 0.18

== Open-Source Citizenship ==
Any updates with our libraries or with libraries we use? Anyone looking for help with a library they maintain?

* (garbas) [https://github.com/zalando/connexion connexion], Swagger/OpenAPI First framework for Python on top of Flask with automatic endpoint validation & OAuth2 support. Since few weeks, not only Flask is supported (Pyramid/Falcon) and it is possible to plugin you custom framework too.

== New Hires / Interns / Volunteers / Contributors ==
Anyone new that we want to induct into the secret order? Paid, unpaid, contract, doesn't matter!

* (irc nick of new person) New Person

== Roundtable ==
Anything else to talk about?

* (garbas) Sphinx theme with new Mozilla branding?

Webdev/Meetings/2017/March

2017-03-03T13:34:20Z

Garbas: adding mozilla-releng things that happen

{{WebdevMeetingInfobox
|date=March 7th, 2017
|airmo=https://air.mozilla.org/webdev-extravaganza-march-2017/
}}

Webdev Extravaganzas are open to the public and serve as a gathering point for anyone in the Mozilla community who is interested in web development and what Mozilla has been doing in it.

The meeting will be streamed and recorded on Air Mozilla: https://air.mozilla.org/webdev-extravaganza-march-2017/

== Shipping Celebration ==
What did we ship this month? Alternatively, what didn't we ship to spare the world the horror?

* [garbas] TreeStatus: https://mozilla-releng.net/treestatus
* (irc nick) Topic

== Open-Source Citizenship ==
Any updates with our libraries or with libraries we use? Anyone looking for help with a library they maintain?

* (irc nick) Topic

== New Hires / Interns / Volunteers / Contributors ==
Anyone new that we want to induct into the secret order? Paid, unpaid, contract, doesn't matter!

* (irc nick of new person) New Person

== Roundtable ==
Anything else to talk about?

* [garbas] In the summer of 2016 we (in RelEng) decided to use Elm to build our frontends. After 6 months and 10k lines of Elm I can finally share our experience with Elm.
* (irc nick) Topic

MOSS/Foundational Technology/Projects We Use

2017-02-10T14:22:45Z

Garbas: technologies we are using in github.com/mozilla-releng/services

This is an incomplete list of the free software and open source projects Mozilla relies upon. As a starting point, it lists each project along with a short statement of how we depend on it. Where practical it would also be helpful to identify a Mozillian most closely associated with our use of each project.

Note that presence on this list isn't the same as, nor is it a prerequisite for, applying for the MOSS "Foundational Technology" track - projects have to make an application using the process on the [[MOSS/Foundational Technology|Foundational Technology]] page.

This is a work in progress - please contribute to this list.

{| class="wikitable"
|-
! Project !! Description !! Contact within Mozilla
|-
| [http://angleproject.org/ ANGLE] || A conformant OpenGL ES implementation for Windows, Mac and Linux || Platform Graphics Team
|-
| [https://angularjs.org/ angular.js] || Used by A-Team for web apps (eg Treeherder) || Member of Jonathan Griffin's team
|-
| [https://httpd.apache.org Apache Server] || Used by A-Team for web apps || Member of Jonathan Griffin's team
|-
| [https://github.com/ansible/ansible Ansible] || Used by IT (netops) and A-Team to manage deployments || jbarnell , Webops, GPS?
|-
| [https://github.com/h4writer/arewefastyet AreWeFastYet] || Compare JS performance across JS engines and platforms || Hannes Verschore
|-
| [https://github.com/argonjs Argon] || Used as the foundation for mobile AR web browser research by Emerging Tech || Blair MacIntyre
|-
| [https://atom.io/ Atom] || Used (either vanilla or e.g. as [https://github.com/Microsoft/vscode] ) by various developers ||
|-
| [https://babeljs.io/ BabelJS] || JavaScript compiler, Used by Gaia, TaskCluster team || Selena Deckelmann
|-
| [https://www.isc.org/downloads/bind/ BIND] || DNS Server || Brian Hourigan
|-
| [https://pypi.python.org/pypi/bleach/ Bleach] || HTML sanitizing library that escapes or strips markup and attributes used by MDN and a bunch of Mozilla sites || Will Kahn-Greene
|-
| [https://pypi.python.org/pypi/blessings/ Blessings] || Terminal formatting lib used by mozilla-central build process || Erik Rose
|-
| [http://getbootstrap.com/ Bootstrap] || HTML/CSS/JS framework, used by many of Mozilla's sites. || Webdev
|-
| [http://brew.sh/ brew / Homebrew] || Mac OS X packaging system, used to install dev tools || Sam Penrose
|-
| [https://bro.org bro] || The Bro Network Security Monitor || Michal Purzynski
|-
| [http://browserify.org Browserify] || Build JS dependencies for the browser || mozilla-services (kinto.js)
|-
| [http://buildbot.net/ BuildBot] || The base system currently in use for release || Dustin J. Mitchell
|-
| [http://bugzilla.org/ Bugzilla] (upstream) || The base Bugzilla on that we customize for Mozilla's use || Mark Côté
|-
| [http://www.celeryproject.org/ Celery] || Distributed task queue. Used by Treeherder and others. || Member of Jonathan Griffin's team
|-
| [http://cffi.readthedocs.io/en/latest/ cffi] || C Foreign Function Interface for Python. || Cloud Services Team
|-
| [http://ckeditor.com/ CKEditor] || WYSIWYG editor on MDN ||
|-
| [https://cmake.org/ CMake] || cross-platform family of tools designed to build, test and package software, lua_sandbox, lua_sandbox_extensions, hindsight, hindsight_admin || Mike Trinkala
|-
| [http://chaijs.com/ Chai] || JavaScript test and assertion library || Cloud Services, Tarek Ziade's team (kinto.js), Firefox Hello team
|-
| [http://www.llvm.org Clang/LLVM] || C/C++ compiler and infrastructure || Ehsan Akhgari & Sylvestre Ledru
|-
| [https://codemirror.net/ CodeMirror] || Used in DevTools, [https://thimble.mozilla.org Thimble], and other online code tools || David Humphrey/Simon Wex
|-
| [https://pypi.python.org/pypi/cookiecutter/ Cookiecutter] || Templating system used to clone our [https://github.com/mozilla/sugardough Sugardough] Django template || Giorgos Logiotatidis
|-
| [https://conemu.github.io/ ConEmu] || Console emulator for Windows. Used by devs running Windows. || Ed Morley
|-
| [https://pypi.python.org/pypi/ConfigArgParse ConfigArgParse] || A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables || Cloud Services Team
|-
| [http://crt.sh/ crt.sh] || Global certificate corpus search website || Gervase Markham
|-
| [http://curl.haxx.se/ curl] || internet transfer tool and library, used by crashreporter and FirefoxOS || Daniel Stenberg
|-
| [https://www.debian.org/ Debian] || Used on many developer boxes || Sylvestre Ledru or glandium
|-
| [http://deis.io Deis] || Open Source Heroku-like PaaS platform. Hosts www.mozilla.org, masterfirefoxos.mozilla.org, etc. || Member of Benjamin Sternthal's team
|-
| [https://disconnect.me/trackerprotection Disconnect.me ] || Open, transparent, neutral Tracking Protection blocklist used in Firefox Private Browsing Windows || Marshall Erwin, Javaun Moradi, Francois Marier
|-
| [https://discourse.org Discourse] || [https://discourse.mozilla-community.org Community], [https://discourse.webmaker.org/ Webmaker], [https://discourse.mozilla-advocacy.org/ Advocacy], et al || [[IT/Community/WG/Discourse|Community Ops]] (Yousef Alam or Tanner Filip)
|-
| [https://djangoproject.com Django] || Backend web framework used on many of our websites, including addons.mozilla.org, marketplace.mozilla.org, support.mozilla.org, Input, Snippets, MDN (Mozilla Developer Network), mozilla.org, nucleus, Treeherder || Andy McKay and Jannis Leidel are (or have been) on the [https://www.djangoproject.com/foundation/ Django Software Foundation] board, Jannis is core team member
|-
| [http://www.django-rest-framework.org Django REST framework] || API framework. Used by various Mozilla sites including MDN, Firefox marketplace, mozilla.org, support.mozilla.com. || Andy McKay and Jannis Leidel
|-
| [https://github.com/docker/docker Docker] || Used by release engineering for Linux build and test containers and by ateam for managing test and production services. Used by many Mozilla websites as well such as addons.mozilla.org, marketplace.mozilla.org. (Plus docker-compose and docker machine) || Member of Selena Deckelmann's team
|-
| [http://www.stack.nl/~dimitri/doxygen/ Doxygen] || tool for generating documentation from annotated sources, lua_sandbox || Mike Trinkala
|-
| [https://drupal.org/ Drupal] || Open Source CMS. Used by Dev Rel team. || Potch
|-
| [https://pypi.python.org/pypi/ecdsa ecdsa] || ECDSA cryptographic signature library (pure python) || Cloud Services Team
|-
| [https://eclipse.org/ Eclipse] || Integrated Development Environment used by many developers || —
|-
| [https://www.elastic.co/products/elasticsearch elasticsearch] || Search engine for various web sites and analytics || Erik Rose (dxr), Kyle Lahnakoski (ActiveData), Jason Thomas
|-
| [http://elm-lang.org/ Elm] || A delightful language for reliable webapps. || Rok Garbas (garbas), Release Engineering Team
|-
| [http://www.gnu.org/software/emacs/ emacs] || Programmable editor used by many developers || —
|-
| [https://emojione.com/ EmojiOne] || Upstream Emoji artwork used in the Firefox built-in Emoji font || Tim Chien ([https://github.com/mozilla/emojione-colr See emojione-colr])
|-
| [http://eslint.org/ eslint] || Pluggable linting utiltity used by Firefox Hello, DevTools and Firefox Android || Mark Banner
|-
| [https://github.com/ether/etherpad-lite Etherpad] || Used for meeting notes, etc || JP Schneider
|-
| [https://ffmpeg.org/ ffmpeg] || Used for media decoding. || Jean-Yves Avenard
|-
| [https://flake8.readthedocs.org/ flake8] || Wrapper around Python linters. Used by Treeherder and others. || Member of Jonathan Griffin's team
|-
| [http://flask.pocoo.org/ Flask] || Python web framework || Erik Rose
|-
| [https://flowtype.org/ Flow] || A static type checker for JavaScript || Greg Tatum, Jason Laster
|-
| [https://fortawesome.github.io/Font-Awesome/ Font Awesome] || Font and CSS toolkit, used by many of Mozilla's sites. || Webdev
|-
| [http://fontforge.github.io/en-US/ FontForge] || Built-in Emoji font compilation || Tim Chien ([https://github.com/mozilla/emojione-colr See emojione-colr])
|-
| [https://pypi.python.org/pypi/FontTools FontTools] || Built-in Emoji font compilation || Tim Chien ([https://github.com/mozilla/emojione-colr See emojione-colr])
|-
| [http://foundation.zurb.com/ Foundation Framework] || Responsive front-end framework, used by some Mozilla sites and add-ons || Luke Crouch
|-
| [https://www.freetype.org/ FreeType] || Small, efficient, highly customizable, and portable library for producing high-quality output (glyph images) of most vector and bitmap font formats. || Platform team
|-
| [http://gcc.gnu.org GCC] || C/C++ compiler and infrastructure || Nathan Froyd
|-
| [http://https://www.sourceware.org/gdb/ GDB] || The GNU Debugger, the sole debugger for Linux || Jim Blandy
|-
| [https://git-scm.com/ Git] || Version control system - https://git.mozilla.org || Unknown
|-
| [http://graphite.sil.org/ Graphite] || Graphite is a "smart font" system developed specifically to handle the complexities of lesser-known languages of the world. || Platform team
|-
| [http://gunicorn.org/ gunicorn] || Python WSGI HTTP Server. Used by Treeherder, Socorro, Pontoon. || Webdev
|-
| [http://www.freedesktop.org/wiki/Software/HarfBuzz/ HarfBuzz] || International text shaping engine used in Firefox/Servo || Platform team
|-
| [http://hunspell.sourceforge.net/ Hunspell] || Spellchecking engine || Ehsan Akhgari
|-
| [https://github.com/hunspell/hyphen/ Hyphen] || Hyphenation library || Jonathan Kew
|-
| [https://github.com/jkbrzt/httpie HTTPie] || HTTP command-line client || Cloud Services (among many others), Tarek Ziade's team
|-
| [https://python-hyper.org/en/latest/ hyper] || Hyper is a set of related projects that provide HTTP/2 functionality to Python projects. || Cloud Services Team
|-
| [https://github.com/icecc/icecream Icecream (icecc)] || Distributed build system, used to speed up local Firefox build times in some Mozilla offices || Daniel Holbert, or contributors on the [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Using_Icecream Icecream MDN Page]
|-
| [http://www.inspircd.org/ inspircd] || Irc server used by mozilla || :ashish
|-
| [https://theintern.github.io/intern/ Intern] || Intern is a complete test system for JavaScript designed to help you write and run consistent, high-quality test cases for your JavaScript libraries and applications. || jrgm/vladikoff
|-

| [http://canonware.com/jemalloc/ jemalloc] || Memory allocation library || We can ask glandium
|-
| [http://jenkins-ci.org/ Jenkins CI] || Continuous integration system used by WebQA and EE || Unknown
|-
| [https://jquery.com/ jQuery] || JavaScript library, used by many of Mozilla's sites. || Webdev
|-
| [https://github.com/tmpvar/jsdom jsdom] || DOM implementation in full JS || Test suites in Cloud Services, Tarek Ziade's team (kinto.js)
|-
| [https://github.com/open-source-parsers/jsoncpp jsoncpp] || A C++ library for interacting with JSON, used by the crashreporter client and Socorro. || Gabriele Svelto
|-
| [https://kafka.apache.org/ Kafka] || Distributed transaction log, used for hg.mozilla.org replication. || gps
|-
| [http://kombu.readthedocs.org/ Kombu] || Messaging library for Python. Used by Treeherder and others. || Member of Jonathan Griffin's team
|-
| libbz2 || Compression library for .bz2 format || Julian Seward
|-
| [http://libevent.org/ libevent] || Provides a mechanism to execute a callback function when a specific event occurs on a file descriptor or after a timeout has been reached. || Platform team
|-
| [https://sourceware.org/libffi/ libffi] || A Portable Foreign Function Interface Library. || Platform team
|-
| [http://libjpeg-turbo.virtualgl.org/ libjpeg-turbo] || JPEG decoding library || Jeff Muizelaar
|-
| [http://www.libpng.org/pub/png/libpng.html libpng] || PNG decoding library || Jeff Muizelaar
|-
| libvpx (Google) || Library for support of Google’s VP* family of codecs || Tim Terriberry
|-
| Linux || OS kernel used in Firefox OS || Unknown
|-
| LLVM || Compiler foundation for ASAN, static analysis tools, Rust || Unknown
|-
| [https://lodash.com/ lodash] || JavaScript utility library, used by many of Mozilla's sites. || Webdev & Firefox Hello
|-
| [https://www.lua.org/ Lua] || Lua is a powerful, efficient, lightweight, embeddable scripting language, lua_sandbox_extensions || Mike Trinkala
|-
| [https://github.com/macvim-dev/macvim MacVim] || Mac wrapper around Vim || -
|-
| [https://www.list.org/ Mailman] || [https://mail.mozilla.org/listinfo Mailing lists] || Unknown
|-
| [https://pypi.python.org/pypi/marshmallow marshmallow] || A lightweight library for converting complex datatypes to and from native Python datatypes. || Cloud Services Team
|-
| [https://www.mediawiki.org/wiki/MediaWiki MediaWiki] || You are reading this on a wiki || Shyam Mani
|-
| [https://www.mercurial-scm.org/ Mercurial] || Version control system and source code management || GPS
|-
| [https://www.meteor.com/ Meteor] || Build Apps with Javascript - Used in MozDef's Web UI || MozDef Team in EIS
|-
| [http://mochajs.org/ Mocha] || JavaScript test runner || Cloud Services, Tarek Ziade's team (kinto.js); Firefox Hello team
|-
| [https://github.com/kumar303/mohawk Mohawk] || Python library for Hawk HTTP authorization. Used by Treeherder and others. || Member of Jonathan Griffin's team
|-
| [http://mozdef.com mozdef] || Security event monitoring and incident response || Jeff Bryner
|-
| [http://www.mingw.org/wiki/msys msys] || Used to build Firefox on Windows. Note: It's likely best we support the newer MSYS2 project instead: https://github.com/msys2 || Unknown
|-
| [https://www.mysql.com/ MySQL] || Open source relational DB used by many developers, including AMO, SUMO, Input, bugzilla, releng, adminstered by IT || Matt Pressman
|-
| [https://www.nagios.org/ Nagios] || IT management system. Used for notifications of system failures || Linda Ypulong
|-
| [http://netsniff-ng.org/ netsniff-ng] || Linux networking toolkit || Michal Purzynski
|-
| [https://neovim.io/ neovim] (nvim) || Next-generation vim code editor || Ralph Giles
|-
| nICEr || Library for traversing firewalls || Unknown
|-
| [https://nixos.org/Nix Nix] || Reproducible build tool. || Rok Garbas (garbas), Release Engineering Team
|-
| [https://nixos.org NixOS] || Reproducible Linux distribution. Used by some developers || Nicolas B. Pierron
|-
| [https://www.nginx.com/resources/wiki/contributing/github/ nginx] || Web Server. Used by almost everything CloudOps runs. || Travis Blow
|-
| [https://nodejs.org Node.js] || JavaScript runtime for server side applications, command line utilities || Nick Desaulniers
|-
| nrappkit || Toolkit for building standalone applications || Unknown
|-
| [http://nsis.sourceforge.net/Main_Page NSIS] || Windows installer framework || Matt Howell
|-
| [https://wiki.mozilla.org/NSS NSS] || Network Security Services || jcjones?
|-
| [https://github.com/cisco/OpenH264 OpenH264] (Cisco) || H.264 video library || Unknown
|-
| [http://www.openLDAP.org/ OpenLDAP] || User management used by Infra || :jabba?
|-
| [http://openresty.org OpenResty] || a fast web app server by extending nginx, used by cloudops || Benson Wong (mostlygeek)
|-
| [http://www.openssh.com/ OpenSSH] || Remote server management, secure transport for Git and Mercurial || Unknown
|-
| [http://openssl.org/ OpenSSL] || Cryptograpahy and TLS Toolkit || Nick Desaulniers
|-
| [https://www.openstreetmap.org OpenStreetMap] || Online/offline maps, used by the location service and on mozilla.org || Unknown
|-
| [https://openvpn.net/ Openvpn] || VPN used by Infra || :jabba?
|-
| [https://github.com/khaledhosny/ots OTS] || OpenType sanitizer used by Firefox to protect against security bugs in underlying platforms related to malicious fonts || Jonathan Kew
|-
| [https://www.owasp.org/index.php/ZAP OWASP ZAP] || Web security testing tool used by security and QA teams || Simon Bennetts (psiinon)
|-
| [https://owncloud.org/ Owncloud] || Open platform to host your cloud under your control || Some communities have been using it to host files under their control (ask Nukeador)
|-
| [http://pandoc.org/ Pandoc] || universal document converter, lua_sandbox_extensions || Mike Trinkala
|-
| [https://github.com/apache/parquet-cpp parquet-cpp] || C++ library to read and write the Apache Parquet columnar data format, lua_sandbox_extensions || Mike Trinkala
|-
| [https://pypi.python.org/pypi/parsimonious/ Parsimonious] || Parsing lib used by DXR and a few other sites (I think) || Erik Rose
|-
| [https://github.com/mozilla/pdf.js PDF.js] || Used as the PDF Viewer in Firefox and Firefox OS || Brendan Dahl, Yury Delendik
|-
| [https://pdfium.googlesource.com/pdfium/ PDFium] || PDF viewing and printing in Firefox || Peter Van Der Beken, Evelyn Hung
|-
| [https://github.com/PyCQA/pep8 pep8] || Python linter. Used by Treeherder and others. || Member of Jonathan Griffin's team
|-
| [https://www.perl.org/ Perl] || Used by Bugzilla || Member of Jonathan Griffin's team
|-
| [http://python-pillow.org/ pillow] || Python Imaging library || Member of Webcompat team
|-
| [https://github.com/pypa/pip pip] || Python package tool. Used by all Python projects. || Erik Rose and Jannis Leidel, Jannis is core team member ([https://www.pypa.io/ PyPA])
|-
| [http://piwik.org/ Piwik] || Analytic software that gives you the control and respects privacy || Some communities have been using it to avoid GA analytics (Ask Nukeador)
|-
| [http://cmusphinx.sourceforge.net/ Pocketsphinx] || Speech recognition toolkit embedded into Firefox OS|| André Natal
|-
| [http://www.postgresql.org/ PostgreSQL] || Open source relational DB used by many developers, adminstered by IT || Selena Deckelmann
|-
| [https://github.com/PrismJS/prism Prism.js] || Syntax highlighting on code samples on MDN ||
|-
| [https://github.com/puppetlabs/puppet Puppet] || System administration tool || Member of Amy Rich's team, Corey Shield's team
|-
| [https://github.com/JetBrains/intellij-community/tree/master/python PyCharm] || Python IDE || Jonathan Griffin
|-
| [https://github.com/pyflakes/pyflakes pyflakes] || Python linter. Used by Treeherder and others. || Member of Jonathan Griffin's team
|-
| [http://pypy.org/ PyPy] || Python language runtime. Used by Web Push service. || Cloud Services, esp. Ben Bangert
|-
| [https://github.com/Pylons/pyramid/ Pyramid] || Python Web framework || Cloud Services Team
|-
| [http://pytest.org/ pytest] || Python testing tool. Used by Treeherder and others. || Dave Hunt
|-
| Python || Scripting language || [http://python.org/psf/ Python Software Foundation], Selena Deckelmann is a former board member
|-
| [https://pypi.python.org/pypi/cryptography python cryptography] || cryptography is a package which provides cryptographic recipes and primitives to Python developers. || Cloud Services Team
|-
| [https://pypi.python.org/pypi/pyelasticsearch/ pyelasticsearch] || Python client for elasticsearch || Erik Rose
|-
| [https://python-jose.readthedocs.io/en/latest/ python-jose] || a JavaScript Object Signing and Encryption (JOSE) implementation in python || Cloud Services Team
|-
| [https://www.rabbitmq.com/ RabbitMQ] || Distributed Queue, used by Socorro, Pulse (all our publicly available build/test/commit information flows through this), Treeherder, addons.mozilla.org (with Celery), marketplace.mozilla.org (with Celery) || Selena Deckelmann, Jason Thomas
|-
| [http://http://rapidjson.org/ RapidJSON] || A fast JSON parser/generator for C++ with both SAX/DOM style API, lua_sandbox_extensions || Mike Trinkala
|-
| [https://pypi.python.org/pypi/raven Raven] || Raven is a python client for Sentry || Cloud services Team
|-
| [http://facebook.github.io/react/index.html ReactJS] || Javascript library for building user interfaces. Used by Firefox Hello & DevTools || Mark Banner
|-
| [https://readthedocs.org/ Read the Docs] || Hosted automatically-built documentation, used by Cloud Services and a wide variety of mozilla Github projects || Ben Bangert, Jannis Leidel, Gervase Markham
|-
| [http://redis.io/ Redis] || Really fast data structure store, cache and message broker || Cloud Services Tarek Ziade's team, Loop Server (Hello) team
|-
| [http://redux.js.org/ Redux] || Redux is a predictable state container for JavaScript apps. || Jason Laster
|-
| [http://python-requests.org/ Requests] || "Python HTTP Requests for Humans". Used by many many Mozilla Python projects. || Member of Jonathan Griffin's team
|-
| [https://www.reviewboard.org/ Review Board] || The base of MozReview, the new review tool being developed to replace Splinter || Steven MacLeod, Mike Conley
|-
| [https://sass-lang.com/ Sass] || CSS Preprocessor. Most big team web projects use Sass these days. ||
|-
| [http://wordlist.aspell.net/ SCOWL] || en-US word list used for spell checking. || ehsan
|-
| [http://www.seleniumhq.org/ Selenium] || Browser test driver || David Burns, Dave Hunt
|-
| [http://sinonjs.org/ Sinon] || JavaScript mock library || Cloud Services, Tarek Ziade's team (kinto.js); Firefox Hello team
|-
| [https://skia.org/ Skia] || A complete 2D graphic library for drawing Text, Geometries, and Images || Platform Graphics Team
|-
| [https://slimerjs.org/ SlimerJS] || Scriptable browser, based on Gecko, used for functional tests in some few Mozilla projects ( [[PluotSorbet]], some FxOS apps ? and others ?) || Myk Melez
|-
| [http://www.sqlalchemy.org/ SQLAlchemy]] || Database Toolkit and ORM for Python || bhearsum
|-
| [https://www.sqlite.org/ SQLite] || File-based database || asuth
|-
| [http://learnboost.github.io/stylus/ Stylus] || CSS Pre-processor on several sites || webdev
|-
| [https://subversion.apache.org/ Subversion] || https://svn.mozilla.org ''(Planning to decommission in near future)'' || Unknown
|-
| [https://suricata-ids.org/ suricata] || IDS / IPS / NSM engine || Michal Purzynski
|-
| [http://www.squid-cache.org/ Squid] || Caching proxy || Brian Hourigan
|-
| [https://tox.readthedocs.org/en/latest/ Tox] || Test automation || Dave Hunt
|-
| [https://travis-ci.org/ Travis] || Continuous integration system used by several teams (eg Treeherder) || Jonathan Griffin's team
|-
| [https://github.com/twitter/twemoji twemoji] || Upstream Emoji artwork used in the Firefox built-in Emoji font (minor source) || Tim Chien ([https://github.com/mozilla/emojione-colr See emojione-colr])
|-
| [https://uwsgi-docs.readthedocs.io/en/latest/ UWSGI] || Full hosting stack used in MozDef || MozDef Team in EIS
|-
| [https://github.com/mitchellh/vagrant Vagrant]|| Build and distribute dev envs, used by Treeherder and others || Member of Jonathan Griffin's team
|-
| [http://valgrind.org/ Valgrind] || Memory error detection and profiling of C and C++ code || jseward, njn
|-
| [https://github.com/microsoft/vscode/ vscode] || Code editor || unknown
|-
| [http://vim.org/ vim] || editor used by many developers || —
|-
| [https://waffle.io/mozilla/ waffle.io] || github project management used by various teams ||
|-
| [https://github.com/w3c/web-platform-tests web-platform-tests] || Testcases and tooling for cross-browser testing of web-platform APIs || jgraham
|-
| [http://www.webrtc.org WebRTC.org] || Components to support real-time communication in browsers and mobile applications || Randell Jesup
|-
| [http://www.wordpress.org WordPress] || Powers our blogs, blog.mozilla.org || Craig Cook
|-
| [https://www.webtoolkit.eu/wt Wt] || C++ library for developing web applications, hindsight_admin || Mike Trinkala
|-
| [https://xiph.org Xiph.Org] || Media codecs ship in Firefox, encoding tools || Ralph Giles
|-
| [https://github.com/Cyan4973/xxHash xxHash] || Extremely fast non-cryptographic hash algorithm, lua_sandbox_extensions || Mike Trinkala
|-
| [http://tukaani.org/xz/embedded.html XZ Embedded] || Decompressor for the XZ format ||
|-
| [http://yasm.tortall.net/ yasm] || Assembler used by the build system. || unknown
|-
| [https://valloric.github.io/YouCompleteMe/ YouCompleteMe] || Code completion engine for vim editors || Ehsan Akhgari
|-
| [http://zlib.net zlib] || Streaming compression and decompression for HTTP, PNG, etc. ||
|-
| [https://zookeeper.apache.org/ ZooKeeper] || Distributed synchronization, use by hg.mozilla.org replication || gps
|}