EngineeringProductivity/Projects/Everything

Project Table

The project table has been moved to https://wiki.mozilla.org/EngineeringProductivity/Projects

Motivation

This page is to help diagram all the various things the A*Team has built and continue to support.

The plan is to draw high level components/systems, and links between them, so it is easier to see how all the parts work together. Hopefully this diagram will be clickable so the reader can get more detail on each of the components.

Something like what RelEng has, but without all the internals.

Template can be found below

The List

ActiveData

• Docs - https://wiki.mozilla.org/Auto-tools/Projects/ActiveData
• Code - https://github.com/klahnakoski/ActiveData
• Contact - klahnakoski@mozilla.com
• Inputs
  • Buildbot JSON logs
  • hg.mozilla.org
  • OrangeFactor
  • Perfherder
  • Pulse exchange/build/normalized
  • Structured Logs from Tests
  • Talos
  • Text logs (for buildbot and mozharness steps only)
• Outputs/Services
  • Query endpoint: http://activedata.allizom.org/query
  • Query Tool: http://activedata.allizom.org/tools/query.html

AlertManager

• Docs -https://github.com/jmaher/alert_manager/
• Code - https://github.com/jmaher/alert_manager/
• Contact' - jmaher@mozilla.com
• Inputs
  • mozilla.dev.tree-management alerts from graph server
• Outputs/Services
  • webUI for organizing and taking action on these alerts.

Autoland

• Docs - http://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview/autoland.html
• Code - http://hg.mozilla.org/hgcustom/version-control-tools/file/tip/autoland
• Contact - [:mcote], [:glob]
• Inputs
  • MozReview UI & MozReview repo
• Outputs
  • Commits land on the repository of record (or try)
  • Status reported to MozReview

Autophone

• Docs - https://wiki.mozilla.org/Auto-tools/Projects/AutoPhone
• Code - https://github.com/mozilla/autophone/
• Contact -
• Inputs
• Outputs/Services
  • http://phonedash.mozilla.org/ (used by who?)

Bugzilla / BMO

• Docs - https://bmo.readthedocs.io
• Code - https://github.com/mozilla-bteam/bmo
• Contact - Mark Cote, David Lawrence, Dylan Hardison
• Inputs
  • Web UI
  • WebServices API (REST, XMLRPC, JSONRPC)
• Outputs/Services
  • https://bugzilla.mozilla.org (bug reporters, developers, etc)
  • Pulse, Bugzfeed (via the [[1]])
  • BMO Push service (custom notifications for specific applications)

BugzFeed

• Docs - BMO/ChangeNotificationSystem
• Code - https://github.com/mozilla/bugzfeed
• Contact - mcote@mozilla.com
• Inputs
  • BMO via Pulse
• Outputs/Services
  • WebSocket servers:
    • bugzfeed.mozilla.org
    • bugzfeed-dev.allizom.org

Bugzilla/ES Cluster

• Docs - https://wiki.mozilla.org/BMO/ElasticSearch
• Code - https://github.com/klahnakoski/Bugzilla-ETL
• Contact' - klahnakoski@mozilla.com
• Inputs -
  • Bugzilla (direct database access)
• Outputs/Services
  • https://esfrontline.bugzilla.mozilla.org:443/public_bugs/bug_version

Buildbot

• Docs -
• Code -
• Contact -
• Inputs
  • hg.mozilla.org
• Outputs/Services
  • builder slaves
  • tester slaves
  • https://wiki.mozilla.org/ReleaseEngineering/BuildAPI ??
  • JSON Logs: http://builddata.pub.build.mozilla.org/builddata/buildjson/

C++ Code Coverage

• Docs - https://bugzilla.mozilla.org/show_bug.cgi?id=890116
• Contact - [:ahal]
• Used to be possible to trigger code coverage builds with 'try: -b o -p linux64-cc -u all -t none', but I think this is broken now. Probably isn't terribly hard to revive this. Needs further investigation.

charts.mozilla.org

• Docs -
• Code - https://github.com/mozilla/charts
• Contact' - klahnakoski@mozilla.com
• Inputs -
  • BMO/ES Cluster (https://esfrontline.bugzilla.mozilla.org:443/public_bugs/bug_version)
• Outputs/Services
  • http://charts.mozilla.org (for management)

CodeCoverage

• Docs - https://wiki.mozilla.org/EngineeringProductivity/Projects/CodeCoverage
• Code
  • UI - https://github.com/chinhodado/codecoverage_presenter
  • ETL - https://github.com/klahnakoski/ActiveData-ETL/blob/dev/activedata_etl/transforms/jscov_to_es.py
• Contact - jmaher@mozilla.com, klahnakoski@mozilla.com, chmanchester@mozilla.com
• Inputs
  • Various tests have been configured with a code coverage option
• Outputs/Services
  • ActiveData - https://wiki.mozilla.org/Auto-tools/Projects/ActiveData
  • http://chinhodado.github.io/codecoverage_presenter/

Datazilla (retired)

• Docs - https://datazilla.readthedocs.org/en/latest/
• Code - https://datazilla.readthedocs.org/en/latest/
• Contact - See Treeherder
• Inputs
  • data pushed from Talos (see Graph Server)
• Outputs/Services
  • charts at https://datazilla.mozilla.org
  • raw results at https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob

DevelopmentMetrics

• Docs - https://wiki.mozilla.org/Auto-tools/Projects/DevelopmentMetrics
• Code - https://github.com/klahnakoski/MoDevMetrics
• Contact - klahnakoski@mozilla.com
• Inputs
  • Bugzilla/ES Cluster
• Outputs/Services
  • various charts and dashboards

DevTools Harness

• Docs - Needs project page!
• Contact - [:ted.mielczarek]

dzAlerts (retired)

• Docs - https://wiki.mozilla.org/Auto-tools/Projects/Alerts
• Code - https://github.com/klahnakoski/datazilla-alerts
• Contact - klahnakoski@mozilla.com
• Inputs
  • Datazilla web service (https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob)
  • Eideticket web service (http://eideticker.mozilla.org)
• Outputs/Services
  • fxos-perf-alerts@mozilla.org (b2g sheriffs?)

Eideticker

• Docs - https://wiki.mozilla.org/Project_Eideticker#Documentation
• Code - https://github.com/mozilla/eideticker
• Contact - William Lachance, Dave Hunt
• Inputs
  • Nightly/inbound b2g and android builds.
• Outputs/Services
  • Eideticker dashboard (http://eideticker.mozilla.org)

m21s

• Docs - https://wiki.mozilla.org/Auto-tools/Projects/m21s
• Code - see project page
• Contacts - [:chmanchester][:ahal][:whimboo]
• Inputs
  • Mozmill release tests
• Outputs
  • 'Firefox Greenlight Tests' in Marionette, reporting to Treeherder

Marionette

• Docs - https://wiki.mozilla.org/Auto-tools/Projects/Marionette
• Code -
• Contact - [:AutomatedTester][:ato]
• What - Mozilla's native WebDriver implementation

Mozharness

• Docs - https://wiki.mozilla.org/Auto-tools/Projects/Mozharness
• Code - https://hg.mozilla.org/build/mozharness/
• Contact - [:armenzg]
• Inputs
  • command-line invocation via buildbot or locally
• Outputs/Services
  • build/test logs consumed by buildbot, Treeherder, TBPL

MozReview (Review Board)

• Docs - Auto-tools/Projects/CodeReviewTool
• Code - various pieces of https://hg.mozilla.org/hgcustom/version-control-tools; see docs for more.
• Contact - mcote@mozilla.com
• Inputs
  • BMO
  • hg.mozilla.org
• Outputs/Services
  • BMO

OrangeFactor

• Docs - Auto-tools/Projects/OrangeFactor
• Code - https://hg.mozilla.org/automation/orangefactor/
• Contact - :jgriffin, :mcote
• Inputs
  • buildbot messages (using pulse)
  • build logs
  • comments from TBPL
  • bugzilla bug data
• Outputs/Services
  • https://brasstacks.mozilla.com/orangefactor/
  • email status [War on Orange]

Ouija

• Docs - https://wiki.mozilla.org/Auto-tools/Projects/Ouija
• Code - https://github.com/dminor/ouija.git
• Contact - dminor, jmaher
• Inputs
  • https://hg.mozilla.org/
  • tbpl.mozilla.org
• Outputs/Services
  • http://54.215.155.53

Perfherder

• Docs - https://wiki.mozilla.org/EngineeringProductivity/Projects/Perfherder
• Code - https://github.com/mozilla/treeherder/tree/master/treeherder/perf
• Contact - :wlach
• Inputs
  • PERFHERDER_DATA
    • schema https://github.com/mozilla/treeherder/blob/master/schemas/performance-artifact.json
    • example http://wrla.ch/blog/2015/11/perfherder-onward/
• Outputs/Services
  • https://treeherder.mozilla.org/perf.html#/graphs

Pulse

• Docs - https://wiki.mozilla.org/Auto-tools/Projects/Pulse
• Code - https://hg.mozilla.org/automation/mozillapulse
• Contact - Mark Côté
• Inputs (exchanges? http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/)
  • a multitude of other systems
• Outputs/Services
  • a multitude of other systems
  • https://pulse.mozilla.org/

PulseTranslator

• Docs - https://github.com/mozilla/pulsetranslator/blob/master/README.md
• Code - https://github.com/mozilla/pulsetranslator
• Contact - jgriffin@mozilla.com
• Inputs
  • Pulse exchange/build
• Outputs/Services
  • Pulse exchange/build/normalized

Structured Logging

• Docs - https://mozbase.readthedocs.org/en/latest/loggingreporting.html, https://wiki.mozilla.org/Auto-tools/Projects/Structured_Logging
• Contacts - [:chmanchester][:ahal][:jgraham]

Talos

• Docs - https://wiki.mozilla.org/Buildbot/Talos
• Code - https://hg.mozilla.org/build/talos
• Contact - jmaher@mozilla.com
• Inputs
  • build for desktop or android
  • commandline (done via mozharness now) to specify which test(s) to run
• Outputs/Services
  • log file with raw data
  • uploaded summarization of data to graph server
  • uploaded raw data to datazilla

Can be referred to as what is run by buildbot or by tbpl since this is the only performance test run on our per revision CI system. Other tools are run at a different cadence.

TBPL (retired)

• Docs - Sheriffing/TBPL
• Code - https://hg.mozilla.org/webtools/tbpl/
• Contact - Sheriffing/TBPL#Developer_Contacts
• Inputs
• Outputs/Services

Test Informant

• Docs - https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant
• Code - See project page
• Contact - [:ahal]
• Inputs
  • build notifications via pulse
  • test manifests in tests.zip
• Outputs
  • reports at e.g., http://brasstacks.mozilla.com/testreports/weekly

Treeherder

• Docs - Auto-tools/Projects/Treeherder
• Code - Auto-tools/Projects/Treeherder#Source_and_Docs
• Contact - [:camd][:mdoglio][:edmorley]
• Inputs
  • Pushlog from hg.mozilla.org via json-pushes
  • Build/tests from Buildbot via buildapi
  • Direct pushes via ``treeherder-client``
  • Build/tests from Taskcluster (under development)
• Outputs/Services
  • https://treeherder.mozilla.org/
  • https://treeherder.mozilla.org/api/

web-platform-tests

• Docs - https://wptrunner.readthedocs.org/en/latest/, Needs project page?
• Code
  • https://github.com/w3c/web-platform-tests/
  • https://github.com/mozilla/gecko-dev/blob/master/testing/web-platform/README.md
  • https://github.com/mozilla/gecko-dev/blob/master/testing/web-platform/tests/README.md
• Contact - [:jgraham]

Example

Use the EMPTY TEMPLATE to add more entries. Include ones you know about, even if you can not fill them.

EXAMPLE

• Docs - some wiki, or read the docs to learn more
• Code - link to code, if it makes sense
• Contact - for ekyle to contact if he has questions
• Inputs -
  • automated resources
  • services consumed
• Outputs/Services
  • automated services provided
  • dashboards (and teams that consume them)

EMPTY TEMPLATE

• Docs -
• Code -
• Contact -
• Inputs
• Outputs/Services