EngineeringProductivity/Projects/Everything: Difference between revisions

m (typo)
 
(118 intermediate revisions by 13 users not shown)
Line 1: Line 1:


== Motivation ==  
== Project Table ==


This page is to help diagram all the various things the A*Team has built and continue to support.
There is a project table that is a short list of the most active, projects: https://wiki.mozilla.org/EngineeringProductivity/Projects


The plan is to draw high level components/systems, and links between them, so it is easier to see how all the parts work together.  Hopefully this diagram will be clickable so the reader can get more detail on each of the components.
== Motivation ==


Something like [https://wiki.mozilla.org/images/f/f3/Releng_flow_onepage_treeclose_reasons.pdf what RelEng has], but without all the internals.
This page exists to provide a low-maintenance list of all our projects. I do not beleive anyone knows how they '''all''' interact, but at least this list ensures here are no hidden moving parts.


== History (blah blah blah..) ==
''Template can be found below''


My name is Kyle Lahnakoski, and I joined the A*Team in April 2013.
== The List ==


When I started, the plan was to improve the alerting mechanism in Datazilla. Jmaher and jeads were kind enough to give me an overview of the various systems surrounding [https://datazilla.mozilla.org/ Datazilla]. I drew my notes and diagrams on paper, and for months I carried my "secret decoder" so I could talk reasonably intelligently about the various A*Team systems and how they connected.
=== ActiveData ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/ActiveData
* '''Code''' - https://github.com/klahnakoski/ActiveData
* '''Contact''' - klahnakoski@mozilla.com
* '''Inputs'''
** Buildbot JSON logs
** hg.mozilla.org
** OrangeFactor
** Perfherder
** Pulse exchange/build/normalized
** Structured Logs from Tests
** Talos
** Text logs (for buildbot and mozharness steps only)
* '''Outputs/Services'''
** Query endpoint: http://activedata.allizom.org/query
** Query Tool: http://activedata.allizom.org/tools/query.html


It is late June 2014 now, and my notes are long lost.  I recently had a request from another team's member (:avih) about how dzAlerts fits into the greater ecosystem.  In a [https://wiki.mozilla.org/Auto-tools/Meetings/2014-06-23 recent meeting] jgriffin mentioned it would be nice to have a summary of all the great things the A*Team works on and supports.
=== AlertManager ===


I would like to draw another not-so-secret decoder, but this time with everything!
* '''Docs''' -https://github.com/jmaher/alert_manager/
 
* '''Code''' - https://github.com/jmaher/alert_manager/
Please help by adding the A*Team systems to The List below
* '''Contact' - jmaher@mozilla.com
 
* '''Inputs'''
== The List ==
** mozilla.dev.tree-management alerts from graph server
 
Use the EMPTY TEMPLATE to add more entries.  Include ones you know about, even if you can not fill them.
 
=== EXAMPLE ===
* '''Docs''' - some wiki, or read the docs to learn more
* '''Code''' - link to code, if it makes sense
* '''Contact''' - for ekyle to contact if he has questions
* '''Inputs''' -
** automated resources
** services consumed
* '''Outputs/Services'''
* '''Outputs/Services'''
** automated services provided
** webUI for organizing and taking action on these alerts.
** dashboards (and teams that consume them)


=== EMPTY TEMPLATE ===
=== Autoland ===
* '''Docs''' -  
* '''Docs''' - http://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview/autoland.html
* '''Code''' -
* '''Code''' - http://hg.mozilla.org/hgcustom/version-control-tools/file/tip/autoland
* '''Contact''' -  
* '''Contact''' - [:mcote], [:glob]
* '''Inputs'''
* '''Inputs'''
**
** MozReview UI & MozReview repo
* '''Outputs/Services'''
* '''Outputs'''
**
** Commits land on the repository of record (or try)
** Status reported to MozReview


=== Autophone ===
=== Autophone ===
Line 53: Line 58:
** http://phonedash.mozilla.org/ (used by who?)
** http://phonedash.mozilla.org/ (used by who?)


=== B2gPerf ===
=== Bugzilla / BMO ===
* '''Docs''' -
* '''Docs''' - https://bmo.readthedocs.io
* '''Code''' -
* '''Code''' - https://github.com/mozilla-bteam/bmo
* '''Contact''' -
* '''Contact''' - Mark Cote, David Lawrence, Dylan Hardison
* '''Inputs'''
**
* '''Outputs/Services'''
**
 
=== Bugzilla ===
* '''Docs''' - http://bugzilla.readthedocs.org
* '''Code''' - http://git.mozilla.org/?p=webtools/bmo/bugzilla.git
* '''Contact''' - Mark Cote, Byron Jones, David Lawrence, Dylan Hardison
* '''Inputs'''
* '''Inputs'''
** Web UI
** Web UI
Line 71: Line 67:
* '''Outputs/Services'''
* '''Outputs/Services'''
** https://bugzilla.mozilla.org (bug reporters, developers, etc)
** https://bugzilla.mozilla.org (bug reporters, developers, etc)
** WebServices API (For REST https://bugzilla.mozilla.org/rest)
** Pulse, Bugzfeed (via the [[https://wiki.mozilla.org/BMO/ChangeNotificationSystem]])
** BMO Push service (custom notifications for specific applications)


=== BugzFeed ===
=== BugzFeed ===
* '''Docs''' -  
* '''Docs''' - [[BMO/ChangeNotificationSystem]]
* '''Code''' -
* '''Code''' - https://github.com/mozilla/bugzfeed
* '''Contact''' -  
* '''Contact''' - mcote@mozilla.com
* '''Inputs'''
* '''Inputs'''
**
** BMO via [[Auto-tools/Projects/Pulse|Pulse]]
* '''Outputs/Services'''
** WebSocket servers:
*** bugzfeed.mozilla.org
*** bugzfeed-dev.allizom.org
 
=== Bugzilla/ES Cluster ===
 
* '''Docs''' - https://wiki.mozilla.org/BMO/ElasticSearch
* '''Code''' - https://github.com/klahnakoski/Bugzilla-ETL
* '''Contact' - klahnakoski@mozilla.com
* '''Inputs''' -
** Bugzilla (direct database access)
* '''Outputs/Services'''
* '''Outputs/Services'''
**  
** https://esfrontline.bugzilla.mozilla.org:443/public_bugs/bug_version
**


=== Buildbot ===
=== Buildbot ===
Line 88: Line 96:
* '''Contact''' -  
* '''Contact''' -  
* '''Inputs'''
* '''Inputs'''
** hg.mozilla.org?
** hg.mozilla.org
* '''Outputs/Services'''
* '''Outputs/Services'''
** builder slaves
** builder slaves
** tester slaves
** tester slaves
** https://wiki.mozilla.org/ReleaseEngineering/BuildAPI ??
** https://wiki.mozilla.org/ReleaseEngineering/BuildAPI ??
** where do buildbot results go, a file?  picked up by TBPL
** JSON Logs: http://builddata.pub.build.mozilla.org/builddata/buildjson/
 
=== C++ Code Coverage ===
 
* '''Docs''' - https://bugzilla.mozilla.org/show_bug.cgi?id=890116
* '''Contact''' - [:ahal]
* Used to be possible to trigger code coverage builds with 'try: -b o -p linux64-cc -u all -t none', but I think this is broken now. Probably isn't terribly hard to revive this. Needs further investigation.
 
=== charts.mozilla.org ===
 
* '''Docs''' -
* '''Code''' - https://github.com/mozilla/charts
* '''Contact' - klahnakoski@mozilla.com
* '''Inputs''' -
** BMO/ES Cluster (https://esfrontline.bugzilla.mozilla.org:443/public_bugs/bug_version)
* '''Outputs/Services'''
** http://charts.mozilla.org (for management)


==== Buildbot/Talos ====
=== CodeCoverage ===
* '''Docs''' - https://wiki.mozilla.org/Buildbot/Talos
* '''Docs''' - https://wiki.mozilla.org/EngineeringProductivity/Projects/CodeCoverage
* '''Code''' - http://hg.mozilla.org/build/talos
* '''Code'''
* '''Contact''' - jmaher@mozilla.com
** '''UI''' - https://github.com/chinhodado/codecoverage_presenter
** '''ETL''' - https://github.com/klahnakoski/ActiveData-ETL/blob/dev/activedata_etl/transforms/jscov_to_es.py
* '''Contact''' - jmaher@mozilla.com, klahnakoski@mozilla.com, chmanchester@mozilla.com
* '''Inputs'''
** Various tests have been configured with a code coverage option
* '''Outputs/Services'''
** ActiveData - https://wiki.mozilla.org/Auto-tools/Projects/ActiveData
** http://chinhodado.github.io/codecoverage_presenter/
 
=== Datazilla (retired) ===
 
* '''Docs''' - https://datazilla.readthedocs.org/en/latest/
* '''Code''' - https://datazilla.readthedocs.org/en/latest/
* '''Contact''' - See Treeherder
* '''Inputs'''
** data pushed from Talos (see Graph Server)
* '''Outputs/Services'''
** charts at https://datazilla.mozilla.org
** raw results at https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob
 
=== DevelopmentMetrics ===
 
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/DevelopmentMetrics
* '''Code''' - https://github.com/klahnakoski/MoDevMetrics
* '''Contact''' - klahnakoski@mozilla.com
* '''Inputs'''
* '''Inputs'''
** build for desktop or android
** Bugzilla/ES Cluster
** commandline (done via mozharness now) to specify which test(s) to run
* '''Outputs/Services'''
* '''Outputs/Services'''
** log file with raw data
** various charts and dashboards
** uploaded summarization of data to graph server
 
** uploaded raw data to datazilla
=== DevTools Harness ===
 
* '''Docs''' - Needs project page!
* '''Contact''' - [:ted.mielczarek]


Can be referred to as what is run by buildbot or by tbpl since this is the only performance test run on our per revision CI system.  Other tools are run at a different cadence.
=== dzAlerts (retired)===


=== Treeherder ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Alerts
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/TreeHerder_2013Q2
* '''Code''' - https://github.com/klahnakoski/datazilla-alerts
* '''Code'''
* '''Contact''' - klahnakoski@mozilla.com
** Service - https://github.com/mozilla/treeherder-service
** UI - https://github.com/mozilla/treeherder-ui
** Client - https://github.com/mozilla/treeherder-client
* '''Contact''' - [:jeads][:camd][:mdoglio]
* '''Inputs'''
* '''Inputs'''
** Pushlog from hg.mozilla.org via json-pushes
** Datazilla web service (https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob)
** Build/tests from Buildbot via buildapi
** Eideticket web service (http://eideticker.mozilla.org)
** Direct pushes via ``treeherder-client``
** Build/tests from Taskcluster (under development)
* '''Outputs/Services'''
* '''Outputs/Services'''
** https://treeherder.mozilla.org/ui/#/jobs
** fxos-perf-alerts@mozilla.org (b2g sheriffs?)
** https://treeherder.mozilla.org/api/


=== Datazilla ===
=== Eideticker (retired) ===


* '''Docs''' -  
* '''Docs''' - https://wiki.mozilla.org/Project_Eideticker#Documentation
* '''Code''' -
* '''Code''' - https://github.com/mozilla/eideticker
* '''Contact''' -  
* '''Contact''' - William Lachance, Dave Hunt
* '''Inputs'''
** Nightly/inbound b2g and android builds.
* '''Outputs/Services'''
** Eideticker dashboard (http://eideticker.mozilla.org)
 
=== Janitor ===
* '''Docs''' - https://mana.mozilla.org/wiki/display/ateam/Janitor
* '''Code''' - https://github.com/jankeromnes/janitor, https://github.com/jankeromnes/dockerfiles
* '''Contact''' - Jan Keromnes [:janx]
* '''Inputs'''
* '''Inputs'''
**
** (none yet)
* '''Outputs/Services'''
* '''Outputs/Services'''
**
** https://janitor.technology
** BMO: new patches uploaded by users
** Treeherder: new pushes from users
** Docker Hub: image updates https://hub.docker.com/u/janx/


=== Graph Server ===
=== m21s ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/m21s
* '''Code''' - see project page
* '''Contacts''' - [:chmanchester][:ahal][:whimboo]
* '''Inputs'''
** Mozmill release tests
* '''Outputs'''
** 'Firefox Greenlight Tests' in Marionette, reporting to Treeherder


* '''Docs''' -  
=== Marionette ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Marionette
* '''Code''' -
* '''Code''' -
* '''Contact''' -  
* '''Contact''' - [:AutomatedTester][:ato]
* '''What''' - Mozilla's native WebDriver implementation
 
=== Mozharness ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Mozharness
* '''Code''' - https://hg.mozilla.org/build/mozharness/
* '''Contact''' - [:armenzg]
* '''Inputs'''
** command-line invocation via buildbot or locally
* '''Outputs/Services'''
** build/test logs consumed by buildbot, Treeherder, TBPL
 
=== MozReview (Review Board) ===
* '''Docs''' - http://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview.html
* '''Code''' - various pieces of https://hg.mozilla.org/hgcustom/version-control-tools; see docs for more.
* '''Contact''' - mcote, glob, smacleod
* '''Inputs'''
* '''Inputs'''
**
** BMO
** reviewboard-hg.mozilla.org
* '''Outputs/Services'''
* '''Outputs/Services'''
**
** BMO
** Various repositories via Autoland


=== Orange Factor ===
=== OrangeFactor ===


* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/WarOnOrange
* '''Docs''' - [[Auto-tools/Projects/OrangeFactor]]
* '''Code''' - http://github.com/jmaher/Orange-Factor
* '''Code''' - https://hg.mozilla.org/automation/orangefactor/
* '''Contact''' -  
* '''Contact''' - :jgriffin, :mcote
* '''Inputs'''
* '''Inputs'''
** buildbot messages (using pulse)
** buildbot messages (using pulse)
Line 163: Line 243:
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Ouija
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Ouija
* '''Code''' - https://github.com/dminor/ouija.git
* '''Code''' - https://github.com/dminor/ouija.git
* '''Contact''' - jmaher
* '''Contact''' - dminor, jmaher
* '''Inputs'''
* '''Inputs'''
** https://hg.mozilla.org/
** https://hg.mozilla.org/
** tbpl.mozilla.org
** tbpl.mozilla.org
* '''Outputs/Services'''
* '''Outputs/Services'''
**
** http://54.215.155.53
 
=== Perfherder ===
* '''Docs''' - https://wiki.mozilla.org/EngineeringProductivity/Projects/Perfherder
* '''Code''' - https://github.com/mozilla/treeherder/tree/master/treeherder/perf
* '''Contact''' - :wlach
* '''Inputs'''
** PERFHERDER_DATA
*** schema  https://github.com/mozilla/treeherder/blob/master/schemas/performance-artifact.json
*** example http://wrla.ch/blog/2015/11/perfherder-onward/
* '''Outputs/Services'''
** https://treeherder.mozilla.org/perf.html#/graphs


=== Pulse ===
=== Pulse ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Pulse  
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Pulse  
* '''Code''' -
* '''Code''' - development library: https://hg.mozilla.org/automation/mozillapulse
* '''Contact''' -  
* '''Contact''' - mcote
* '''Inputs''' (exchanges? http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/)
* '''Inputs'''
**
** a multitude of other systems
* '''Outputs/Services'''
** a multitude of other systems
** https://pulse.mozilla.org/
 
=== PulseGuardian ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Pulse/PulseGuardian
* '''Code''' - https://github.com/mozilla/pulseguardian/
* '''Contact''' - mcote
* '''Inputs'''
** Persona (for auth) (soon to be Okta or TaskCluster auth)
** Pulse
** Web UI
* '''Outputs/Services'''
* '''Outputs/Services'''
**
** Pulse
** Email notifications


=== Review Board ===
=== PulseTranslator ===
* '''Docs''' -  
* '''Docs''' - https://github.com/mozilla/pulsetranslator/blob/master/README.md
* '''Code''' -
* '''Code''' - https://github.com/mozilla/pulsetranslator
* '''Contact''' -  
* '''Contact''' - jgriffin@mozilla.com
* '''Inputs'''  
* '''Inputs'''  
**
** Pulse exchange/build
* '''Outputs/Services'''
** Pulse exchange/build/normalized
 
=== Structured Logging ===
* '''Docs''' - https://mozbase.readthedocs.org/en/latest/loggingreporting.html, https://wiki.mozilla.org/Auto-tools/Projects/Structured_Logging
* '''Contacts''' - [:chmanchester][:ahal][:jgraham]
 
=== Talos ===
* '''Docs''' - https://wiki.mozilla.org/Buildbot/Talos
* '''Code''' - https://hg.mozilla.org/build/talos
* '''Contact''' - jmaher@mozilla.com
* '''Inputs'''
** build for desktop or android
** commandline (done via mozharness now) to specify which test(s) to run
* '''Outputs/Services'''
* '''Outputs/Services'''
**
** log file with raw data
** uploaded summarization of data to graph server
** uploaded raw data to datazilla
 
Can be referred to as what is run by buildbot or by tbpl since this is the only performance test run on our per revision CI system.  Other tools are run at a different cadence.


=== TBPL (for the sake of history) ===
=== TBPL (retired) ===


* '''Docs''' -  
* '''Docs''' - [[Sheriffing/TBPL]]
* '''Code''' -
* '''Code''' - https://hg.mozilla.org/webtools/tbpl/
* '''Contact''' -  
* '''Contact''' - [[Sheriffing/TBPL#Developer_Contacts]]
* '''Inputs'''
* '''Inputs'''
**
**
Line 198: Line 320:
**
**


=== Eideticker ===
=== Test Informant ===


* '''Docs''' -  
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant
** https://wiki.mozilla.org/Project_Eideticker#Documentation
* '''Code''' - See project page
* '''Code''' -  
* '''Contact''' - [:ahal]
** http://github.com/mozilla/eideticker
* '''Inputs'''
* '''Inputs''' -
** build notifications via pulse
** Nightly/inbound b2g and android builds.
** test manifests in tests.zip
* '''Outputs/Services'''
* '''Outputs'''
** Eideticker dashboard (http://eideticker.mozilla.org)
** reports at e.g., http://brasstacks.mozilla.com/testreports/weekly


=== BMO/ES Cluster ===
=== Treeherder ===
 
* '''Docs''' - [[Auto-tools/Projects/Treeherder]]
* '''Docs''' - https://wiki.mozilla.org/BMO/ElasticSearch
* '''Code''' - [[Auto-tools/Projects/Treeherder#Source_and_Docs]]
* '''Code''' - https://github.com/klahnakoski/Bugzilla-ETL
* '''Contact''' - [:camd][:mdoglio][:edmorley]
* '''Contact' - klahnakoski@mozilla.com
* '''Inputs'''
* '''Inputs''' -  
** Pushlog from hg.mozilla.org via json-pushes
** Bugzilla (direct database access)
** Build/tests from Buildbot via buildapi
** Direct pushes via ``treeherder-client``
** Build/tests from Taskcluster (under development)
* '''Outputs/Services'''
* '''Outputs/Services'''
** https://esfrontline.bugzilla.mozilla.org:443/public_bugs/bug_version
** https://treeherder.mozilla.org/
** https://treeherder.mozilla.org/api/


=== dzAlerts ===
=== web-platform-tests ===
* '''Docs''' - https://wptrunner.readthedocs.org/en/latest/, Needs project page?
* '''Code'''
** https://github.com/w3c/web-platform-tests/
** https://github.com/mozilla/gecko-dev/blob/master/testing/web-platform/README.md
** https://github.com/mozilla/gecko-dev/blob/master/testing/web-platform/tests/README.md
* '''Contact''' - [:jgraham]


* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Alerts
== Example ==
* '''Code''' - https://github.com/klahnakoski/datazilla-alerts
* '''Contact' - klahnakoski@mozilla.com
* '''Inputs'''
** Datazilla web service (https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob)
** Eideticket web service (http://eideticker.mozilla.org)
* '''Outputs/Services'''
** fxos-perf-alerts@mozilla.org (b2g sheriffs?)


=== AlertManager ===
Use the EMPTY TEMPLATE to add more entries.  Include ones you know about, even if you can not fill them.


* '''Docs''' -https://github.com/jmaher/alert_manager/
=== EXAMPLE ===
* '''Code''' - https://github.com/jmaher/alert_manager/
* '''Docs''' - some wiki, or read the docs to learn more
* '''Contact' - jmaher@mozilla.com
* '''Code''' - link to code, if it makes sense
* '''Inputs'''
* '''Contact''' - for ekyle to contact if he has questions
** mozilla.dev.tree-management alerts from graph server
* '''Inputs''' -
** automated resources
** services consumed
* '''Outputs/Services'''
* '''Outputs/Services'''
** webUI for organizing and taking action on these alerts.
** automated services provided
 
** dashboards (and teams that consume them)
=== charts.mozilla.org ===


=== EMPTY TEMPLATE ===
* '''Docs''' -  
* '''Docs''' -  
* '''Code''' - https://github.com/mozilla/charts
* '''Code''' -
* '''Contact' - klahnakoski@mozilla.com
* '''Contact''' -  
* '''Inputs''' -
* '''Inputs'''
** BMO/ES Cluster
**
* '''Outputs/Services'''
* '''Outputs/Services'''
** charts.mozilla.org (project managers)
**

Latest revision as of 15:39, 19 October 2016

Project Table

There is a project table that is a short list of the most active, projects: https://wiki.mozilla.org/EngineeringProductivity/Projects

Motivation

This page exists to provide a low-maintenance list of all our projects. I do not beleive anyone knows how they all interact, but at least this list ensures here are no hidden moving parts.

Template can be found below

The List

ActiveData

AlertManager

Autoland

Autophone

Bugzilla / BMO

BugzFeed

Bugzilla/ES Cluster

Buildbot

C++ Code Coverage

  • Docs - https://bugzilla.mozilla.org/show_bug.cgi?id=890116
  • Contact - [:ahal]
  • Used to be possible to trigger code coverage builds with 'try: -b o -p linux64-cc -u all -t none', but I think this is broken now. Probably isn't terribly hard to revive this. Needs further investigation.

charts.mozilla.org

CodeCoverage

Datazilla (retired)

DevelopmentMetrics

DevTools Harness

  • Docs - Needs project page!
  • Contact - [:ted.mielczarek]

dzAlerts (retired)

Eideticker (retired)

Janitor

m21s

Marionette

Mozharness

MozReview (Review Board)

OrangeFactor

Ouija

Perfherder

Pulse

PulseGuardian

PulseTranslator

Structured Logging

Talos

Can be referred to as what is run by buildbot or by tbpl since this is the only performance test run on our per revision CI system. Other tools are run at a different cadence.

TBPL (retired)

Test Informant

Treeherder

web-platform-tests

Example

Use the EMPTY TEMPLATE to add more entries. Include ones you know about, even if you can not fill them.

EXAMPLE

  • Docs - some wiki, or read the docs to learn more
  • Code - link to code, if it makes sense
  • Contact - for ekyle to contact if he has questions
  • Inputs -
    • automated resources
    • services consumed
  • Outputs/Services
    • automated services provided
    • dashboards (and teams that consume them)

EMPTY TEMPLATE

  • Docs -
  • Code -
  • Contact -
  • Inputs
  • Outputs/Services