CloudServices/DataPipeline: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎Overview: tweak)
(Remove cruft)
Line 1: Line 1:
= Overview =
= Overview =
The cloud services data pipeline ingests data for analysis, monitoring and reporting. The pipeline is currently used for processing desktop and device [[Telemetry|Telemetry]] data and cloud services server logs. The ingestion pipeline is one component of the [[Data/Platform|Fx Data Platform]].
The cloud services data pipeline ingests data for analysis, monitoring and reporting. The pipeline is currently used for processing desktop and device [[Telemetry|Telemetry]] data and cloud services server logs. The ingestion pipeline is one component of the [[Data/Platform|Fx Data Platform]].
= Team Communication =
* IRC channel: #datapipeline
* Mailing list: dev-metrics-pipeline@mozilla.com
* Standup meeting: https://etherpad.mozilla.org/data-pipeline-meeting-notes
* Bugzilla: http://mzl.la/1DOOBZt
= Cross Team Communication =
* FHR mailing list: [https://mail.mozilla.org/listinfo/fhr-dev fhr-dev]
* FHR v4 standup meeting: https://etherpad.mozilla.org/fhr-v4-status
* Cross team coordination meeting (ended 3/19): https://etherpad.mozilla.org/data-pipeline-coordination


= Resources =
= Resources =
Line 32: Line 21:
* [https://mana.mozilla.org/wiki/display/CLOUDSERVICES/Data+Sources List of Data Sources]
* [https://mana.mozilla.org/wiki/display/CLOUDSERVICES/Data+Sources List of Data Sources]
* [https://mana.mozilla.org/wiki/display/CLOUDSERVICES/V1+Pipeline V1 Pipeline & Data Sources]
* [https://mana.mozilla.org/wiki/display/CLOUDSERVICES/V1+Pipeline V1 Pipeline & Data Sources]
= Pipeline Milestones =
* '''Q1 2015''': Launch pipeline prototype
** Architecture decisions completed; production stack up and running with monitoring dashboards
** Business Intelligence/Data Warehouse proof of concept implemented
** Ingestion process completed for FHR+telemetry (start collecting on 2015-02-23)
** Backprocessing from pipeline datastore implemented
** By client ID analysis supported
** Pipeline runs in parallel to existing infrastructure; not yet source of truth
* '''Q2 2015''': Pipeline officially supports business use cases
** FHR v4 feeds executive dashboard
** Complete set of use cases tbd (most likely primarily FHR+telemetry use cases)
** Complete set of monitoring and reporting outputs tbd: dashboards, data warehouse, monitoring, self-service access to data
** FHR+telemetry hits full release 2015-05-19, handle full production load
* '''Q3 2015''': Fill out monitoring and reporting capabilities; add sources and use cases
= Related Dates and Schedules =
* '''FHR+Telemetry client work'''
** Current plan: FF38
** 2015-02-23 Nightly
** 2015-03-30 Aurora
** 2015-05-11 Release
= Work Queue =
Tracking tasks in bugzilla: http://mzl.la/1DOOBZt
=== Risks and Open Questions ===
* Old-FHR data through pipeline? Yes/No: [telliot]
* Deletes & legal policy [telliot]
* Security review [telliot]


= Code =
= Code =

Revision as of 18:33, 24 June 2016

Overview

The cloud services data pipeline ingests data for analysis, monitoring and reporting. The pipeline is currently used for processing desktop and device Telemetry data and cloud services server logs. The ingestion pipeline is one component of the Fx Data Platform.

Resources

Pipeline specs/docs

Reporting and tools

Data sets and other documentation

Planning

Code

V2 Pipeline

Link Description
https://github.com/mozilla-services/data-pipeline Mozilla Services Data Pipeline
https://github.com/mozilla-services/lua_sandbox Generic Lua sandbox for dynamic data analysis
https://github.com/mozilla-services/mozilla-pipeline-schemas JSON Schema specifications of pipeline data
https://github.com/mozilla/pipeline-monitoring-dashboard Monitoring data quality issues for metrics pipeline
https://github.com/mozilla-services/heka Data collection and processing made easy
https://github.com/mozilla-services/nginx_moz_ingest HTTP Data Pipeline Ingestion
https://github.com/trink/hindsight Data collection and processing made light weight, fast, and more reliable

Telemetry

Link Description
https://github.com/vitillo/telemetry-onboarding Slides / notebooks for Telemetry Onboarding
https://github.com/mozilla/telemetry-server Code for analysis.telemetry.mozilla.org among other things
https://github.com/bsmedberg/telemetry-experiments-dashboard A dashboard to track the deployment of Firefox Telemetry Experiments
https://github.com/mozilla/telemetry-batch-view A Scala framework to build derived datasets, aka batch views, of Telemetry data.
https://github.com/mozilla/cerberus Automatic alert system for telemetry histograms
https://github.com/mozilla/emr-bootstrap-spark AWS bootstrap scripts for Mozilla's flavoured Spark setup.
https://github.com/mozilla/moz-crash-rate-aggregates Crash Rate Aggregation code
https://github.com/mozilla/jupyter-notebook-gist Plugin to create, list, and load GitHub Gists from Jupyter notebooks
https://github.com/mozilla/jupyter-spark Jupyter Notebook extension for Apache Spark integration
https://github.com/mozilla/python_mozaggregator Aggregator job for telemetry.mozilla.org
https://github.com/mozilla/python_moztelemetry Spark bindings for Mozilla Telemetry
https://github.com/mozilla/telemetry-analysis-service Eventual home of the revamped a.t.m.o (per Bug 1248688)
https://github.com/vitillo/telemetry-airflow Scheduling / workflow management for Telemetry jobs
https://github.com/vitillo/e10s_analyses Data analysis relating to Electrolysis / E10s
https://github.com/mozilla/telemetry-tools Utility code to work with Mozilla Telemetry data

Archive

Retrieved from "https://wiki.mozilla.org/index.php?title=CloudServices/DataPipeline&oldid=1137586"