CloudServices/DataPipeline: Difference between revisions
Jump to navigation
Jump to search
(→Cross Team Communication: --program) |
(--program) |
||
| Line 11: | Line 11: | ||
* Cross team coordination meeting: https://etherpad.mozilla.org/data-pipeline-coordination | * Cross team coordination meeting: https://etherpad.mozilla.org/data-pipeline-coordination | ||
* FHR mailing list: [https://mail.mozilla.org/listinfo/fhr-dev fhr-dev] | * FHR mailing list: [https://mail.mozilla.org/listinfo/fhr-dev fhr-dev] | ||
* | * [http://mzl.la/1FIEruc Metrics Program] | ||
= Resources = | = Resources = | ||
Revision as of 23:51, 21 February 2015
Overview
The cloud services data pipeline ingests data for analysis, monitoring and reporting. The pipeline is currently used for processing cloud services server logs. We're in the process of improving it to support desktop and device telemetry data. The data pipeline team also works on Heka (a major component of the pipeline implementation), custom dashboards for cloud services projects, and the Telemetry server.
Team Communication
- IRC channel: #datapipeline
- Mailing list: dev-metrics-pipeline@mozilla.com
- Standup meeting: https://etherpad.mozilla.org/data-pipeline-meeting-notes
- Bugzilla: http://mzl.la/1DOOBZt
Cross Team Communication
- Cross team coordination meeting: https://etherpad.mozilla.org/data-pipeline-coordination
- FHR mailing list: fhr-dev
- Metrics Program
Resources
Pipeline specs/docs
Reporting and tools
Planning
Pipeline Milestones
- Q1 2015: Launch pipeline prototype
- Architecture decisions completed; production stack up and running with monitoring dashboards
- Business Intelligence/Data Warehouse proof of concept implemented
- Ingestion process completed for FHR+telemetry (start collecting on 2015-02-23)
- Backprocessing from pipeline datastore implemented
- Pipeline runs in parallel to existing infrastructure; not yet source of truth
- Q2 2015: Pipeline officially supports business use cases
- FHR v4 feeds executive dashboard
- Complete set of use cases tbd (most likely primarily FHR+telemetry use cases)
- Complete set of monitoring and reporting outputs tbd: dashboards, data warehouse, monitoring, self-service access to data
- FHR+telemetry hits full release 2015-05-19, handle full production load
- Q3 2015: Fill out monitoring and reporting capabilities; add sources and use cases
Related Dates and Schedules
- FHR+Telemetry client work
- Current plan: FF38
- 2015-02-23 Nightly
- 2015-03-30 Aurora
- 2015-05-11 Release
Work Queue
Tracking tasks in bugzilla: http://mzl.la/1DOOBZt
Risks and Open Questions
- Old-FHR data through pipeline? Yes/No: [telliot]
- Deletes & legal policy [telliot]
- Security review [telliot]
Code
V2 Pipeline
- https://github.com/mozilla-services/heka
- https://github.com/mozilla-services/data-pipeline
- https://github.com/mozilla/pipeline-monitoring-dashboard
Telemetry
- https://github.com/mozilla/telemetry-server
- https://github.com/bsmedberg/telemetry-experiments-dashboard