|
|
Line 1: |
Line 1: |
| = Overview = | | = Overview = |
| The cloud services data pipeline ingests data for analysis, monitoring and reporting. The pipeline is currently used for processing desktop and device [[Telemetry|Telemetry]] data and cloud services server logs. The ingestion pipeline is one component of the [[Data/Platform|Fx Data Platform]]. | | The cloud services data pipeline ingests data for analysis, monitoring and reporting. The pipeline is currently used for processing desktop and device [[Telemetry|Telemetry]] data and cloud services server logs. The ingestion pipeline is one component of the [[Data/Platform|Fx Data Platform]]. |
|
| |
| = Team Communication =
| |
| * IRC channel: #datapipeline
| |
| * Mailing list: dev-metrics-pipeline@mozilla.com
| |
| * Standup meeting: https://etherpad.mozilla.org/data-pipeline-meeting-notes
| |
| * Bugzilla: http://mzl.la/1DOOBZt
| |
|
| |
| = Cross Team Communication =
| |
| * FHR mailing list: [https://mail.mozilla.org/listinfo/fhr-dev fhr-dev]
| |
| * FHR v4 standup meeting: https://etherpad.mozilla.org/fhr-v4-status
| |
| * Cross team coordination meeting (ended 3/19): https://etherpad.mozilla.org/data-pipeline-coordination
| |
|
| |
|
| = Resources = | | = Resources = |
Line 32: |
Line 21: |
| * [https://mana.mozilla.org/wiki/display/CLOUDSERVICES/Data+Sources List of Data Sources] | | * [https://mana.mozilla.org/wiki/display/CLOUDSERVICES/Data+Sources List of Data Sources] |
| * [https://mana.mozilla.org/wiki/display/CLOUDSERVICES/V1+Pipeline V1 Pipeline & Data Sources] | | * [https://mana.mozilla.org/wiki/display/CLOUDSERVICES/V1+Pipeline V1 Pipeline & Data Sources] |
|
| |
| = Pipeline Milestones =
| |
| * '''Q1 2015''': Launch pipeline prototype
| |
| ** Architecture decisions completed; production stack up and running with monitoring dashboards
| |
| ** Business Intelligence/Data Warehouse proof of concept implemented
| |
| ** Ingestion process completed for FHR+telemetry (start collecting on 2015-02-23)
| |
| ** Backprocessing from pipeline datastore implemented
| |
| ** By client ID analysis supported
| |
| ** Pipeline runs in parallel to existing infrastructure; not yet source of truth
| |
| * '''Q2 2015''': Pipeline officially supports business use cases
| |
| ** FHR v4 feeds executive dashboard
| |
| ** Complete set of use cases tbd (most likely primarily FHR+telemetry use cases)
| |
| ** Complete set of monitoring and reporting outputs tbd: dashboards, data warehouse, monitoring, self-service access to data
| |
| ** FHR+telemetry hits full release 2015-05-19, handle full production load
| |
| * '''Q3 2015''': Fill out monitoring and reporting capabilities; add sources and use cases
| |
|
| |
| = Related Dates and Schedules =
| |
| * '''FHR+Telemetry client work'''
| |
| ** Current plan: FF38
| |
| ** 2015-02-23 Nightly
| |
| ** 2015-03-30 Aurora
| |
| ** 2015-05-11 Release
| |
|
| |
| = Work Queue =
| |
| Tracking tasks in bugzilla: http://mzl.la/1DOOBZt
| |
|
| |
| === Risks and Open Questions ===
| |
| * Old-FHR data through pipeline? Yes/No: [telliot]
| |
| * Deletes & legal policy [telliot]
| |
| * Security review [telliot]
| |
|
| |
|
| = Code = | | = Code = |