CloudServices/DataPipeline: Difference between revisions
Jump to navigation
Jump to search
(edit roadmap) |
m (edit roadmap) |
||
| Line 30: | Line 30: | ||
** 2015-05-19 Release | ** 2015-05-19 Release | ||
= | = Milestone Details = | ||
What sources and reporting are we committed to implementing/supporting for 2015? What are we not supporting? | What sources and reporting are we committed to implementing/supporting for 2015? What are we not supporting? | ||
== Q1 == | == Q1 == | ||
=== Ingestion | === Ingestion === | ||
* FHR+telemetry | * FHR+telemetry | ||
** [https://docs.google.com/a/mozilla.com/document/d/1K-H_6awM9OQgiixM73PPE-1xq7-djA5ECJNF-jWwLxk/edit#heading=h.o2uqf1tvmjzl FHR+Telemetry unified data storage options] | ** [https://docs.google.com/a/mozilla.com/document/d/1K-H_6awM9OQgiixM73PPE-1xq7-djA5ECJNF-jWwLxk/edit#heading=h.o2uqf1tvmjzl FHR+Telemetry unified data storage options] | ||
** [https://docs.google.com/document/d/1IGpzsYGi_sq3YFQDAPyKOkU_BKvXAC95fZYA2i4ceVs/edit FHR+Telemetry unification project kickoff] | ** [https://docs.google.com/document/d/1IGpzsYGi_sq3YFQDAPyKOkU_BKvXAC95fZYA2i4ceVs/edit FHR+Telemetry unification project kickoff] | ||
** Q1 use cases/queries TBD | |||
=== Storage/Reporting | |||
* BI/Data warehouse prototype | |||
* ICE telemetry data => kibana or sentry | |||
== Unscheduled == | == Unscheduled == | ||
* Cloud Services log files | * Cloud Services log files | ||
Revision as of 00:44, 7 January 2015
Overview
Cloud services infrastructure for ingesting data for analysis, monitoring and reporting. The pipeline is currently used for processing cloud services server logs. We're in the process of improving it to support desktop and device telemetry data.
Resources
- IRC channel: #datapipeline
- Pipeline technical proposal
- Reporting and monitoring overview
- Heka
Milestones
- Q4 2014: Telemetry data running through pipeline
- Server stack deploy in github ("opsified")
- Re-implement monitoring dashboards
- Q1 2015: Launch pipeline prototype
- Architecture decisions completed; production stack up and running
- Business Intelligence/Data Warehouse proof of concept implemented
- Ingestion process completed for FHR+telemetry (start collecting on 2015-02-23)
- Backprocessing from pipeline datastore implemented
- Pipeline runs in parallel to existing infrastructure; not yet source of truth
- Q2 2015: Pipeline officially supports business use cases
- Complete set of use cases tbd (most likely primarily FHR+telemetry use cases)
- Complete set of monitoring and reporting outputs tbd: dashboards, data warehouse, monitoring, self-service access to data
- FHR+telemetry hits full release 2015-05-19, handle full production load
- Q3 2015: Fill out monitoring and reporting capabilities; add sources and use cases
Related Dates/Schedules
- FHR+Telemetry client work
- Current plan: FF39 Nightly and uplifted to FF38. May not hit this schedule, but the pipeline needs to be ready
- 2015-02-23 Nightly
- 2015-05-19 Release
Milestone Details
What sources and reporting are we committed to implementing/supporting for 2015? What are we not supporting?
Q1
Ingestion
- FHR+telemetry
- FHR+Telemetry unified data storage options
- FHR+Telemetry unification project kickoff
- Q1 use cases/queries TBD
=== Storage/Reporting
- BI/Data warehouse prototype
- ICE telemetry data => kibana or sentry
Unscheduled
- Cloud Services log files
- FxA related servers
- Loop server
- Sync, Tokenserver
- New sources
- Chronicle
- Github events
- Build system metrics (mach)
- Hg.mozilla.org server logs
- Old sources (may never go through pipeline)
- FHR
- Telemetry
- ADI/Blocklist Ping
Reporting/Outputs
- Existing Reporting
- Elasticsearch/Kibana
- Custom dashboards (Loop & FxA)
- New Reporting
- BI/data warehouse (TBD)
- Spark (?)
- 2015 Dashboards (architecture TBD)