CloudServices/DataPipeline: Difference between revisions
Jump to navigation
Jump to search
(Changes to milestones) |
m (Changed milestone wording) |
||
| Line 13: | Line 13: | ||
** Architecture nailed down, code up and running | ** Architecture nailed down, code up and running | ||
** Business Intelligence/Data Warehouse proof of concept up and running | ** Business Intelligence/Data Warehouse proof of concept up and running | ||
** | ** Ingestion process completed for FHR+telemetry (start collecting on 2015-02-23) | ||
** Backprocessing from pipeline datastore implemented | ** Backprocessing from pipeline datastore implemented | ||
* '''Q2 2015''': Pipeline officially supports real use cases | * '''Q2 2015''': Pipeline officially supports real use cases | ||
** lua filters completed for a set of use cases, including FHR+telemetry | ** lua filters completed for a set of use cases, including FHR+telemetry | ||
** dashboards, data warehouse, monitoring, self-service access to data, etc. | ** dashboards, data warehouse, monitoring, self-service access to data, etc. | ||
** FHR+telemetry hits full release 2015-05-19, | ** FHR+telemetry hits full release 2015-05-19, handle full production load | ||
= Related Dates/Schedules = | = Related Dates/Schedules = | ||
Revision as of 19:33, 6 January 2015
Overview
Cloud services infrastructure for ingesting data for analysis, monitoring and reporting. The pipeline is currently used for processing cloud services server logs. We're in the process of improving it to support desktop and device telemetry data.
Resources
- IRC channel: #datapipeline
- Pipeline technical proposal
- Reporting and monitoring overview
- Heka
Milestones
- Q4 2014: Telemetry data running through pipeline
- Q1 2015: Launch pipeline prototype
- Architecture nailed down, code up and running
- Business Intelligence/Data Warehouse proof of concept up and running
- Ingestion process completed for FHR+telemetry (start collecting on 2015-02-23)
- Backprocessing from pipeline datastore implemented
- Q2 2015: Pipeline officially supports real use cases
- lua filters completed for a set of use cases, including FHR+telemetry
- dashboards, data warehouse, monitoring, self-service access to data, etc.
- FHR+telemetry hits full release 2015-05-19, handle full production load
Related Dates/Schedules
- FHR+Telemetry data in FF39 Nightly and uplifted to FF38. (May not hit this date, but we plan to be ready)
- 2015-02-23 Nightly
- 2015-05-19 Release