CloudServices/DataPipeline

Overview

Cloud services infrastructure for ingesting data for analysis, monitoring and reporting. The pipeline is currently used for processing cloud services server logs. We're in the process of improving it to support desktop and device telemetry data.

Resources

• IRC channel: #datapipeline
• Pipeline technical proposal
• Reporting and monitoring overview
• Heka

Milestones

• Q4 2014: Telemetry data running through pipeline
  • Server stack deploy in github ("opsified")
  • Re-implement monitoring dashboards
• Q1 2015: Launch pipeline prototype
  • Architecture decisions completed; production stack up and running
  • Business Intelligence/Data Warehouse proof of concept implemented
  • Ingestion process completed for FHR+telemetry (start collecting on 2015-02-23)
  • Backprocessing from pipeline datastore implemented
  • Pipeline runs in parallel to existing infrastructure; not yet source of truth
• Q2 2015: Pipeline officially supports business use cases
  • lua filters completed for a set of use cases, including FHR+telemetry
  • dashboards, data warehouse, monitoring, self-service access to data, etc.
  • FHR+telemetry hits full release 2015-05-19, handle full production load

Related Dates/Schedules

• FHR+Telemetry client work
  • Current plan: FF39 Nightly and uplifted to FF38. May not hit this schedule, but the pipeline needs to be ready
  • 2015-02-23 Nightly
  • 2015-05-19 Release