CloudServices/DataPipeline: Difference between revisions

Jump to navigation Jump to search

CloudServices/DataPipeline (view source)

Revision as of 19:52, 22 January 2015

1,066 bytes removed ,  22 January 2015
Add bugzilla list
(section getting turned into bugs)
(Add bugzilla list)
Line 34: Line 34:


= Work Queue =
= Work Queue =
To Do/Doing categories need to be logged in bugzilla.
Tracking tasks in bugzilla: http://mzl.la/1DOOBZt
 
=== Risks and Open Questions ===
=== Risks and Open Questions ===
* Old-FHR data through pipeline? Yes/No: [telliot]
* Old-FHR data through pipeline? Yes/No: [telliot]
* Deletes & legal policy [telliot, mreid to provide cost estimate]
* Deletes & legal policy [telliot]
 
=== To Do ===
* Q4 telemetry: (re) implement telemetry monitoring dashboards [?]
* Q1 BI: define schema for data warehouse (talk to jjensen) [kparlante]
** should use multiple data sources
* Q1 BI: write filter for data warehouse [trink]
* Q1 BI: signal & schedule loading of data warehouse [mreid]
* Q1 BI: redshift output [trink]
* Q1 BI: setup domo and/or tableau to look at mysql or csv or whatever is easy [?]
* Q1: Data format spec [kparlante, trink]
** JSON schema, specifically for FHR+telemetry, also anticipate other sources
* Q1: implement best guess at per user sampling [trink]
** follow up with saptarshi for more complex algorithm
=== Doing ===
* Opsify stack [whd]
* Q4 telemetry: Send telemetry data through the pipeline [mreid]
* Q4 telemetry: Larger payloads (32MB) for telemetry [trink]
* risk mitigation: Estimate cost of “full scan” DWH query [mreid]
* risk mitigation: Estimate cost of single DWH delete [mreid]
=== Done ===
* Parallelize sandbox filters (eg FHRSearch) [trink]
* Enable Lua JIT [trink]


= Archive =
= Archive =
* [https://docs.google.com/a/mozilla.com/document/d/1CTazW99zBK5K40f-fgSyTPw9IXgmFYjQmNhzxTT9Tts/edit?usp=sharing post workweek roadmap]
* [https://docs.google.com/a/mozilla.com/document/d/1CTazW99zBK5K40f-fgSyTPw9IXgmFYjQmNhzxTT9Tts/edit?usp=sharing post workweek roadmap]
* [https://id.etherpad.mozilla.org/data-team old etherpad]
* [https://id.etherpad.mozilla.org/data-team old etherpad]
Kparlante
Confirmed users
539

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1049800"

Navigation menu