Metrics/Telemetry 2013-02-07

Attendees: Daniel Einspanjer, Annie Elliott, Taras Glek, Lawrence Mandel

Status

BIV-70 (bug 776714) and BIV-69 (bug 776716) Adding descriptions to Telemetry dashboard

• done

BIV-224 (bug 810034) Telemetry Dashboard session timeout is too quick

• Still testing

METRICS-1356 (bug 778809) Simple measures

• Taras missed sending e-mail with target measures, measures now added in Jira and bug
• This will be implemented in ETL

BIV-114 (bug 707280) Need simple queries

• Harsha looking at initial work to prototype it out, sidetracked with spike in telemetry submissions METRICS-1366

METRICS-1366 Telemetry submission spike has been taking a lot of time

• 5 mil to 21 mil submissions in a single day is bad for a number of reasons
  • not a steady trend of submissions wondering how valid is data coming in, no evidence of this type of fluctuation within a weekday
  • all channels went up but release channel went up 12 million as it's a bigger channel
  • not equipped to handle this amount of traffic, strain put on resources (elastic search)
  • every few days see a spike of submissions
  • haven't found a pattern
  • Harsha working full time to attempt to fingerprint the documents looking for indication of source
  • comparison of idle daily vs saved sessions and trend is holding stable
  • can deal with capacity side but how can we trust this data if it's behaving in a way that we don't understand?
• would like to get someone on dev side to look at submissions from client side to try and find more useful information
  • Taras: there have been no changes in date range that Daniel provided, nothing else to act on without more guidance from metrics
• Taras requests that Metrics look at the IPs from the submissions

METRICS-1363 (bug 821403) Make raw shutdown times available

• No progress

Process

Lawrence: Engineering didn't know that all work was stopped due to the Telemetry submission investigation. Annie: We should have communicated the work stoppage. Telemetry has 1 FTE allocated, which is why all work stopped while Harsha was 100% on the submission spike investigation.

Iteration 2 planning

• All work is stopped until the submission spike has been resolved in METRICS-1366
  • Taras requests daily updates on this issue until it is resolved, please cc Taras Glek, Vladan Djeric, Nathan Froyd
• BIV-110 not ready to start
• BIV-85 and BIV-224 should be done in the next two weeks