Previous Action Items

• [jmaher] figure out who is a try server expert to make compare talos via try work, follow up with carljm
• [jmaher] talos branch names and operating systems to match tbpl, etc...
• [jmaher] find win7 info for christina

Goals

a production system slurping up data from talos deployed

• • Owner: Jeads
  • Next steps: https://www.pivotaltracker.com/epic/show/107545

ability on this production system to accept data from non-talos projects

• • Owner: Jeads
  • Next steps: https://www.pivotaltracker.com/epic/show/109709
  • This goal might need some refinement. We will be able to accept data across multiple projects. Not sure if the UI will work out of the box for every project, also not sure if the webservice interface really meets all of the different project use cases. I'm currently working on the stone ridge data, we'll see how it goes.

the UI we largely have now

• • Owner: Jeads
  • Next steps: https://www.pivotaltracker.com/epic/show/107671
  • Putting UI work on hold until we finish goal number two completely. Would rather deploy 100% on one goal and get it done right than spread too thin. This task must be completed before release: https://www.pivotaltracker.com/story/show/28968601

Metrics for determining page-centric "did my test pass or not" for at least one suite dromaeo and ts_paint

• • Owner: Christina/Jeads
  • Next steps: Iterate on the development data. We're going to go ahead with the ~26 "good" pages in tp5. It's going to be awhile before we have representative data for dromaeo and wading in on ts_paint could introduce a whole new set of variables that send us into the talos briar patch.
    • [jeads] Build SQL queries to pull out representative data for the ~26 "good" pages in tp5
    • [christina and jeads] Develop the input data set for analysis in R
    • [christina and jeads] Identify an outlier that is outside an allowable range
    • [christina] Develop an allowable range for a test result for each of the ~26 pages
    • [christina] Determine a score for a changeset across the entire test suite. This could be as simple as 0 or 1 for each page, that gets added up across all pages and platforms. So 24/26 would indicate 24 pages were in the allowable range 2 were not for one platform. We will not know the best way to do this until we wade into the data.
    • [everyone] Determine a way to convert the test suite score into a pass, fail, or undetermined result for a given changeset.

a stage tbpl system mocking the Yes/No/Maybe interaction

• • Owner: jmaher/jhammel - eta: after we are side by side staging
  • Next steps: assert we can mock up tbpl to gather data post test completion, will mozilla be ok with that?
    • tbpl can do this: POST {'changeset': 31415926535, 'testname': 'dromaeo', 'platform': 'win7'} http://10.8.73.29/views/api/test_results
    • and recieve this: {'result':'PASS', 'tests': {'passed': 98, 'failed': 0, 'maybe': 2}, 'details': [{'k0s.org': {'actual': 2.71, 'expected': [2.75, 3.21]}, {'askjeads.com': {'actual': 99, 'expected': [88, 98]}]}

Fix tp5 pageset to not have network

• • Owner: jmaher/jhammel/chmanchester
  • http://people.mozilla.org/~jmaher/sxs/tp5n/sxs.html

Convert all page cycle extension based tests to row-major style.

• • Owner: jmaher
  • tests: tdhtml, tsvg, tsvg_opacity, tsspider, tscroll, a11y
  • http://people.mozilla.org/~jmaher/sxs/tp5n/sxs.html

Round Table

• developer meeting, we need to have it

Action Items

• ttest data: ensure we drop the first 5, find a bad changset, view the python code
• determine failed page threshold. If we have 1 page regressed, do we call the push failed?
• [jmaher] talos branch names and operating systems to match tbpl, etc...