Mobile/Testing/01 08 14

From MozillaWiki
Jump to: navigation, search

Previous Action Items

Status reports

Dev team

Rel Eng

  • Moved tegra staging monday -- runs look good
  • Tegras down for 48 hours this weekend (starts with TCW at 9am PT on sat)
    • Hope is for them to be up as of Taiwan Monday Morning, and hopefully without any issues.
    • TCW includes move of the bm-remote servers, used for talos on Android, including for pandas.

IT

A Team

  • tegra failure rate: [1.13%]
  • panda failure rate: [12.51%]
    • excluding retries: 0.26
  • bug 929447 Enable panda reftests, ride trains
    • patch reviewed and ready to go
    • snatched failure from the jaws of success (new oranges on Cedar this morning)

2.3 emulator automation

x86 automation

Autophone

  • Nexus Mochitest-Canvas testing
  • Comparing autophone s1s2 to eideticker, leads me to believe that autophone s1s2 adds value and should be kept but it should be improved to better meet our needs.
  • We should evaluate the test files in s1s2 and to see if they are appropriate.
  • Are the throbberstart, throbberstop messages meaningful to measure? Should we be measuring something else?
  • Experience dealing with older regressions leads me to believe that large regressions are easier to spot even when grouped together in merge changesets but that multiple small regressions are much harder to spot and isolate especially when they occur in the same merge. These small regressions accumulate over time. See http://gbrownmozilla.wordpress.com/2014/01/07/firefox-for-android-performance-measures-2013-in-review/ and especially bug 916099.
    • bug 904088 Autophone - s1s2 - Regression in "time to throbber stop" on Aug 9, 2013
    • bug 909380 Autophone - s1s2 - Regression in "time to throbber stop" on Aug 24 2013
    • bug 916099 Autophone - s1s2 - Regression in "time to throbber stop" on Aug 28 2013
  • If we agree that Autophone s1s2 should be kept, then the next steps should be:
    • bug 957312 - Autophone - add repository to db schema.
    • bug 957360 - Autophone - change phoneid to have hardware as most significant.
    • Should run tests on fx-team, mozilla-central, mozilla-inbound.
      • This will help identify the original regressing changesets on the original repo without having to wait for the merge and without having to deal with a huge merge changeset.
      • Developers should track autophone for their builds on the appropriate repo for regressions.
      • We may have to reduce the number of iterations from 16 and possibly partition repos across devices.
      • Improve bisection processes for isolating regressing changesets. http://inbound-archive.pub.build.mozilla.org/ will be helpful for triaging regressions after May 2013 for mozilla-inbound, Aug 2013 for mozilla-central and Sept 2013 for fx-team.
      • Being able to hide revisions which have been backed out may be helpful in reducing the visual noise in the graphs.
    • We should use tracking bug 953342 so we can easily find issues which have already been reported.

Eideticker

  • Not much new to report other than the usual maintenance work
  • "Should run startup / load tests against Chrome weekly on Galaxy Nexus" (bug 943500) still planned, just haven't had time yet.

Round Table

Action Items

  • blassey to report back on his meeting this Friday, re: future of Pandas, etc.
  • bc ran out of time for autophone...he'll finish up next week
Retrieved from "https://wiki.mozilla.org/index.php?title=Mobile/Testing/01_08_14&oldid=870322"