Mobile/Testing/12 04 13

From MozillaWiki
Jump to: navigation, search

Previous Action Items

Status reports

Dev team

Rel Eng

  • graydon has a spot on Ash to mess with Android 2.3 testing on EC2
  • Callek to deploy SUTAgent 1.20 today
  • xpcshell on panda was not running on all trunk-based trees and did not ride the train, being fixed in bug 909457
  • Blog post about Android Stats put up by Callek (see round table)

IT

A Team

  • tegra failure rate: [1.22%]
  • panda failure rate: [13.6%]
    • excluding retries: 2.03
  • Working on bug 937637 to get cpp unittests scheduled on trunk trees
  • Can't attend this week due to a conflict

x86 automation

  • All tests running hidden on Cedar, with some failures
  • Complete:
    • kvm enabled and deployed: bug 929048
    • chunks coalesced, sets coalesced: bug 935214
    • patched emulator deployed: bug 933918
    • S1 split into 2 sets and output no longer duplicated: bug 939823
    • new AVD definitions (Graydon's) deployed: bug 919812
  • Working on:
    • cleaning up remaining failures: bug 936226

Autophone

  • Temporarily disabled Mochitest-Canvas due to instability in some phones. Will enhance Autophone to be able to run different tests on different devices and reenable this week.
  • Nexus Devices
    • Getting 2 used Nexus 4. Try to locate 2012 Nexus 7 internally. When I have experience with the used devices will order new Nexus 5 and Nexus 7.
  • Improving Autophone to help in detecting regression ranges.
    • Fixed bug 943871 - Autophone - use http instead of ftp for build urls
    • Fixed bug 943873 - Autophone - support inbound archive
    • Filed bug 945390 - Autophone - support build selection via revision
    • Planning to upgrade flot to 0.8.2 and improve graphs and date display on phonedash.mozilla.org
  • Bugs
    • Found regression range for bug 916099 - Regression in "time to throbber stop" on Aug 28 2013
    • Plan to resolve INCOMPLETE bug 819984 - Regression in "time to throbber start" - Dec 5 and bug 825612 - Regression in "time to throbber start" and "time to throbber stop" - Dec 22 since they are outside inbound archive range and since we have other more important regressions to investigate.
    • Plan to file unfiled regressions with tight changesets from inbound archive. Removed obvious outliers from data and will file more recent regressions first.
    • Will also attempt to identify changesets which improve performance and comment in the associated bugs.

Eideticker

Round Table

We have awful wait times on Tegras

  • We currently test Android 2.2 on Tegras
    • However, only less than 2% of our users are using Android 2.2
    • graydon is helping us to have an Android 2.3 story (10.82%-14.22% of our users)
  • We currently test Armv6 builds (~2% of users) and no-ion (we don't ship this - only b2g18 branches)
    • I want to know if we can have a not-per-checkin model testing for Armv6
    • This would improve our current 2.2 testing on Tegras

I think we're OK with Pandas wrt to capacity, however, I want to point out that 4.1 has passed 4.0 in number of users.

Action Items