Mobile/Testing/09 30 15

From MozillaWiki
Jump to: navigation, search

Previous Action Items

  • kmoir to determine the number of pandas currently used to run talos tests.
    • We have 380 pandas in production. We run about 400 ~talos jobs a day on them out of a total of ~2300 panda jobs a day. In other words 17% job current panda jobs are on talos. So in theory if we just run talos jobs this would require 65 pandas to maintain our current wait times.
  • wlach to meet with jmaher/bc concerning perfherder and talos-on-autophone requirements.
    • DONE. We're going to experiment with running talos against a nexus7 reporting to perfherder. Depending on results + feedback from others, we'll figure what kind of devices we need and at what scale to replace the pandas (or if such a thing even makes sense).

Status reports

Dev team

Rel Eng

  • Met with Amazon rep today + device farm representative regarding our testing requirements. Notes are here https://releng.etherpad.mozilla.org/device-farm They said we could work with us on a trial basis if we are interested. Will talk to others regarding if this is a priority for Q4.
  • Debugging why android debug chunking is failing on cedar bug 1201236

A Team

  • android-2-3-armv7-api9 Total failure rate: [1.94%]
    • last week 3.4%
    • total failure excluding retries 1.7%
  • android-4-3-armv7-api11 Total failure rate: [1.91%]
    • last week 3.43%
    • total failure excluding retries 0.94%

Data fetched for: 2015-09-23 00:08 — 2015-09-29 22:56

Autophone

  • bug 1207836 - Autophone - 2015-09-23 deployment
    • bug 1205864 - Autophone - phonetest.py:Logcat collects duplicate messages
    • bug 1206954 - Autophone - better handle failures to submit results to PhoneDash
  • bug 1209796 - Autophone - next deployment (In progress)
    • bug 1205836 - Autophone - investigate orange for remote nytimes s1s2
    • bug 1208782 - Autophone - do not attempt to get response json during Treeherder submission error if response is None
    • bug 1209647 - Autophone - eliminate startup check for network connectivity
    • bug 1209651 - Autophone - do not allow logcat device error to prevent setup_job initialization
    • bug 1209653 - Autophone - after clearing logcat, specifying -b main can hang
    • bug 1209675 - Autophone - Logcat should use PhoneTest loggerdeco
    • bug 1209691 - Autophone - handle incorrect logcat dates emitted by devices.
    • Working on issue with webappstartup test crash reporting.

Perfherder

  • Work to ease landing perf changes from autophone to land shortly (bug 1175295)

Round Table

Action Items

  • snorp - investigate when to disable the webappstartup test.
  • wlach - file bug to get deterministic behavior from A/B testing.