Mobile/Testing/10 14 15

Previous Action Items

• snorp - investigate when to disable the webappstartup test.
• wlach - file bug to get deterministic behavior from A/B testing.

Status reports

Rel Eng

• Running chunked Android debug (jsreftest/crashtest) tests on Try to generate data for SETA bug 1201236 https://treeherder.mozilla.org/#/jobs?repo=try&revision=ad89a095e9c5. Will enable reftests later to generate data as well while keeping pending counts low.

A Team

• android-2-3-armv7-api9 Total failure rate: [0.86%]
  • last week 3.11%
  • total failure excluding retries 0.81%
• android-4-3-armv7-api11 Total failure rate: [2.02%]
  • last week 1.68%
  • total failure excluding retries 1.45%

Data fetched for: 2015-10-06 09:54 — 2015-10-12 22:18

• [gbrown] Working on:
  • bug 1087791 Add |mach reftest| and |mach crashtest| for mobile/android
  • bug 1197716 Package fonts separately for reftests

Autophone

• Phonedash vs. Perfherder

Perfherder will only support Autophone Talos. We will revisit improving Phonedash for S1S2/Webapp performance tests. Since any noise in the device's behavior or the framework tend to increase the measurements, will investigate using the "Greatest Lower Bound" of a measurement set as an indicator of the "Best Measurement" possible which will hopefully be a better indicator of regressions than the mean.

• Regressions
  • Bug 1213175 - Autophone - Throbber start regression Android 2.3 Remote Tests 2015-09-28 on mozilla-inbound
  • Bug 1209653 reversed a framework caused improvement which occurred during the 2015-08-21 deployment of Bug 1156812 - [mozdevice] Support specifying alternate log buffers when retrieving or clearing logcat
  • Bug 1212995 - Autophone - Throbber stop regression 2015-10-07 on fx-team (Fixed by backout)
  • Will investigate trend of death by a 1000 paper cuts
• Deployments
  • Bug 1209796 - Autophone - 2015-10-04 deployment
    • Bug 1208782 - Autophone - do not attempt to get response json during Treeherder submission error if response is None
    • Bug 1209647 - Autophone - eliminate startup check for network connectivity
    • Bug 1209651 - Autophone - do not allow logcat device error to prevent setup_job initialization
    • Bug 1209653 - Autophone - after clearing logcat, specifying -b main can hang
    • Bug 1209675 - Autophone - Logcat should use PhoneTest loggerdeco
    • Bug 1209691 - Autophone - handle incorrect logcat dates emitted by devices.
    • Bug 1211227 - Autophone - work around webapp Crash Reporter moving dmp files
    • Bug 1211228 - Autophone - prefer "Start proc" over Gecko related messages when determining start time
    • Bug 1211229 - Autophone - reduce stderrp_accept to improve reproducibility
  • 2015-10-05 Deployment
    • Bug 1211655 - Autophone - Ensure adb.py methods propagate root to other called methods
  • Bug 1212757 - Autophone - 2015-10-09 deployment
    • Bug 1212758 - Autophone - ADBDevice.mkdir - reproduce behavior of mkdir -p
    • Bug 1212766 - Autophone - make webapp profiles.ini readable and handle failure to obtain the remote profile path
  • Bug 1214110 - Autophone - 2015-10-13 deployment
    • Bug 1211252 - Autophone - Unit Test not (un)installing robocop.apk
    • Bug 1213866 - Autophone - adb.py - add a logcat cleared message to the logcat whenever it is cleared so it is never completely empty
    • Bug 1213868 - Autophone - catch device logcat exceptions in PhoneTest.setup_job
    • Bug 1213879 - Autophone - work harder to create clean profile directory during PhoneTest install_profile
    • Bug 1213922 - Autophone - call power_on after reboots and before s1s2 iterations
  • 2015-10-13 Deployment
    • Bug 1214273 - autophone - 2015-10-13: deploy nexus-7 for try only and tp4m/tsvgx-remote on mozilla-inbound
    • Bug 1192069 - Autophone should submit data to perfherder

Round Table

• tcheck on talos is very noisy, trying to figure out what to do with it in bug 1213032
  • in addition it is the test with the highest chance of failure on autophone- when we switch devices it is 50/50 if it will work or not