Mobile/Testing/09 30 15: Difference between revisions
Jump to navigation
Jump to search
(Created page with "= Previous Action Items = = Status reports = == Dev team == == Rel Eng == == A Team == <!-- http://ouija.allizom.org/wiki_update.html --> == Autophone == == Perfherder =...") |
|||
| (10 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
= Previous Action Items = | = Previous Action Items = | ||
* kmoir to determine the number of pandas currently used to run talos tests. | |||
** We have 380 pandas in production. We run about 400 ~talos jobs a day on them out of a total of ~2300 panda jobs a day. In other words 17% job current panda jobs are on talos. So in theory if we just run talos jobs this would require 65 pandas to maintain our current wait times. | |||
* wlach to meet with jmaher/bc concerning perfherder and talos-on-autophone requirements. | |||
** DONE. We're going to experiment with running talos against a nexus7 reporting to perfherder. Depending on results + feedback from others, we'll figure what kind of devices we need and at what scale to replace the pandas (or if such a thing even makes sense). | |||
= Status reports = | = Status reports = | ||
| Line 6: | Line 10: | ||
== Rel Eng == | == Rel Eng == | ||
* Met with Amazon rep today + device farm representative regarding our testing requirements. Notes are here https://releng.etherpad.mozilla.org/device-farm They said we could work with us on a trial basis if we are interested. Will talk to others regarding if this is a priority for Q4. | |||
* Debugging why android debug chunking is failing on cedar {{bug|1201236}} | |||
== A Team == | == A Team == | ||
<!-- http://ouija.allizom.org/wiki_update.html --> | <!-- http://ouija.allizom.org/wiki_update.html --> | ||
* android-2-3-armv7-api9 Total failure rate: [[http://ouija.allizom.org/android_failures.html?platform=android-2-3-armv7-api9 1.94%]] | |||
** last week 3.4% | |||
** total failure excluding retries 1.7% | |||
* android-4-3-armv7-api11 Total failure rate: [[http://ouija.allizom.org/android_failures.html?platform=android-4-3-armv7-api11 1.91%]] | |||
** last week 3.43% | |||
** total failure excluding retries 0.94% | |||
Data fetched for: 2015-09-23 00:08 — 2015-09-29 22:56 | |||
== Autophone == | == Autophone == | ||
* {{Bug|1207836}} - Autophone - 2015-09-23 deployment | |||
** {{Bug|1205864}} - Autophone - phonetest.py:Logcat collects duplicate messages | |||
** {{Bug|1206954}} - Autophone - better handle failures to submit results to PhoneDash | |||
* {{Bug|1209796}} - Autophone - next deployment (In progress) | |||
** {{Bug|1205836}} - Autophone - investigate orange for remote nytimes s1s2 | |||
** {{Bug|1208782}} - Autophone - do not attempt to get response json during Treeherder submission error if response is None | |||
** {{Bug|1209647}} - Autophone - eliminate startup check for network connectivity | |||
** {{Bug|1209651}} - Autophone - do not allow logcat device error to prevent setup_job initialization | |||
** {{Bug|1209653}} - Autophone - after clearing logcat, specifying -b main can hang | |||
** {{Bug|1209675}} - Autophone - Logcat should use PhoneTest loggerdeco | |||
** {{Bug|1209691}} - Autophone - handle incorrect logcat dates emitted by devices. | |||
** Working on issue with webappstartup test crash reporting. | |||
== Perfherder == | == Perfherder == | ||
* Work to ease landing perf changes from autophone to land shortly ({{bug|1175295}}) | |||
= Round Table = | = Round Table = | ||
= Action Items = | = Action Items = | ||
* snorp - investigate when to disable the webappstartup test. | |||
* wlach - file bug to get deterministic behavior from A/B testing. | |||
Latest revision as of 17:37, 30 September 2015
Previous Action Items
- kmoir to determine the number of pandas currently used to run talos tests.
- We have 380 pandas in production. We run about 400 ~talos jobs a day on them out of a total of ~2300 panda jobs a day. In other words 17% job current panda jobs are on talos. So in theory if we just run talos jobs this would require 65 pandas to maintain our current wait times.
- wlach to meet with jmaher/bc concerning perfherder and talos-on-autophone requirements.
- DONE. We're going to experiment with running talos against a nexus7 reporting to perfherder. Depending on results + feedback from others, we'll figure what kind of devices we need and at what scale to replace the pandas (or if such a thing even makes sense).
Status reports
Dev team
Rel Eng
- Met with Amazon rep today + device farm representative regarding our testing requirements. Notes are here https://releng.etherpad.mozilla.org/device-farm They said we could work with us on a trial basis if we are interested. Will talk to others regarding if this is a priority for Q4.
- Debugging why android debug chunking is failing on cedar bug 1201236
A Team
- android-2-3-armv7-api9 Total failure rate: [1.94%]
- last week 3.4%
- total failure excluding retries 1.7%
- android-4-3-armv7-api11 Total failure rate: [1.91%]
- last week 3.43%
- total failure excluding retries 0.94%
Data fetched for: 2015-09-23 00:08 — 2015-09-29 22:56
Autophone
- bug 1207836 - Autophone - 2015-09-23 deployment
- bug 1205864 - Autophone - phonetest.py:Logcat collects duplicate messages
- bug 1206954 - Autophone - better handle failures to submit results to PhoneDash
- bug 1209796 - Autophone - next deployment (In progress)
- bug 1205836 - Autophone - investigate orange for remote nytimes s1s2
- bug 1208782 - Autophone - do not attempt to get response json during Treeherder submission error if response is None
- bug 1209647 - Autophone - eliminate startup check for network connectivity
- bug 1209651 - Autophone - do not allow logcat device error to prevent setup_job initialization
- bug 1209653 - Autophone - after clearing logcat, specifying -b main can hang
- bug 1209675 - Autophone - Logcat should use PhoneTest loggerdeco
- bug 1209691 - Autophone - handle incorrect logcat dates emitted by devices.
- Working on issue with webappstartup test crash reporting.
Perfherder
- Work to ease landing perf changes from autophone to land shortly (bug 1175295)
Round Table
Action Items
- snorp - investigate when to disable the webappstartup test.
- wlach - file bug to get deterministic behavior from A/B testing.