Mobile/Testing/06 19 13: Difference between revisions
Jump to navigation
Jump to search
(Created page with "= Previous Action Items = * <s>(jmaher) File bug to get access to foopy and some pandas or tegras to further investigate {{bug|877779}}</s> * <s>(dminor) File bug to run Pand...") |
|||
| (20 intermediate revisions by 9 users not shown) | |||
| Line 3: | Line 3: | ||
* <s>(jmaher) File bug to get access to foopy and some pandas or tegras to further investigate {{bug|877779}}</s> | * <s>(jmaher) File bug to get access to foopy and some pandas or tegras to further investigate {{bug|877779}}</s> | ||
* <s>(dminor) File bug to run Panda reftests on Cedar</s> | * <s>(dminor) File bug to run Panda reftests on Cedar</s> | ||
* <s>(kmoir) Document process for getting rogue pandas and tegras removed by buildduty</s> It's easier just to ask who's on buildduty to open a bug but here | * <s>(kmoir) Document process for getting rogue pandas and tegras removed by buildduty</s> It's easier just to ask who's on buildduty to open a bug but here's a [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Release%20Engineering%3A%20Machine%20Management link to our bugzilla machine management component] where you can just ask for a specific device to be disabled. | ||
= Status reports = | = Status reports = | ||
== Dev team == | == Dev team == | ||
* Working on | |||
** {{bug|883539}} Frequent Android 4.0 rc2 retries: "Remote Device Error: unable to connect to %s after %s attempts" % (self.host, retryLimit) | |||
** {{bug|882932}} Remote mochitests should invoke the httpd.js from hostutils, not from the changeset | |||
== Rel Eng == | == Rel Eng == | ||
* kmoir will land final changes for non-talos android mozharness/mozpool tests early next week when she returns from PTO. Tests are green in staging and patches have been reviewed. {{bug|829211}} | |||
== IT == | == IT == | ||
* Still working on a higher density chassis. Waiting for DCOps to rack in Data Center. - https://www.dropbox.com/sc/j5fpbwwd7ki2knb/fPQplsqiya | |||
* Tegra move - mrz is working on a proposal to present to the team for review | |||
** Draft https://mana.mozilla.org/wiki/display/~mzeier@mozilla.com/Tegra+-+Migration+out+of+650+Castro | |||
** There has been email back and forth on this, nothing is finalized yet. | |||
== A Team == | == A Team == | ||
tegra failure rate [[https://docs.google.com/spreadsheet/ccc?key=0ArS97F99-BEZdEhyZThMSkhrcW13V2dVVXpqdDZuMXc&usp=sharing 8.68%]] | |||
* tp, r3, j1, rc2, m3 <- all >15% failure rate | |||
panda failure rate [[https://docs.google.com/spreadsheet/ccc?key=0ArS97F99-BEZdDZ1YnZRWVlFeFg4YmZKLUd4RTF0clE&usp=sharing 19.62%]] | |||
* rpr, ts, tpn, rc1, rc2, m7, m2 <- all >15% failure rate | |||
top 5 orange factor android bugs: | |||
* {{bug|807230}} - Intermittent DMError: Automation Error: Timeout in command {ls,ps,isdir,mkdr} | |||
** dminor working on this, most common on pandas, but across all mochitest/reftests. NOTE: not seen on talos!!! | |||
* {{bug|711725}} - Tegras and Pandas disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames) ... | |||
** callek attempted fix, no luck, large majority on tp4m talos test, rarely on other suites | |||
* {{bug|874972}} - Intermittent Android retry "Remote Device Error: updateApp() call failed - exiting" | |||
** a newer issue, tegra only! Only occurs on a handful of naughty tegras. | |||
* {{bug|817024}} - Tegras/Pandas disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames): ... | |||
** similar to other one, but specific to inside of test case, mostly on tp4m | |||
* {{bug|845162}} - Intermittent Android timeout in test_replay_metadata.html,test_playback_rate.html... | |||
** m2 on pandas, possibly a root cause in a test case early on causing OOM or other state changes? | |||
Panda Reftests (dminor) | |||
* Got agreement on syntax - will retest patch today and put up for another review once it looks good locally | |||
== x86 automation == | == x86 automation == | ||
== Autophone == | == Autophone == | ||
* New production site: http://phonedash.mozilla.org | |||
* Locally increasing devicemanagerSUT.reboot_settling_time to work around slow sdcard performance on nexus ones. | |||
* No progress on getting reftest unittests running. | |||
== Eideticker == | == Eideticker == | ||
* LG G2X seems to have been dying on the "imgur" test. Worried it might be memory exhaustion or something. Filed {{bug|884907}} to investigate. | |||
= Round Table = | = Round Table = | ||
* (gbrown) What can we do about {{bug|883539}}? Can we update the Panda image? | |||
** (jmaher) we could update the image (possibly with a new watcher?), is there other hacks we could make as well? | |||
* (jmaher)watcher update? | |||
= Action Items = | = Action Items = | ||
* Joduinn going to follow up with bmoss about his ideas on the tegra move decision | |||
* Jmaher and team going to keep digging into the failure rate spike, potentially rolling back the recent sutagent change | |||
* Bc will have an update on autophone reftest testing next week | |||
* callek has on his plate this week to write up a concise plan to update the watcher across the board. | |||
* gbrown - will start on making a [https://wiki.mozilla.org/Auto-tools/Projects/Pandaboard_Setup new image] for the pandas to address the [https://bugzilla.mozilla.org/show_bug.cgi?id=883539 retry bug], and then we can use mozpool to distribute it. | |||
Latest revision as of 17:48, 19 June 2013
Previous Action Items
(jmaher) File bug to get access to foopy and some pandas or tegras to further investigate bug 877779(dminor) File bug to run Panda reftests on Cedar(kmoir) Document process for getting rogue pandas and tegras removed by builddutyIt's easier just to ask who's on buildduty to open a bug but here's a link to our bugzilla machine management component where you can just ask for a specific device to be disabled.
Status reports
Dev team
- Working on
- bug 883539 Frequent Android 4.0 rc2 retries: "Remote Device Error: unable to connect to %s after %s attempts" % (self.host, retryLimit)
- bug 882932 Remote mochitests should invoke the httpd.js from hostutils, not from the changeset
Rel Eng
- kmoir will land final changes for non-talos android mozharness/mozpool tests early next week when she returns from PTO. Tests are green in staging and patches have been reviewed. bug 829211
IT
- Still working on a higher density chassis. Waiting for DCOps to rack in Data Center. - https://www.dropbox.com/sc/j5fpbwwd7ki2knb/fPQplsqiya
- Tegra move - mrz is working on a proposal to present to the team for review
- Draft https://mana.mozilla.org/wiki/display/~mzeier@mozilla.com/Tegra+-+Migration+out+of+650+Castro
- There has been email back and forth on this, nothing is finalized yet.
A Team
tegra failure rate [8.68%]
- tp, r3, j1, rc2, m3 <- all >15% failure rate
panda failure rate [19.62%]
- rpr, ts, tpn, rc1, rc2, m7, m2 <- all >15% failure rate
top 5 orange factor android bugs:
- bug 807230 - Intermittent DMError: Automation Error: Timeout in command {ls,ps,isdir,mkdr}
- dminor working on this, most common on pandas, but across all mochitest/reftests. NOTE: not seen on talos!!!
- bug 711725 - Tegras and Pandas disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames) ...
- callek attempted fix, no luck, large majority on tp4m talos test, rarely on other suites
- bug 874972 - Intermittent Android retry "Remote Device Error: updateApp() call failed - exiting"
- a newer issue, tegra only! Only occurs on a handful of naughty tegras.
- bug 817024 - Tegras/Pandas disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames): ...
- similar to other one, but specific to inside of test case, mostly on tp4m
- bug 845162 - Intermittent Android timeout in test_replay_metadata.html,test_playback_rate.html...
- m2 on pandas, possibly a root cause in a test case early on causing OOM or other state changes?
Panda Reftests (dminor)
- Got agreement on syntax - will retest patch today and put up for another review once it looks good locally
x86 automation
Autophone
- New production site: http://phonedash.mozilla.org
- Locally increasing devicemanagerSUT.reboot_settling_time to work around slow sdcard performance on nexus ones.
- No progress on getting reftest unittests running.
Eideticker
- LG G2X seems to have been dying on the "imgur" test. Worried it might be memory exhaustion or something. Filed bug 884907 to investigate.
Round Table
- (gbrown) What can we do about bug 883539? Can we update the Panda image?
- (jmaher) we could update the image (possibly with a new watcher?), is there other hacks we could make as well?
- (jmaher)watcher update?
Action Items
- Joduinn going to follow up with bmoss about his ideas on the tegra move decision
- Jmaher and team going to keep digging into the failure rate spike, potentially rolling back the recent sutagent change
- Bc will have an update on autophone reftest testing next week
- callek has on his plate this week to write up a concise plan to update the watcher across the board.
- gbrown - will start on making a new image for the pandas to address the retry bug, and then we can use mozpool to distribute it.