Mobile/Testing/03 21 12: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Created page with "= Previous Action Items = * Joel to take care of Ts-moz-afterpaint * finkle to give joel a list of perf tests he wants to write * ctalbert to flash new ES board with build and se...")
 
 
(18 intermediate revisions by 7 users not shown)
Line 1: Line 1:
= Previous Action Items =
= Previous Action Items =
* Joel to take care of Ts-moz-afterpaint
* [carry-over] Joel to take care of Ts-moz-afterpaint
* finkle to give joel a list of perf tests he wants to write
* <strike>[DONE][carry-over] finkle to give joel a list of perf tests he wants to write</strike>
* ctalbert to flash new ES board with build and see what new problems we find and what ends up fixed
* [carry-over] ctalbert to flash new ES board with build and see what new problems we find and what ends up fixed
* joduinn to figure out plan for mobile mozharness moving forward since aki and armenzg are with full hands


= Status reports =
= Status reports =
== Dev team ==
== Dev team ==
* {{bug|735461}}: Robocop not using -profile argument consistently
* Robocop: PixelTests are failing
* {{bug|734456}}: Reftest/Crashtest suites C1, R2, R3 hidden
* Reftests: still investigating
** Overlap with {{bug|732498}}, etc.?
* Fennec -profile flag: bug 735461
 
* Make it easy to use Robocop to add some UI talos performance tests for various UI pieces:
** Time to open the Awesome Screen
** Time to load basic Awesome Screen results
** Time to open Tabs List with X tabs open
** Time to display core UI
** Time to display about:home


== Rel Eng ==
== Rel Eng ==
* {{bug|723667}} - talos robocop
* update SUT agent was deployed last Friday
** sut agent upgrade patches being reviewed and soon to be landed
** talos robocop did not work after the upgrade (known)
*** armenzg verifying with jmaher that sut agent upgrade does not cause regressions to other suites
** jmaher has asked me to test https://tbpl.mozilla.org/?tree=Try&rev=b12a718cb1f1 which I will today
** talos robocop jobs are still going red
* tegras' screen resolution deployed around 9:30-10:30 AM PDT today
*** jmaher got it working manually when its first run went red
** no expected fallout from it
*** armenzg has triggered another 3 jobs of each to see if anything changes
* I believe a lot of "disconnection in a non-clean manner" are due to have all foopies/tegras talking to only two masters and 16 foopies (foopy[07-20,22-24]) during high peak hours
** add ganglia to the foopies
** get stats from the DB to correlate theory
** move 1/3 of foopies to a 3rd active master
** the end result might be:
*** fixed!
*** we need Linux foopies
*** move to foopy-less setup
*** we disconnect a % of all tegra boards to reduce #boards per #foopy (this would reduce # purples but might increase wait times)
*** we need more masters
* work on talos.json
** I can start as soon as I run through the high load/purples investigation


== A Team ==
== A Team ==
Robocop:
* not copying fresh profile over each test saves us about 1.5 minutes per test, or almost 20 minutes for a full run
** patch in progress, eta friday
* when launching via sutagent, we seem to take 30 seconds longer to run a test vs launching with adb.  A future area for investigation to save time


Reftests:
* Eideticker: Dashboard up with daily updates: http://wrla.ch/eideticker/dashboard/#/canvas
* in looking at the reftests that require larger resolution (I believe there are 10 of them), we do not run them on android and should consider not adjusting our resolution.  This will save us 10 minutes per reftest job or 180 minutes of machine time per checkin.
** Canvas framerate benchmark working.
** side benefit is we will have more memory for fennec and might reduce our hangs or random failures (not proven or tested much)
** Checkerboarding/scrolling benchmark under development, currently using taskjs.org as a baseline. May need additional/more representative samples to get a better idea of where we're at.
* {{bug|732498}} - need somebody from development to help drive/fix the reftest failures between xul/native/e10s and android in general


Talos:
* General:  
* {{bug|732206}} - maple ts and txul are busted, ts appears to be green the last few runs
** investigated speeding up reftests, deployed
* new talos.zip is being used, now we can iterate faster on talos changes
** investigated speeding up robocop, backed up
* need to do ts-mozafterpaint


Upcoming work:
* Talos
* xpcshell (need new sutagent rolled out- in progress)
** With patch (which breaks other things), we should be able to run robocop on talos now
* powermeter


== S1/S2 Automation ==
== S1/S2 Automation ==
* public facing graphs will be available by end of the week
* We are [http://mrcote.info/phonedash/#/rawfennecstartup public, finally!]
* Looking at the numbers, you can see maple landing. Was looking at this yesterday and am not seeing anything this week, will be investigating.


== Eideticker ==
== Mobile Crash ==
* automated tests are running
[[File:CrashStats_2012-03-19.png]]
* Screenshot and data from 2012-03-19 at 2.25.08 PM PDT
Top crashes:
* see [https://wiki.mozilla.org/Platform/2012-03-11#Mobile_2 Stability Report 2012-03-20]


== Crash Stats ==
Note :
* Socorro/Breakpad work:
** {{bug|726385}} Please create skiplist implementation for the Java field
** {{bug|726693}} Some java crashes are not being mapped correctly to bugs in bugzilla even though the crash signature fields have the "signatures" listed
** {{bug|672606}} - Aggregate numbers and topcrashes for Nightly and Aurora channels based on build ID date instead of crash date
** {{Bug|727234}} - Crash Report [@ EMPTY: no crashing thread identified; corrupt dump ]
** {{Bug|727286}} - Incorrect messaging when failing to submit a crash report


== Panda Boards ==
* ADU for Aurora is low.  We will see spikes for Aurora.
* from ctalbert: "panda still needs a new build, going to rebuild it today"
* Java field needs to be implemented for XUL Fennec
* skiplist for touchbadmemory to be implemented on wed.
* Beta channel has no ADU/crash reported due to {{Bug|738316}} - No crashes found for Fennec 12.0b1 - wrong channel set
* Crash criteria : get less than 10 % per 100 adus
 
== Panda boards ==
* {{bug|731669}}: the panda board ES has the same reboot issue
** have we asked in panda forums?
* Built new build for board without resolution changes, trying to flash onto sdcard - caught in wierd unflash state.


=== Carry over info ===
=== Carry over info ===
Line 67: Line 84:
* {{bug|725544}} - Android 4.0 tracking bug
* {{bug|725544}} - Android 4.0 tracking bug
* [https://wiki.mozilla.org/Auto-tools/Projects/Pandaboard_Setup Pandaboard Setup docs]
* [https://wiki.mozilla.org/Auto-tools/Projects/Pandaboard_Setup Pandaboard Setup docs]
== QA Robocop Coverage ==
* https://docs.google.com/spreadsheet/ccc?key=0AhE7m4JB2j6tdDJBT2dlbVJwUk9PSy1RbHo4WVNiUGc#gid=2
** Starter sheet
== Crash Stats ==
[[File:CrashStats_2012-03-12.png]]
* Screenshot and data from 3/12/2012 11:11 PM PDT
Top crashes:
* see [https://wiki.mozilla.org/Platform/2012-03-11#Mobile_2 Stability Report 2012-03-11]
Note :
* Socorro/Breakpad work:
** {{bug|726385}} Please create skiplist implementation for the Java field
** {{bug|726693}} Some java crashes are not being mapped correctly to bugs in bugzilla even though the crash signature fields have the "signatures" listed
** {{bug|672606}} - Aggregate numbers and topcrashes for Nightly and Aurora channels based on build ID date instead of crash date
** {{Bug|727234}} - Crash Report [@ EMPTY: no crashing thread identified; corrupt dump ]
** {{Bug|727286}} - Incorrect messaging when failing to submit a crash report
* ADU went up on March 7th a little bit due to missing pings; this was resolved on the 8th.
* 9th had changes to the libdvm so that they should now be reporting proper Java crashes.


= Round Table =
= Round Table =
* We have to improve our purple/red VS orange/green ratio
* (coop) Callek on short-term contract with releng
** These bugs have been filed for long and are at the top of WOO
** tasked with fixing top WoO-related releng bugs, many of which are Android-related
** {{Bug|724738}} - Intermittent Android mochitest and mochitest-robocop timeout
** {{Bug|686084}} - Intermittent Tegra "Cleanup Device failed" from "command timed out: 1200 seconds without output, process killed by signal 9"
** {{Bug|687116}} - Intermittent "unrecognized output format" running Talos on Android
** {{Bug|722166}} - Android tests intermittently just don't start


= Action Items =
= Action Items =
* [carry-over] Joel to take care of Ts-moz-afterpaint
* [carry-over] finkle to give joel a list of perf tests he wants to write
* [carry-over] ctalbert to flash new ES board with build and see what new problems we find and what ends up fixed
* joduinn to figure out plan for mobile mozharness moving forward since aki and armenzg are with full hands

Latest revision as of 17:09, 22 March 2012

Previous Action Items

  • [carry-over] Joel to take care of Ts-moz-afterpaint
  • [DONE][carry-over] finkle to give joel a list of perf tests he wants to write
  • [carry-over] ctalbert to flash new ES board with build and see what new problems we find and what ends up fixed
  • joduinn to figure out plan for mobile mozharness moving forward since aki and armenzg are with full hands

Status reports

Dev team

  • Robocop: PixelTests are failing
  • Reftests: still investigating
  • Fennec -profile flag: bug 735461

Rel Eng

  • update SUT agent was deployed last Friday
  • tegras' screen resolution deployed around 9:30-10:30 AM PDT today
    • no expected fallout from it
  • I believe a lot of "disconnection in a non-clean manner" are due to have all foopies/tegras talking to only two masters and 16 foopies (foopy[07-20,22-24]) during high peak hours
    • add ganglia to the foopies
    • get stats from the DB to correlate theory
    • move 1/3 of foopies to a 3rd active master
    • the end result might be:
      • fixed!
      • we need Linux foopies
      • move to foopy-less setup
      • we disconnect a % of all tegra boards to reduce #boards per #foopy (this would reduce # purples but might increase wait times)
      • we need more masters
  • work on talos.json
    • I can start as soon as I run through the high load/purples investigation

A Team

  • Eideticker: Dashboard up with daily updates: http://wrla.ch/eideticker/dashboard/#/canvas
    • Canvas framerate benchmark working.
    • Checkerboarding/scrolling benchmark under development, currently using taskjs.org as a baseline. May need additional/more representative samples to get a better idea of where we're at.
  • General:
    • investigated speeding up reftests, deployed
    • investigated speeding up robocop, backed up
  • Talos
    • With patch (which breaks other things), we should be able to run robocop on talos now

S1/S2 Automation

  • We are public, finally!
  • Looking at the numbers, you can see maple landing. Was looking at this yesterday and am not seeing anything this week, will be investigating.

Mobile Crash

CrashStats 2012-03-19.png

  • Screenshot and data from 2012-03-19 at 2.25.08 PM PDT

Top crashes:

Note :

  • Socorro/Breakpad work:
    • bug 726385 Please create skiplist implementation for the Java field
    • bug 726693 Some java crashes are not being mapped correctly to bugs in bugzilla even though the crash signature fields have the "signatures" listed
    • bug 672606 - Aggregate numbers and topcrashes for Nightly and Aurora channels based on build ID date instead of crash date
    • bug 727234 - Crash Report [@ EMPTY: no crashing thread identified; corrupt dump ]
    • bug 727286 - Incorrect messaging when failing to submit a crash report
  • ADU for Aurora is low. We will see spikes for Aurora.
  • Java field needs to be implemented for XUL Fennec
  • skiplist for touchbadmemory to be implemented on wed.
  • Beta channel has no ADU/crash reported due to bug 738316 - No crashes found for Fennec 12.0b1 - wrong channel set
  • Crash criteria : get less than 10 % per 100 adus

Panda boards

  • bug 731669: the panda board ES has the same reboot issue
    • have we asked in panda forums?
  • Built new build for board without resolution changes, trying to flash onto sdcard - caught in wierd unflash state.

Carry over info

concerns

  • IT is concerned about not having remote re-imaging
    • jake has ideas to try once he has time to use pxe
    • this is not a high priority right now
  • the boards sometimes become unstable (much more than the tegras)
    • we have to remember that we are using tip of ICS and will take time to settle
  • armenzg: do we have an alternative to the panda boards?
    • blassey: no fallback plan
  • bug 725544 - Android 4.0 tracking bug
  • Pandaboard Setup docs

Round Table

  • (coop) Callek on short-term contract with releng
    • tasked with fixing top WoO-related releng bugs, many of which are Android-related

Action Items