Mobile/Testing/05 01 13: Difference between revisions
Jump to navigation
Jump to search
| (16 intermediate revisions by 8 users not shown) | |||
| Line 1: | Line 1: | ||
= Previous Action Items = | = Previous Action Items = | ||
* (jmaher) explain your round table items | * (jmaher) explain your round table items | ||
* (ctalbert) get kim a known good build | * <s>(ctalbert) get kim a known good build | ||
* (kim) run tests over the weekend | * (kim) run tests over the weekend</s> | ||
* (ctalbert) to email bad news | * (ctalbert) to email bad news | ||
* ( | * (kim) split out webgl from mochitest-1 | ||
* (wlach) to add stock and chome to new test | * (wlach) to add stock and chome to new test | ||
** Mostly done... see below | |||
* (blassey) follow up with karen to get 2.2 end of life plan | * (blassey) follow up with karen to get 2.2 end of life plan | ||
= Status reports = | = Status reports = | ||
== Dev team == | == Dev team == | ||
* Trying to eliminate {{bug|663657}} "2400 seconds without output" failures | |||
** sutAgent updated (thanks again :kmoir!) | |||
** {{bug|865944}} Use foreground activity instead of process name to determine if remote browser has terminated | |||
* bug 865006 Run more xpcshell tests on Android | |||
== Rel Eng == | == Rel Eng == | ||
* (kmoir) {{Bug|866228}} - deploy sutagent 1.17 to the tegras and pandas (done), {{bug|865677}} - rerun android panda tests on newly rewired chassis with fennec 19.0 (same results as production), working on {{bug|865443}} create webgl mochitest run for android | |||
* (Callek) {{Bug|861295}} Modify --testing-modules-dir for android xpcshell tests - landed | |||
== IT == | == IT == | ||
* Still working on a higher density chassis. Just waiting for the prototype chassis to be fabricated. | |||
== A Team == | == A Team == | ||
* tegra failure rate [[https://docs.google.com/spreadsheet/ccc?key=0ArS97F99-BEZdEpuZFdmSEhwWUVEQUgzZWFxRENoSlE&usp=sharing 11.8%]] | |||
* panda failure rate [[https://docs.google.com/spreadsheet/ccc?key=0ArS97F99-BEZdDhsZ3YwSjdralpmaXdIMjNJVGxjOFE&usp=sharing 13.2%]] | |||
== x86 automation == | == x86 automation == | ||
* I've now run through all of the test types, tracking in {{Bug|828571}} | |||
* Crashtests and JSRefTests seem stable, so I've filed a bug to get started on running this in Cedar | |||
** {{Bug|867377}} | |||
* Reftests have a small number of unexpected results (20) but should also be straight forward to get running | |||
* Mochitest and robocop have timeouts and larger number of unexpected results | |||
** Robocop needs to specify the user to the ActivityManager since the emulator images I'm using are 4.2 | |||
* Talos tests will run. | |||
** tsvg, ts_paint work ok | |||
** tp4m runs for a long time, then has problems parsing the output | |||
** haven't tried robocop based talos tests yet | |||
* I have been running some of the tests in parallel with two emulators running on my machine | |||
** In one case I got timeouts in both at the same time due to high CPU load | |||
* I'm planning to focus on getting mochitests running properly next | |||
== Autophone == | == Autophone == | ||
* [bc] Working on improving reproducibility of results in {{bug|866195}}. Will post test runs for the last several days with production phones to http://mrcote.info/phonedash_bc for comparison with previous production results at http://mrcote.info/phonedash later today. | |||
== Eideticker == | == Eideticker == | ||
* Comparative numbers for Chrome on pageload-post-startup test (forgot to generate for stock, will do later today) | |||
** http://eideticker.wrla.ch/#/samsung-gn/nytimes-load-poststartup/timetostableframe | |||
** Numbers indicate that pages are slow to load even after the browser has started. We are very much not competitive here. | |||
* Working on getting HTTP request times added to dash | |||
* Profiling support broke last week which meant no items added to galaxy nexus dash for a bit ({{bug|865915}}). Now fixed. | |||
= Round Table = | = Round Table = | ||
= Action Items = | = Action Items = | ||
* (jmaher) figure out why our tegra orange rate went up | |||
** one build from April 30 yielded a series of blue (retries) across most all tests | |||
** reftest 1-4 seem to have an orange and red for each chunk, this is different than previously. If trend continues, we will need to look into this more | |||
* (kim) split webgl tests out from mochitest-1 | |||
* (blassey) follow up with necko team on the new eideticker tests | |||
Latest revision as of 14:20, 2 May 2013
Previous Action Items
- (jmaher) explain your round table items
(ctalbert) get kim a known good build(kim) run tests over the weekend- (ctalbert) to email bad news
- (kim) split out webgl from mochitest-1
- (wlach) to add stock and chome to new test
- Mostly done... see below
- (blassey) follow up with karen to get 2.2 end of life plan
Status reports
Dev team
- Trying to eliminate bug 663657 "2400 seconds without output" failures
- sutAgent updated (thanks again :kmoir!)
- bug 865944 Use foreground activity instead of process name to determine if remote browser has terminated
- bug 865006 Run more xpcshell tests on Android
Rel Eng
- (kmoir) bug 866228 - deploy sutagent 1.17 to the tegras and pandas (done), bug 865677 - rerun android panda tests on newly rewired chassis with fennec 19.0 (same results as production), working on bug 865443 create webgl mochitest run for android
- (Callek) bug 861295 Modify --testing-modules-dir for android xpcshell tests - landed
IT
- Still working on a higher density chassis. Just waiting for the prototype chassis to be fabricated.
A Team
x86 automation
- I've now run through all of the test types, tracking in bug 828571
- Crashtests and JSRefTests seem stable, so I've filed a bug to get started on running this in Cedar
- Reftests have a small number of unexpected results (20) but should also be straight forward to get running
- Mochitest and robocop have timeouts and larger number of unexpected results
- Robocop needs to specify the user to the ActivityManager since the emulator images I'm using are 4.2
- Talos tests will run.
- tsvg, ts_paint work ok
- tp4m runs for a long time, then has problems parsing the output
- haven't tried robocop based talos tests yet
- I have been running some of the tests in parallel with two emulators running on my machine
- In one case I got timeouts in both at the same time due to high CPU load
- I'm planning to focus on getting mochitests running properly next
Autophone
- [bc] Working on improving reproducibility of results in bug 866195. Will post test runs for the last several days with production phones to http://mrcote.info/phonedash_bc for comparison with previous production results at http://mrcote.info/phonedash later today.
Eideticker
- Comparative numbers for Chrome on pageload-post-startup test (forgot to generate for stock, will do later today)
- http://eideticker.wrla.ch/#/samsung-gn/nytimes-load-poststartup/timetostableframe
- Numbers indicate that pages are slow to load even after the browser has started. We are very much not competitive here.
- Working on getting HTTP request times added to dash
- Profiling support broke last week which meant no items added to galaxy nexus dash for a bit (bug 865915). Now fixed.
Round Table
Action Items
- (jmaher) figure out why our tegra orange rate went up
- one build from April 30 yielded a series of blue (retries) across most all tests
- reftest 1-4 seem to have an orange and red for each chunk, this is different than previously. If trend continues, we will need to look into this more
- (kim) split webgl tests out from mochitest-1
- (blassey) follow up with necko team on the new eideticker tests