FirefoxOS/TestingInfrastructureImprovements
Meeting notes July 28
Intermittent failures
We still have problems with intermittent failures. In particular, we still have a few infrastructure issues which is causing intermittent failures in *all* Gij tests.
Some of these infrastructure issues are due to known problems that are being worked on, Aus is on top of that. But the big one is that we still have a problem with the socket connection randomly disconnecting. This is a really tough problem and people have worked on it for years with no progress.
These intermittent issues are somewhat less of a problem because we now run tests three times and only if all three fail will we consider it a failure. Unfortunately this means that bugs in our product which only happen intermittently are no longer caught.
We also have some intermittent issues in individual tests, but unclear how much. One proposed solution was to write helper functions which help with writing stable tests. The "Page object pattern" used by calendar tests were pointed out as a good example.
Action items
- Fix the known causes of infrastructure issues (Aus is already working on this).
- Catch the socket disconnect issue in rr.
Get more info when a test fails
When a test fails, we would like to collect
Gecko log (bug 1188648) Error stack Screen shot crash dump in case of crashes
Action items
- Add the above to Gij infrastructure
Which platforms do we want to maintain
We currently have too many development platforms: mulet/gaia-in-nightly/b2g-desktop/simulator/emulator-ICS/emulator-L/B2GDroid/on-device.
We should narrow it down to: Mulet, Emulator-L, B2GDroid (not yet ready), and on-device.
B2G-desktop, gaia-in-nightly and simulator should be removed from the tree and from test automation. But first we need to make sure that all tests that are currently running on B2G-desktop are also running in Mulet.
We also want to kill |DEBUG=1 make|. But first we need to improve the "shift-f5 reload" development flow. Possibly by hooking it in to a super-fast rebuild step.
Action items
- Ensure that all tests that are run in B2G-desktop is also run in Mulet (Lissy is working on it)
- Kill B2G-desktop, gaia-in-nightly and simulator
- Stand up Emulator-L (Hsin-Yi is working on it)
- Figure out how to make "shift-f5 reload" work (Tim is thinking about this, no one is working on this yet)
What tests do we need to run on-device/in emulator
Running on emulator is really slow. In part because of arm. In part because we restart emulator. We would like to run most tests in emulator (except unit tests). However we need Emulator-x86 for that to be viable.
We have tests for Bluetooth/RIL/Wifi/NFC which run on emulator, but they are not reporting to treeherder because we only have emulator-ics on treeherder and it doesn't support those features.
We are running perf tests on-device, but no one knew if they are reporting to raptor.
Action items
- Stand up Emulator-x86 and Emulator-L on treeherder.
- Enable Bluetooth/RIL/Wifi/NFC tests on treeherder.
- Make sure perf tests that are running on hardware report to raptor.