QA/TDAI/Meetings/RelEng And QA 2009-07-14
parent - all chrome tab has native widget run by child process
- 1 buildbot with mozmill (catlee)
- 2 js ref test
- 3 long running, non-deterministic runs
- 4 Breaking out Mochitest
- 5 code coverage
- 6 Change the way updates are verified/tested on the build side
- 7 Would automating updates reduce the QA
- 8 unittests/talos on release builds
- 9 Fennec buildbots QA Results server
buildbot with mozmill (catlee)
- virtualenv might work
- install deps into ve then deploy that directory structure directly into the slaves
- download zip unpack and run
- do we need c-setups?
- unit test slaves (vms)
- work on nightly first not per checkin
- the test deletion will destroy that directory for the ve
- --> buildbot patches (me, catlee to r)
- --> build the zip (in hg test repo) (me with catlee's help?)
- --> crash & frozen browsers (mikeal) (we should add frozen check to mozmill), add timeouts to buildbot, timeout until no output
js ref test
- runs exactly like ref test and crash test
- new suite
- runs within 10 mins for browser and shell tests
- how many environments to test?
- defaults only on tinderbox
- turn jit on by default in the shell since that is the standard
- Need to be able to run it concurrently
- We also have the ECMA script 5 tests coming own the pipe line, and they are in the same type of league.
- hopefully they will be much more modern
- going to investigate the framework for them.
- (catlee): need to be sure that the test do not depend on the build, so doin't use the make target use a python script to run it instead
- if the test requires a special type of build, then it must be treated specifically -
- (bc) there are some flags that would be nice to use.
- it might be something better done individually rather than on tinderbox (same for valgrind smc-check all too).
- Bob has a Dell Dual Quad core box to run ESx VMs.
- Set up this machine to a mirror environment to what the build team is using and that way we could set it up and it could be a test sandbox for running test VMs
- they will help us to set it up
long running, non-deterministic runs
- fuzzing, code coverage, valgrind, specialized build testing
- might have a non-build affiliated tinderbox only for testing and run on a scheduler
- But we need it to be able to be robust
- If it finds a problem you have a week old regression range and that is a mess
- got to see what we can do about chunking tests and running them in smaller bits so that they can be done concurrently.
- More value per build with more tests running on it (alice, good quote)
- As much as possible break things into small discreet chunks
Breaking out Mochitest
- We had to break layout/style and DOM tests down
- Murali has a patch that breaks it into chunks
- Two problems
- parallelize the tests, get the whole set of results back faster
- also solves the code coverage issue (which is a subset of it)
- they have a wrapper around runtests.py
What about using manifest for mochitest and xpcshell test
- that would solve the known failure issue that we have with fennec
- xpcshell generates a manifest but they aren't checked in and so no way to track known failures
- ( --> need <-- to figure out if we want to use manifests or not, catlee, joduinn, jmaher, ctalbert)
- jmaher will put the "manifest" file into a bug for releng
- need to split the test from the build
- consolidate the coverage info would be much better
Change the way updates are verified/tested on the build side
- currently we only have partials and compeletes from the previous version to the version to be released
- for ex: having partial/complete mars from 3.0.11 for testing, but it'd be nice to have 3.0.10 or 3.0.9
- if we had the more recent update mars across locales for the most recent versions that would be good.
- catlee: we could test a subset of locales from older versions
- update verify form 3.0.11 to 3.0.12 on windows took an hour and a half (approx)
- this could be easily parallelized as well.
- Build can do that. (joduinn)
- Currently the QA team does a bunch of manual checks and having the checks on the build side would help a lot.
- Probably should also do a major update automatic check since we are creating a major update mar for each release as well. Need to do that update verify as well. Always test from 18.104.22.168 to 3.0.12 (most current release).
- And this will also cover the 3.5.x track as well.
- Will follow up in the bug (todo - what is bug number )
Would automating updates reduce the QA
- if we could download it from mirrors, then yes.
- but do we test downloading from mirrors and applying the mar file and then ensure that it is correct.
- the automation doesn't actually downlod from the mirrors.
- Could this be done in Mozmill?
- Possibly, it would help us to detect mirror issues and AUS communication issues with the mirrors
- We need to figure out how fast we can get a release out the door.
- Signing was always the biggest chunk, but that going down now, is there anything else that can be automated to help that?
- (juan) the complete updates would be the best thing for us, as well as the mirror-download update
- the process breaks slightly with windows because of the signing process (they do a binary comparison on the files)
unittests/talos on release builds
- run unittests on incremental builds and not on the nightly and the release files.
- can install the built pieces as extensions...
- mac we need a packaging manifest for the tests as we don't want to package the tests into the dmg.
- --> xpcshell build issue on windows
- --> mac packaging
Fennec buildbots QA Results server
- We have a log parser and throws it up into CouchDB
- Need to remove buildbot from the machines themselves - want to run as little on the machines as possible
- Will have the results server which will be able to query by build ID
- The boxes just have to upload the results to the server
- --> Clint will be telling john ford about it who is about to re-invent the thing
- Aki is go to do this, we just need to work with him to get the scripts in it.
- We will roll this out to all unittests later
Might do this once a month...joduinn will work on this, and next time we will perhaps have the testdev and releng meetups at the same time.