parent - all chrome tab has native widget run by child process

buildbot with mozmill (catlee)

• virtualenv might work
• install deps into ve then deploy that directory structure directly into the slaves
  • download zip unpack and run
  • do we need c-setups?
  • unit test slaves (vms)
  • work on nightly first not per checkin
  • the test deletion will destroy that directory for the ve
  • --> buildbot patches (me, catlee to r)
  • --> build the zip (in hg test repo) (me with catlee's help?)
  • --> crash & frozen browsers (mikeal) (we should add frozen check to mozmill), add timeouts to buildbot, timeout until no output

js ref test

• runs exactly like ref test and crash test
• new suite
• runs within 10 mins for browser and shell tests
• how many environments to test?
  • defaults only on tinderbox
  • turn jit on by default in the shell since that is the standard
• Need to be able to run it concurrently
• We also have the ECMA script 5 tests coming own the pipe line, and they are in the same type of league.
  • hopefully they will be much more modern
  • going to investigate the framework for them.
  • (catlee): need to be sure that the test do not depend on the build, so doin't use the make target use a python script to run it instead
  • if the test requires a special type of build, then it must be treated specifically -
  • (bc) there are some flags that would be nice to use.
  • it might be something better done individually rather than on tinderbox (same for valgrind smc-check all too).

• Bob has a Dell Dual Quad core box to run ESx VMs.
  • Set up this machine to a mirror environment to what the build team is using and that way we could set it up and it could be a test sandbox for running test VMs
  • they will help us to set it up

long running, non-deterministic runs

• fuzzing, code coverage, valgrind, specialized build testing
• might have a non-build affiliated tinderbox only for testing and run on a scheduler
• But we need it to be able to be robust
• If it finds a problem you have a week old regression range and that is a mess
• got to see what we can do about chunking tests and running them in smaller bits so that they can be done concurrently.
• More value per build with more tests running on it (alice, good quote)
• As much as possible break things into small discreet chunks

Breaking out Mochitest

• We had to break layout/style and DOM tests down
• Murali has a patch that breaks it into chunks
• Two problems
  • parallelize the tests, get the whole set of results back faster
  • also solves the code coverage issue (which is a subset of it)
• they have a wrapper around runtests.py

What about using manifest for mochitest and xpcshell test

• that would solve the known failure issue that we have with fennec
• xpcshell generates a manifest but they aren't checked in and so no way to track known failures
• ( --> need <-- to figure out if we want to use manifests or not, catlee, joduinn, jmaher, ctalbert)
• jmaher will put the "manifest" file into a bug for releng

code coverage

• need to split the test from the build
• consolidate the coverage info would be much better

Change the way updates are verified/tested on the build side

• currently we only have partials and compeletes from the previous version to the version to be released
  • for ex: having partial/complete mars from 3.0.11 for testing, but it'd be nice to have 3.0.10 or 3.0.9
  • if we had the more recent update mars across locales for the most recent versions that would be good.
  • catlee: we could test a subset of locales from older versions
  • update verify form 3.0.11 to 3.0.12 on windows took an hour and a half (approx)
  • this could be easily parallelized as well.
  • Build can do that. (joduinn)
  • Currently the QA team does a bunch of manual checks and having the checks on the build side would help a lot.
  • Probably should also do a major update automatic check since we are creating a major update mar for each release as well. Need to do that update verify as well. Always test from 2.0.0.20 to 3.0.12 (most current release).
  • And this will also cover the 3.5.x track as well.
  • Will follow up in the bug (todo - what is bug number )

Would automating updates reduce the QA

• if we could download it from mirrors, then yes.
• but do we test downloading from mirrors and applying the mar file and then ensure that it is correct.
• the automation doesn't actually downlod from the mirrors.
• Could this be done in Mozmill?
  • Possibly, it would help us to detect mirror issues and AUS communication issues with the mirrors
  • We need to figure out how fast we can get a release out the door.
  • Signing was always the biggest chunk, but that going down now, is there anything else that can be automated to help that?
    • (juan) the complete updates would be the best thing for us, as well as the mirror-download update
    • the process breaks slightly with windows because of the signing process (they do a binary comparison on the files)

unittests/talos on release builds

• run unittests on incremental builds and not on the nightly and the release files.
• can install the built pieces as extensions...
• mac we need a packaging manifest for the tests as we don't want to package the tests into the dmg.
• --> xpcshell build issue on windows
• --> mac packaging

Fennec buildbots QA Results server

• We have a log parser and throws it up into CouchDB
• Need to remove buildbot from the machines themselves - want to run as little on the machines as possible
• Will have the results server which will be able to query by build ID
• The boxes just have to upload the results to the server
• --> Clint will be telling john ford about it who is about to re-invent the thing
• Aki is go to do this, we just need to work with him to get the scripts in it.
• We will roll this out to all unittests later

Might do this once a month...joduinn will work on this, and next time we will perhaps have the testdev and releng meetups at the same time.