QA/TDAI/Meetings/RelEng And QA 2009-07-14

From MozillaWiki
< QA‎ | TDAI‎ | Meetings
Jump to: navigation, search

parent - all chrome tab has native widget run by child process

buildbot with mozmill (catlee)

  • virtualenv might work
  • install deps into ve then deploy that directory structure directly into the slaves
    • download zip unpack and run
    • do we need c-setups?
    • unit test slaves (vms)
    • work on nightly first not per checkin
    • the test deletion will destroy that directory for the ve
    • --> buildbot patches (me, catlee to r)
    • --> build the zip (in hg test repo) (me with catlee's help?)
    • --> crash & frozen browsers (mikeal) (we should add frozen check to mozmill), add timeouts to buildbot, timeout until no output

js ref test

  • runs exactly like ref test and crash test
  • new suite
  • runs within 10 mins for browser and shell tests
  • how many environments to test?
    • defaults only on tinderbox
    • turn jit on by default in the shell since that is the standard
  • Need to be able to run it concurrently
  • We also have the ECMA script 5 tests coming own the pipe line, and they are in the same type of league.
    • hopefully they will be much more modern
    • going to investigate the framework for them.
    • (catlee): need to be sure that the test do not depend on the build, so doin't use the make target use a python script to run it instead
    • if the test requires a special type of build, then it must be treated specifically -
    • (bc) there are some flags that would be nice to use.
    • it might be something better done individually rather than on tinderbox (same for valgrind smc-check all too).
  • Bob has a Dell Dual Quad core box to run ESx VMs.
    • Set up this machine to a mirror environment to what the build team is using and that way we could set it up and it could be a test sandbox for running test VMs
    • they will help us to set it up

long running, non-deterministic runs

  • fuzzing, code coverage, valgrind, specialized build testing
  • might have a non-build affiliated tinderbox only for testing and run on a scheduler
  • But we need it to be able to be robust
  • If it finds a problem you have a week old regression range and that is a mess
  • got to see what we can do about chunking tests and running them in smaller bits so that they can be done concurrently.
  • More value per build with more tests running on it (alice, good quote)
  • As much as possible break things into small discreet chunks

Breaking out Mochitest

  • We had to break layout/style and DOM tests down
  • Murali has a patch that breaks it into chunks
  • Two problems
    • parallelize the tests, get the whole set of results back faster
    • also solves the code coverage issue (which is a subset of it)
  • they have a wrapper around

What about using manifest for mochitest and xpcshell test

  • that would solve the known failure issue that we have with fennec
  • xpcshell generates a manifest but they aren't checked in and so no way to track known failures
  • ( --> need <-- to figure out if we want to use manifests or not, catlee, joduinn, jmaher, ctalbert)
  • jmaher will put the "manifest" file into a bug for releng

code coverage

  • need to split the test from the build
  • consolidate the coverage info would be much better

Change the way updates are verified/tested on the build side

  • currently we only have partials and compeletes from the previous version to the version to be released
    • for ex: having partial/complete mars from 3.0.11 for testing, but it'd be nice to have 3.0.10 or 3.0.9
    • if we had the more recent update mars across locales for the most recent versions that would be good.
    • catlee: we could test a subset of locales from older versions
    • update verify form 3.0.11 to 3.0.12 on windows took an hour and a half (approx)
    • this could be easily parallelized as well.
    • Build can do that. (joduinn)
    • Currently the QA team does a bunch of manual checks and having the checks on the build side would help a lot.
    • Probably should also do a major update automatic check since we are creating a major update mar for each release as well. Need to do that update verify as well. Always test from to 3.0.12 (most current release).
    • And this will also cover the 3.5.x track as well.
    • Will follow up in the bug (todo - what is bug number )

Would automating updates reduce the QA

  • if we could download it from mirrors, then yes.
  • but do we test downloading from mirrors and applying the mar file and then ensure that it is correct.
  • the automation doesn't actually downlod from the mirrors.
  • Could this be done in Mozmill?
    • Possibly, it would help us to detect mirror issues and AUS communication issues with the mirrors
    • We need to figure out how fast we can get a release out the door.
    • Signing was always the biggest chunk, but that going down now, is there anything else that can be automated to help that?
      • (juan) the complete updates would be the best thing for us, as well as the mirror-download update
      • the process breaks slightly with windows because of the signing process (they do a binary comparison on the files)

unittests/talos on release builds

  • run unittests on incremental builds and not on the nightly and the release files.
  • can install the built pieces as extensions...
  • mac we need a packaging manifest for the tests as we don't want to package the tests into the dmg.
  • --> xpcshell build issue on windows
  • --> mac packaging

Fennec buildbots QA Results server

  • We have a log parser and throws it up into CouchDB
  • Need to remove buildbot from the machines themselves - want to run as little on the machines as possible
  • Will have the results server which will be able to query by build ID
  • The boxes just have to upload the results to the server
  • --> Clint will be telling john ford about it who is about to re-invent the thing
  • Aki is go to do this, we just need to work with him to get the scripts in it.
  • We will roll this out to all unittests later

Might do this once a month...joduinn will work on this, and next time we will perhaps have the testdev and releng meetups at the same time.