Mobile/Fennec Maemkit
From MozillaWiki
< Mobile
Maemkit Toolset
NEW: Download the initial drop of [maemkit-chunked].
Until we get a lot of changes backported into the build system (makefiles, test running scripts, etc...) we will have a temporary test framework that wraps around the build system and test scripts to run all tests on maemo. We will call this the maemkit.
Here are a list of things the maemkit needs to do:
- Package the tests
- Ted has resolved this with 'make package-tests'
- copy to device/unpack (leave manual for now)
- run tests in a chunked method
- mochitest - run_tests_chunked.py
- test 1 directory at a time
- splits known large directories into smaller chunks
- chrometest - run_chrome_chunked.py
- make _tests/testing/chrome.bak
- copies 1 directory at a time from chrome.bak to chrome
- reftest - reftest_chunked.py
- parse manifest, run 1 dir at a time
- for large dir, split into multiple manifests
- crashtest - reftest_chunked.py
- parse manifest, run 1 dir at a time
- mochitest - run_tests_chunked.py
- concat logs into a common log format
- mochitest - run script to parse each chunk and cat together with summary
- chrometest - run script to parse each chunk and cat together with summary
- reftest - run script to remove file:///<rootdir> tags and cat together
- crashtest - cat log files together
- copy logs off device (leave manual for now)
- report results with new failures highlighted
- a system to keep track of known failures (json data provider)
- a system to show results of tests including historical data
I propose writing a master script and 1 website to deal with this:
- maemkit-chunked.py - initial draft [here]
- run a test (or all) as chunked
- reads a config file to determine
- what directories to ignore, split
- how many directories/tests to run
- which directories/tests to run (if we are running in parallel)
- create a log file that is compatible for comparison purposes
- maemkit-viewer website
- provider on known failures
- provides quick summary of tests
- provides in depth view of tests
- provides access to download logs
- TODO: determine a method for delivering failed test cases and related info to viewer. ctalbert suggests using json and mikeal has a module written for mozmill we might be able to use