B2G/QA/RTL/RTLImageCompare
RTL Imagecompare Tests
Objective
This wiki explains the use of Imagecompare tool to reduce the manual testing effort for RTL implementation on Firefox OS device. (currently Flame)
Imagecompare tool takes screenshots of the displayed screen on the phone, and later compares it to the captured images of the subsequent runs. This is ideal for verifying whether there is visual bug on the UI.
Imagecompare Tool
For more information and setup information, please visit this page.
Running on Jenkins
Currently, the jenkins job is created on MTV lab, scheduled to run daily. Because only one device (with SIM + SD Card) is allocated, it would overload the queue if we run it for every new nightly build.
Job locations are as follows:
To check the results, click the latest build from the build history box on the left, click Build Artifacts link, and go to screenshots/mismatches to view the failures.
Running Locally
RTL Imagecompare tests are located in /gaia/tests/python/gaia-ui-tests/gaiatest/tests/graphics/RTL folder in Gaia repo, and can be run locally after setting up the Imagecompare framework. (as described in the link mentioned in the parent) The command to run are: To obtain the reference image, execute:
gaiatest --address=localhost:2828 --testvars=gaiatest/b2g-7.json --type b2g --restart --locale=[ar|fr|it] --store-reference-image gaiatest/tests/graphics/RTL/manifest.ini
After this reference run is completed, verify the images are proper by examining the screenshots in the reference_images folder within gaia-ui-tests.
Finally, run the following command to run comparisons against the reference images.
gaiatest --address=localhost:2828 --testvars=gaiatest/b2g-7.json --type b2g --restart --locale=[ar|fr|it] gaiatest/tests/graphics/RTL/manifest.ini
Investigating Failures
If the failures occur, (and they will occur- this does not mean that there is a RTL bug) go to /mismatches folder within /screenshots folder to see what kind of discrepancies are there. Most of them are red herrings, such as:
- Date/Time
- List of available wifi access points
- List of available bluetooth devices
- Graphical (image related) mismatches due to timing of the execution
Ignore above types of issues, and look for the failures that seem directly related to the RTL/l10n layout. If they are found, a bug should be raised with the diff image as the evidence.
If the tests fail for any reason other than image mismatch, then the code has changed, and our test script needs to be updated.
Current Coverage
The latest set of reference images are in:
Click (all files in zip) link below to download the entire set of files.
Note that currently only the Settings app is covered, with following limitations:
- Not all menus in Cellular & Data -> SIM 1 -> APN Settings are explored, since they all give out same submenus.
- Not all ... buttons in Sound -> Manage Tones are explored, since they all give out same submenus.
- In Date & Time, Date and Time selection dialog is not explored.
- In Date & Time, Region and City selection dialog is not explored.
- In Language, the language selection dialog is not explored.
- In Keyboards -> Built-in Keyboards, the list of installed keyboards are not explored.
- In Keyboards -> Select Keyboards, the list of built-in keyboards are not explored.
- In Add-ons, only the first add on is tapped to check its content.
- In App Permissions, only the Camera app is selected to check its content.
- Developer menu is not covered