Compatibility/System Addon/Automated Testing
Running the automated tests for site interventions
The easiest way to currently run the tests is to use a local build of Firefox on a desktop computer (artifact builds are recommended to save a lot of otherwise-wasted time).
Almost all of the automated tests run on their own in bulk. A desktop build of Firefox will test the desktop site interventions, while an Android build will start an emulator and test the Android interventions.
It is recommended to re-direct the output to a logfile.
# run desktop/laptop tests
./mach bootstrap # choose desktop artifact
./mach build
./mach test-interventions --headless | tee desktop.log
# run Android-specific tests (on an emulator)
./mach bootstrap # choose android artifact
./mach build
./mach test-interventions --headless | tee android.log
However, there are some tests which require a specific platform, VPN, or other conditions like logins, or they will be skipped (which should be mentioned in the resulting log):
Tests requiring Linux
# samsung_com.py
./mach test-interventions --headless --bug=1799968
Tests requiring Windows or MacOS
# saxoinvestor.fr
./mach test-interventions --headless --bug=1719859
Tests requiring Linux or Windows
# livescience.com
./mach test-interventions --headless --bug=1650317
Tests requiring Linux or MacOS
# bankofamerica.py.com
./mach test-interventions --headless --bug=1472075
Tests requiring a VPN connection to the USA
# directv.com.co
./mach test-interventions --headless --bug=1610344
# stream.directv.com
./mach test-interventions --headless --bug=1577519
Tests requiring VPN connections elsewhere
(to be determined/confirmed)
Tests requiring a login
# missingmail.usps.com
# note this test can take several minutes to run, as the site is very slow
./mach test-interventions --headless --bug=1644830 --config=logins.json
# Steam Community
# note that Steam requires two-factor authentication, so you will be asked
# to provide a code twice by the test as it runs.
./mach test-interventions --headless --bug=1570108 --do2fa --config=logins.json
Example logins.json file
The logins.json looks like this, only with actual passwords:
{
"1570108": {"username":"steam_username", "password":"steam_password"},
"1644830": {"username":"usps_username", "password":"usps_password"}
}
Note that the file must be valid JSON, so of course care should be taken with characters like quotes and brackets and commas. For example, if a password itself contains a double-quote, it must be escaped with a backslash:
"password": "password with a \" double quote"
Interpreting the results
The logfile will capture two summaries of which tests FAILED and were SKIPPED, once with the intervention active (to see if it is still working) and once with it disabled (to see if it is still even necessary, or if the site issues has been fixed).
For instance, here is one such summary:
=========================== short test summary info ============================ FAILED testing/webcompat/interventions/tests/test_1799994_vivobarefoot_com.py::test_enabled[with_interventions] FAILED testing/webcompat/interventions/tests/test_1830752_afisha_ru.py::test_enabled[with_interventions] FAILED testing/webcompat/interventions/tests/test_1836182_www_flatsatshadowglen_com.py::test_enabled[with_interventions] SKIPPED [1] testing/webcompat/interventions/tests/test_1448747_discountcoffee_co_uk.py:12: Bug #1448747 skipped on platform (mac) SKIPPED [1] testing/webcompat/interventions/tests/test_1448747_dylantalkstone_com.py:12: Bug #1448747 skipped on platform (mac)
The SKIPPED tests should be run on the appropriate platform, with logins, or checked manually if necessary.
FAILED tests require need to be checked to confirm whether the intervention is still working or necessary. Sometimes a test will fail because a site has changed, but it may fail simply due to a networking or other hiccup. As such it may be worth running each failing bug number's tests a second or third time, using the --bug= parameter as shown above (including the config file if the test requires a login).