Motivation

WebCompat Go Faster Addon Version 2 and beyond, we plan to support more complicated function to patch website in complicated Firefox. Website changes from time to time; changes may break our patch, also break user experience when using Firefox. To prevent this situation, we use autotest to detect out-dated patch and fix/remove patch in affordable period. This document describes a minimum viable prototype to verify our concept, learn more experience and establish a formal product.

Architecture

Current autotest leverages great works of Jenkins Continuous Integration. In the following sections, I will describe each component in the workflow.

Jenkins Master

Jenkins master is the control tower of autotest. Each site patch in WebCompat Go Faster Addon maps to a job in Jenkins autotest. We use the schedule function to create a task and run test for the site patch periodically.

We don't really run test on Jenkins Master, but assign the task to a Jenkins slave. Each job defines:

• Which Jenkins slave to run the test
• The test case repository
• Batch command (or script) to run test
• Postbuild script to upload result

You need to check if these 4 are configured properly or refer to the dummy test case.

Jenkins Slave

We plan to have Windows, Mac OS X and Linux machines as Jenkins slaves to serve all tests, but currently we only have 1 Windows 7 machine runs platform independent test case and Windows specific test case.

Test cases are on GitHub. While task begins, Jenkins slave pulls the repository from github, runs the batch command (download latest Firefox for this platform, and test case). Any error level not 0 fail the task.

Firebase & Dashboard

It's not good enough to use Jenkins to see the test result, 2 reasons:

1. We can't easily see which test failed.
2. Jenkins master and slave are in Taipei office private network. For those who are not in this office, VPN to Mozilla data center is required.

To solve this issue, a Post Build Script is used to get latest build result and upload to my Firebase database. We store test name, build ID, result, timestamp and console output in Firebase database using the schema(?):

• webcomapt-sysaddon-autotest
  • job
    •  %JOB_NAME%
      •  %BUILD_ID%
        • • { result: %RESULT%, timestamp: %TIMESTAMP% }
  • job_console
    •  %JOB_NAME%
      •  %BUILD_ID%
        • { output: %CONSOLE_OUTPUT% }

To display the result, you can visit the Dashboard too see the result. The page grabs the result from Firebase directly. Move cursor on build can see the timestamp, click to see the full console output can help to fix the test.

Limitation

• Geo-restriction: If a test case requires run in specific country, this is not ready or well tested. But it's a problem indeed.
• Private network: As said, this architecture is in a private network, not friendly for everyone to access.

Plans

• Move this prototype to try server which will be more stable and public for everyone.