QA/Automation/Projects/Mozmill Automation/On Demand Test Framework/CI Documentation
From MozillaWiki
< QA | Automation | Projects | Mozmill Automation | On Demand Test Framework(Redirected from Auto-tools/Automation Development/Projects/Mozmill Automation/On Demand Test Framework/CI Documentation)
Contents
NOTE
This is a work in progress and should be more detailed soon
Overview
Master Responsibilities
- Fetch build environment artifacts
- Centralized on master to limit dependencies
- Master itself is configured as a build node specifically to fetch these environments
- Create core jobs on itself via Jenkins API
- Grabs from davehunt's Github repo
- Uses python-jenkins to update the jobs on itself
- Will leave old jobs behind if a platform is removed from requirements
- Run the job cascade from the configuration file via the on-demand job
- Push jobs to slaves
Slave Responsibilities
- Pull build environment artifacts from master
- Copy Artifact plugin
- Run the test jobs (Mozmill scripts)
- Mozmill scripts push to brasstacks (slaves need network visibility to results repo)
- Currently one build executor per node because no simultaneous Mozmill. New focus monitor changes gives us a shot at parallelism
Flow
Fetch Mozmill environments
- We kick off an environment job on master
- Master contacts itself as a build node
- Master fetches environment from http://people.mozilla.com/~hskupin/downloads/mozmill-env/
- Now available as an artifact
Test job
- We kick off on-demand job on master w/ configuration file
- Configuration job used to create a new specific job w/ UUID in name
- New specific job has build triggers in it for each configuration to trigger parametrized core jobs. Immediately finishes after kicking all these off.
- Core jobs added to build queue and processed as nodes come available
- Core jobs have a dependencies on environments. Use artifacts as built above.
- Core jobs each translate to one run of the Mozmill scripts.
Environment Targets
- Retrieve them from a known place
- Poll for newer environments and fetch on demand
Build Targets
Coming soon
Test Responsibilities
Coming soon
Installation
Install Prerequisites
- Java
- Python
- pip (for jenkins-python)
- python-jenkins (http://pypi.python.org/pypi/python-jenkins/0.2)
- Git
- curl
- Jenkins
Instructions
- Install and run Jenkins (java -jar jenkins.war)
- Open http://localhost:8080
- Install the following plugins (Manage Jenkins > Manage Plugins)
- Mercurial Plugin
- Green Balls
- Xvnc (for Linux slaves)
- Parametrized Trigger
- GitHub
- Git
- Show Build Parameters
- Rebuild
Configuration
Master
SSH login
- Generate public/private keys using RSA
- Keep the public key
Jenkins
- Set # of executors to 1
Windows Slave
Jenkins
- Set # of executors to 1
Linux, OS X Slave
SSH login
- Create activated_keys with master's public key
Jenkins
- Set # of executors to 1
Creating jobs
Mozmill Environments
- Create a new (multi-configuration) job named download-mozmill-environments with the following configuration:
- Source code management: None
- Add axis:
- Type: Slaves
- Name: NODE
- Check 'master' under 'Individual nodes'
- Add axis:
- Type: User-defined
- Name: PLATFORM
- Values: linux win mac
- Add build step: Execute Shell with the following:
- Check 'Archive the artifacts'
- Files to archive: $MOZMILL_VERSION.*.zip
Core Testrun Jobs
- Create a new (free-style) project named create-testrun-jobs with the following configuration:
- GitHub project: https://github.com/davehunt/mozmill-jenkins-config/
- Check 'Restrict where this project can run'
- Label Expression: master
- Source code management: Git
- Repository URL: https://github.com/davehunt/mozmill-jenkins-config.git
- Branch Specifier: master
- Add build step: Execute Shell with the following:
- python create_testrun_jobs.py
- Save and run this job - it will create the core testrun jobs in Jenkins
Trigger On-Demand Job
- Create a new (free-style) project named trigger-ondemand with the following configuration:
- GitHub project: https://github.com/davehunt/mozmill-jenkins-config/
- Check 'This build is parametrized'
- Add a file parameter
- File location: config.ini
- Check 'Restrict where this project can run'
- Label Expression: master
- Source code management: Git
- Repository URL: https://github.com/davehunt/mozmill-jenkins-config.git
- Branch Specifier: master
- Add build step: Execute Shell with the following:
- python trigger_on_demand_testrun.py config.ini
Running test suites
On-demand test suite
- Open the Jenkins front end: http://localhost:8080/
- Locate the 'trigger-ondemand' job. Use the 'admin' view to make this easier to find.
- Click 'Schedule a build' to the right of the job (or click the job name followed by 'Build Now' on the left)
- You will be prompted for a configuration file. This is almost identical to the file used by the existing testrun_release.py script. You can find two suitable examples below.
- Click 'Build'
- That's all you need to do! What happens next is:
- The job uses the Jenkins API to create a new job named 'ondemand-<testrun_type>-<uuid>'
- This job does nothing itself, but triggers additional jobs with parameters as specified by the configuration file.
- The triggered jobs are added to the build queue and executed as the appropriate slaves become available.
- These trigger jobs perform the following:
- Clear the workspace.
- Copy the latest artifact from the appropriate mozmill-environment job.
- Unzip and configure the Mozmill environment for use.
- Clone mozmill-automation repository.
- Run the download.py script to get the appropriate Firefox binary.
- Run the testrun_<testrun_type>.py script.
- The configuration of all of these testrun jobs is controlled by a rather crude script named create_testrun_jobs.py available here: https://github.com/davehunt/mozmill-jenkins-config. It creates a job for each node type (determined by combining a number of labels). At the moment this is limited to OS name/version but could be more granular in the future.
Daily test suite
For now these need to be set up manually, but it would only require a few small tweaks to the configuration file in order to have these created for us. I have manually configured an example daily test suite for addons, endurance, functional, l10n, and remote on our MPT instance
Proof of Concepts
Selenium
- CI server: Atlassian Bamboo (xserve.openqa.org:8085)
- Trigger: The build for Selenium IDE is triggered whenever a change is committed to the SVN repository, or an upstream job is built. Whenever the add-on is built the tests are executed.
- Reports:
- JUnit compatible report (patch in review) is generated and presented by Bamboo.
- Test reports are sent to http://mozmill-archive.brasstacks.mozilla.com/#/addons.
- Log is generated and archived as a build artifact.
- Prerequisites: The job is run on a Bamboo agent that has Python, Mercurial, and Mozmill preinstalled. Any updates to these would have to be taken care of manually. It would be possible to install Mozmill as part of the job, however Mercurial is an absolute requirement as it's needed to check out mozmill-tests and mozmill-automation from hg.mozilla.org
- Customizations: As we are running against a snapshot version of Selenium IDE, it is necessary to overwrite the addons.ini file with the location of the snapshot XPI file. This requires first checking out mozmill-tests from hg.mozilla.org, creating the replacement file, committing the change to the local repository, and then specifying the location of the local repository when running the mozmill-automation script.
- Notifications:
- An e-mail report is sent to Dave Hunt.
- selbot2 in #selenium on irc.freenode.net reports build results via RSS feed.
Mozmill
- CI server: Jenkins (running in a local VM at this time)
download-latest-nightly
- Purpose: Download the latest Nightly build for testing.
- Execute: ./download.py --directory=latest-nightly --platform=linux64 --type=daily --branch=mozilla-central
- Trigger: This job is triggered whenever the contents at http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/ is updated. This may fail if the contents has updated but the build is not yet fully available, so we are likely to want a more robust build trigger in the future.
- Reports: Standard Jenkins build report.
- Prerequisites: Requires Python and Mercurial.
- Notifications: An e-mail report is sent to Dave Hunt if this job fails.
- Notes:
- The downloaded build is accessible to other jobs.
- On completion, the Mozmill jobs listed below are executed.
mozmill-addons-latest-nightly
- Purpose: Run the addons tests against the latest Nightly build.
- Execute: ./testrun_addons.py --junitxml=report.xml --logfile=addons.log --report=http://mozmill-archive.brasstacks.mozilla.com/db/ --with-untrusted /var/lib/jenkins/jobs/download-latest-nightly/workspace/latest-nightly
- Reports:
- JUnit compatible report (patch in review) is generated and presented by Jenkins.
- Test reports are sent to http://mozmill-archive.brasstacks.mozilla.com/#/endurance.
- Log is generated and archived as a build artifact.
- Notifications: An e-mail report is sent to Dave Hunt if this job fails.
mozmill-endurance-latest-nightly
- As mozmill-addons-latest-nightly but for endurance tests.
mozmill-functional-latest-nightly
- As mozmill-addons-latest-nightly but for functional tests.
mozmill-l10n-latest-nightly
- As mozmill-addons-latest-nightly but for l10n tests.
mozmill-remote-latest-nightly
- As mozmill-addons-latest-nightly but for remote tests.
Notes
- URL build trigger for downloading latest build is not the best solution. We need to come up with another.
- Mozmill automation scripts always exit with a successful exit code. This means that the jobs are reported as successful even if there are failures. Fortunately, at least Jenkins updates the job result based on the JUnit report, so failures would not go unnoticed. See bug 626712 for introducing valid exit codes.
- Jenkins jobs support parameters when manually triggered. We could use this to target specific locales, or to make the download script more flexible. These parameters cannot be controlled when using build triggers however, and defaults would be used.
- We can use multiple-configuration builds to run the same job with different parameters. For example I have set up download-latest-nightly-multiple-locales to download de, en-GB, and en-US. This triggers mozmill-l10n-latest-nightly-multiple-locales with the same configuration.
Questions/Proposals
- A plugin like the following could help us to prepare the workspace for testruns across slaves of the same platform. Downloads will not have to happen in parallel by the slaves but will all be done by the "master" and handed over to the slaves
- Do we want to place the master's Jenkins folder under the home directory of the user? Therefore we should work with the .war file instead of installing a native Jenkins build onto the system?
- Do we have a fallback strategy if something is broken? Could we still trigger/execute tests manually for a given platform.
- For environment targets: Unzipping vs downloading without packing. Which is more complicated?